Sunday, March 28, 2021

Developer tools can be magic. Instead, they collect dust.

I started working on advanced developer tools 9 years ago. Back when I started, “programming tools” meant file format viewers, editors, and maybe variants of grep. I’d mention a deep problem such as inferring the underlying intent of a group of changes, and get questions about how it compares to find-and-replace.

Times have changed. It’s no longer shocking when I meet a programmer who has heard of program synthesis or even tried a verification tool. There are now several1 popular products based on advanced tools research, and AI advances in general have changed expectations. One company, Facebook, has even deployed automated program-repair internally.

In spite of this, tools research is still light-years ahead of what’s being deployed. It is not unusual at all to read a 20 year-old paper with a tool empirically shown to make programmers 4x faster at a task, and for the underlying idea to still be locked in academia.

I’d like to give a taste of what to expect from advanced tools — and the ways in which we are sliding back. I will now present 3 of my favorite tools from the last 30 years, all of which I’ve tried to use, none of which currently run.

Reflexion Models

We often think of software in terms of components. For an operating system, it might be: file system, hardware interface, process manager. An experienced engineer on the project asked to make certain files write to disk faster will know exactly where to go in the code; a newcomer will see an amorphous blob of source files.

In 1995, as a young grad student at the University of Washington, Gail C. Murphy came up with a new way of learning a codebase called reflexion models.

First, you come up with a rough hypothesis of what you think the components are and how they interact:

Then, you go through the code and write down how you think each file corresponds to the components.

Now, the tool runs, and computes the actual connectivity of the files (e.g.: class inheritance, call graph). You compare it to your hypothesis.

Armed with new evidence, you refine your hypothesis, and make your mental model more and more detailed, and better and better aligned with reality.

Around this time, a group at Microsoft was doing an experiment to see if they could re-engineer the Excel codebase to extract out some high-level components. They needed a pretty strong understanding of the codebase, but getting it wouldn’t be so easy, because they were a different team in a different building. One of them saw Gail’s talk on reflexion models and liked it.

In one day, he created his first cut of a reflexion model for Excel. He then spent the next four weeks refining it as he got more acquainted with the code. Doing so, he reached a level of understanding that he estimates would have taken him 2 years otherwise.

Today, Gail’s original RMTool is off the Internet. The C++ analysis tool from AT&T it’s based on, Ciao, is even more off the Internet. They later wrote a Java version, jRMTool, but it’s only for an old version of Eclipse with a completely different API. The code is written in Java 1.4, and is no longer even syntactically correct. I quickly gave up trying to get it to run.

Software engineering of 2021: Still catching up to 1995.


The WhyLine

About 10 years later, at the Human-Computer Interaction Institute at Carnegie Mellon, Amy Ko was thinking about another problem. Debugging is like being a detective. Why didn’t the program update the cache after doing a fetch? What was a negative number doing here? Why is it so much work to answer these questions?

Amy had an idea for a tool called the Whyline, where you could ask questions like “Why did ___ happen?” in an interactive debugger? She built a prototype for Alice, CMU’s graphical programming tool that let kids make 3D animations. People were impressed.

Bolstered by their success, Amy spent another couple years working hard, building up the technology to do this for Java.


They ran a study. 20 programmers were asked to fix two bugs in ArgoUML, a 150k line Java program. Half of them were given a copy of the Java WhyLine. The programmers with the WhyLine were 4 times more successful than those without, and worked twice as fast.

A couple years ago, I tried to use the Java Whyline. It crashed when faced with modern Java bytecode.

MatchMaker

In 2008, my advisor, Armando Solar-Lezama, was freshly arrived at MIT after single-handedly reviving the field of program synthesis. He had mostly focused on complex problems in small systems, like optimizing physics simulations and bit-twiddling. Now he wanted to solve simple problems in big systems. So much of programming is writing “glue code,” taking a large library of standard components and figuring out how to bolt them together. It can take weeks of digging through documentation to figure out how to do something in a complex framework. Could synthesis technology help? Kuat Yessenov, the Kazakh genius, was tasked with figuring out how.

Glue code is often a game of figuring out what classes and methods to use. Sometimes it’s not so hard to guess: the way you put a widget on the screen in Android, for instance, is with the container’s addView method. Often it’s not so easy. When writing an Eclipse plugin that does syntax highlighting, you need a chain of four classes to connect the TextEditor object with the RuleBasedScanner.

class UserConfiguration extends SourceViewerConfiguration {
  IPresentationReconciler getPresentationReconciler() {
    PresentationReconciler reconciler = new PresentationReconciler();
    RuleBasedScanner userScanner = new UserScanner();
    DefaultDamagerRepairer dr = new 
    DefaultDamagerRepairer(userScanner);
    reconciler.setRepairer(dr, DEFAULT_CONTENT_TYPE);
    reconciler.setDamager(dr, DEFAULT_CONTENT_TYPE);
    return reconciler;
  }
}

class UserEditor extends AbstractTextEditor {
  UserEditor() {
    userConfiguration = new UserConfiguration();
    setSourceViewerConfiguration(userConfiguration);
  }
}
class UserScanner extends RuleBasedScanner {...}

If you can figure out the two endpoints of a feature, what class uses it and what class provides it, he reasoned, then you could ask a computer to figure out what’s in-between. There are other programs out there that implement the functionality you’re looking for. By running them and analyzing the traces, you can find the code responsible for “connecting” those two classes (as a chain of pointer references). You then boil the reference program down to exactly the code that does this — voila, a tutorial! The MatchMaker tool was born.

In the study, 8 programmers were asked to build a simple syntax highlighter for Eclipse, highlighting two keywords in a new language. Half of them were given MatchMaker and a short tutorial on its use. Yes, there were multiple tutorials on how to do this, but they contained too much information and weren’t helpful. The control group floundered, and averaged 100 minutes. The MatchMaker users quickly got an idea what they were looking for, and took only 50 minutes. Not too bad, considering that an Eclipse expert with 5 years experience took a full 16 minutes.

I did actually get to use Matchmaker, seeing as I was asked to work on its successor in my first month of grad school. Pretty nice; I’d love to see it fleshed out and made to work for Android. Alas, we’re sliding back. A few years back, my advisor hired a summer intern to work on MatchMaker. He instantly ran into a barrier: it didn’t work on Java 8.

Lessons

The first lesson is that the tools we use are heavily shaped by the choices of eminent individuals. The reason that Reflexion Models are obscure while Mylyn is among the most popular Eclipse plugins is quite literally because Gail C. Murphy, creator of Reflexion Models, decided to go into academia, while her student Mik Kersten, creator of Mylyn, went into industry.

Programming tools are not a domain where advances are “an idea whose time has come.” That happens when there are many people working on similar ideas; if one person doesn’t get their idea adopted, then someone else will a few years later. In programming tools, this kind of competition is rare. To illustrate: A famous professor went on sabbatical to start a company building a tool for making websites. I asked him why, if his idea was going to beat all the previous such tools, it hadn’t been done before. His answer was something like “because it requires technology that only I can build.”

The second lesson is that there is something wrong with how we build programming tools. Other fields of computer science don’t seem to have such a giant rift between the accomplishments of researchers and practitioners. I’ve argued before that this is because the difficulty of building tools depends more on the complexity of programming languages (which are extremely complicated; just see C++) than on the idea, and that, until this changes, no tool can arise without enough sales to pay the large fixed cost of building it. This is why my Ph. D. has been devoted to making tools easier to build. It is also why I am in part disheartened by the proliferation of free but not-so-advanced tools: it lops off the bottom of the market and makes these fixed-costs harder to pay off.

But the third lesson is that we as developers can demand so much more from our tools. If you’ve ever thought about building a developer tool, you have so much impressive work to draw from. And if you’re craving better tools, this is what you have to look forward to.


Sources


1 I’d list some, but I don’t want to play favorites. I’ll just mention CodeQL, which is quite advanced and needs no touting.

Monday, March 15, 2021

Why Programmers Should(n't) Learn Theory

I’m currently taking my 5-person advanced coaching group on a month-long study of objects. It turns out that, even though things called “objects” are ubiquitous in modern programming languages, true objects are quite different from the popular understanding, and it requires quite a bit of theory to understand how to recognize true objects and when they are useful. As our lessons take this theoretical turn, one asks me “what difference will this make?”

I recently hit my five-year anniversary of teaching professional software engineers, and now is a great time to reflect on the role that theoretical topics have played in my work, and whether I’d recommend someone looking to become the arch-engineer of engineers should include in their path steps I’ve taken in developing my own niche of expertise.

I’ve sometimes described my work as being a translator of theory, turning insights from research into actionable advice from engineers. So I’ve clearly benefited from it myself. And yet I spend a lot of time telling engineers not to study theory, or that it will be too much work for the benefit, or that there are no good books available.

Parts of it are useful sources of software-engineering insight, parts are not. Parts give nourishment immediately; parts are rabbit holes. And some appear to have no relevance to practical engineering until someone invents a new technique based on it.

I now finally write up my thoughts: how should someone seeking to improve their software engineering approach learning theory?

What is theory?

“If I were a space engineer looking for a mathematician to help me send a rocket into space, I would choose a problem solver. But if I were looking for a mathematician to give a good education to my child, I would unhesitatingly prefer a theorizer.”

— Gian-Carlo Rota, “Problem Solvers and Theorizers

“Theory” in software development to me principally means the disciplines studied by academic researchers in programming languages and formal methods. It includes type theory, program analysis, program synthesis, formal verification, and some parts of applied category theory.

Some people have different definitions. There’s a group called SEMAT, for instance, that seeks to develop a “theory” of software engineering, which apparently means something like writing a program that generates different variants of Agile processes. I don’t really understand what they’ve been up to, and never hear anyone talking about them.

What about everything else which is important when writing software? UX design has a theory. Distributed systems have a theory. Heck, many books about interpersonal topics can be called “theory.”

Well, I’ve stated what I think of when I hear “software engineering” and “theory” together. Plenty of people disagree. You can call me narrow-minded. More importantly though, I’m only writing about things that I’m an expert in by the standards of industry programmers. If you read a couple of textbooks about any of those other subjects, you’ll probably know more than I do.

One more preliminary: I’ve named 5 different subfields, but the boundary between them gets very blurry. There is a litmus test for whether a topic is part of category theory, namely whether it deals with mathematical objects named “categories,” but, otherwise, most specific topics are studied and used in multiple disciplines, and the boundaries between them are defined as much by history and the social ties of researchers as by actual technical differences.

So, for many things, if you ask “What about topic X,” I could say “X is a part of type theory,” but what I’d really mean is “The theoretical aspects of X are likely to show up in a paper or book that labels itself ‘type theory.’”

In the remainder of this piece, I’ll discuss these 5 disciplines of programming languages and formal methods, and discuss how each of them does or does not provide useful lessons for the informal engineer.


Type Theory

“Type theory” can roughly be described as the study of small/toy programming languages (called “calculi”) designed to explore some kind of programming language feature. Usually, those features come with types that constrain how they are used, and, occasionally, those types have some deep mathematical significance, hence the name “type theory.” It can be quite deep indeed  did you know that “goto” statements are connected to Aristotle’s law of the excluded middle? Outside of things like that though, type theory is not about the types.

Studying type theory is the purest way to understand programming language features, which are often implemented in industrial languages in a much more complicated form. And, for this reason, studying type theory is quite useful.

To the uninitiated, programming is a Wild West of endless possibilities. If your normal bag of tricks doesn’t work, you can try monkey-patching in a dynamic proxy to generate a metaclass wildcard. To the initiated, the cacophony of options offered by the word-salad in the previous sentence boils down to a very limited menu: “this is just a Ruby idiom for doing dynamic scope.”

So many thorny software engineering questions become clear when one simply thinks about how to express the problem in one of these small calculi. I think learning how to translate problems and solutions into type theory should be a core skill of any senior developer. Instead of looking at the industry and seeing a churning sea of novelty, one sees but a polishing of old wisdom. Learning type theory, more than any other subfield of programming languages, fulfills the spirit of the Gian-Carlo Rota quote above, that one should pick a theorizer for a good education.

Perhaps the most profound software-engineering lesson of type theory comes from understanding existential types. They are utterly alien to one only familiar with industrial languages. Yet they formalize “abstraction boundaries,” and the differences between different ways of abstraction such as objects vs. modules are best explained by their different encodings into existential types. In fact, I will claim, however, that it is quite difficult (and perhaps impossible, depending on your standards) to fully grasp an abstraction boundary without understanding how to translate it into type theory. Vague questions like “is it okay to use this knowledge here?” become crisp when one imagines an existential package bounding the scope.

Type theory helps greatly in understanding many principles of software design. I’ve found many times I can teach some idea in a few minutes to an academic that takes me over an hour to teach a layprogrammer. But it is by no means inevitable that picking up a type theory book will lead to massive improvement without also studying how to tie it to everyday programming. It’s easy to read over the equality rule for mutable references, for instance, and regard it as a curiosity of symbolic manipulation. And yet I used it as the kernel of a 2,500-word lesson on data modeling. The ideas are in the books, but the connections are not. And thus, indeed, I’ve encountered plenty of terrible code written by theoretical experts.


Program Analysis

Program analysis means using some tool to automatically infer properties of a program in order to e.g.: find bugs. It can be broadly split into techniques that run the code (dynamic analysis) vs. those that inspect it without running it (static analysis); both of these have many sub-families. For example, a dynamic deadlock detector might run the program and inspect what order it acquires locks, and conclude that a program that does not follow sound lock-discipline and is in danger of deadlocking. (This is different from testing, which may be unable to discover a deadlock without being exceptionally (un)lucky in getting the right timings.) In contrast, a static analyzer would trace through all possible paths in the source code to discover all possible lock orderings. I’ll focus on static analysis for the rest of this section, where most of the theory lies.


First, something to get out of the way: Static analysis has become a bit of a buzzword in industry, where it’s used to describe a smorgasbord of tools that run superficial checks on a program. To researchers, while the term “static analysis” technically describes everything that can be done to a program without running it, it is typically used to describe a family of techniques that, loosely speaking, involve stepping through a program and tracking how it affects some abstract notion of intermediate state, a bit like how humans trace through a program. Industrial bug-finding tools such as Coverity, CodeQL, FindBugs, and the Clang static analyzer all include this more sophisticated kind of static analysis, though they all also mix in some more superficial-but-valuable checks as well. I refer to this excellent article by Matt Might as a beginner-level intro.

This deeper kind of static analysis is done by a number of techniques which have names such as dataflow analysis, abstract interpretation, and effect systems. The lines between the approaches get blurrier the deeper you go, and I’ve concluded that the distinctions between them often boil down to little things such as “constraints about the values of variables cannot affect the order of statements” (dataflow vs. constraint-based analysis) and “the only way to merge information from multiple branches (e.g.: describing the state of a program after running a conditional) is to consider the set containing both” (model-checking vs. dataflow analysis).

Is studying static analysis useful for understanding software engineering? I’ve changed my mind on this recently.

I think the formal definition of an abstraction as seen in static analysis, specifically the subfield called “abstract interpretation,” is useful for any engineer to know. Sibling definitions of abstraction also appear in other disciplines as well such as verification, but one cannot study static analysis without understanding it.

Beyond that, however, I cannot recall any instance where I’ve used any concept from static analysis in a lesson about software engineering.

Static analysis today is focused on tracking simple properties of programs, such as whether two variables may reference the same underlying object (pointer/alias analysis) or whether some expression is within the bounds of an array (covered by “polyhedral analysis” and its simplified forms). When more complex properties are tracked, it is typically centered around usage of some framework or library (e.g.: tracking whether files are opened/closed, tracking the dimensions of different tensors in a TensorFlow program) or even tailored to a specific program (example). Practical deployments of static analysis are a balancing act in finding problems which are important enough to merit building a tool, difficult enough to need one, and shallow enough to be amenable to automated tracking.

A common view is that, to get a static analyzer to track the deeper properties of a program that humans care about, one must simply take existing techniques and just add more effort. As my research is on making tools easier to build, I recently spent weeks thinking about specific examples of which such deeper properties could be tracked upon magically conjuring more effort, and concluded that, on the contrary, building such a “human-level analyzer” is well beyond present technology.

Static analysis is the science of how to step through a program and track what it is doing. Unfortunately, the science only extends to tracking shallow properties. But there is plenty of work tracking more complex properties: it’s done in mechanized formal verification.


Program Synthesis

Program synthesis is exactly what it says on the tin: programs that write programs. I gave an entire talk on software engineering lessons to be drawn from synthesis. There is much inspiration to take from the ideas of programming by refinement and of constraining the search space of programs.

But, that doesn’t mean you should go off and learn the latest and greatest in synthesis research.

First, while all the ideas in the talk are used in synthesis, many of the big lessons are from topics that do not uniquely belong to synthesis. The lessons about abstraction boundaries, for instance, really come from the intersection of type theory and verification.

Second, the majority of that talk is about derivational synthesis, the most human-like of the approaches. Most of the action these days is in the other schools: constraint-based, enumerative, and neural synthesis. All of these are distinctly un-human-like in their operation  well, maybe not for neural, but no-one understands what those neural nets are doing anyway. There are nonetheless software-engineering insights to be had from studying these schools as well, such as seeing how different design constraints affect the number of allowed programs, but if you spend time reading a paper titled “Automated Synthesis of Verified Firewalls,” to use a random recent example, you’re unlikely to get insight into any aspect of software engineering other than how to configure firewalls. (But if that’s what you’re doing, then go ahead. Domain-specific synthesizers do usually involve deep insights about the domain.)


Formal Verification

Formal verification means using a tool to rigorously prove some property (usually correctness) of a program, protocol, or other system. It can broadly be split into two categories: mechanized and automated. In mechanized verification/theorem-proving, engineers attempt to prove high-level statements about a program’s properties by typing commands into an interactive “proof assistant” such as Coq, Isabelle, or Lean. In automated verification, they instead pose a query to the tool, which returns an answer after much computation. Both require extensive expertise to model a program and its properties.

Mechanized verification can provide deep insights about everyday software engineering. For instance, one criteria for choosing test inputs in unit testing is “one input per distinct case,” and doing mechanized verification teaches one what exactly is a “case.” Its downside: in my personal experience, mechanized verification outclasses even addictive video games in its ability to make hours disappear. As much as programming can suck people into a state of flow and consume evenings, doing proofs in Coq takes this to another level. There’s something incredibly addicting about having a computer tell you every few seconds that your next tiny step of a proof is valid. Coq is unfortunately also full of gotchas that are hard to learn about without expert guidance. I remember a classmate once made a subtle mistake early on in a proof, and then spent 10 hours working on this dead end.

I do frequently draw on concepts from this kind of verification. Most notably, I teach students Hoare logic, the simplest technique for proving facts about imperative programs. But I do it mostly from the perspective of showing that it’s possible to rigorously think about the flow of assumptions and guarantees in a program. I tell students to handwave after the first week in lieu of finding an encoding of “The system has been initialized” in formal logic, and even leave off the topic of loop invariants, which are harder than the rest of the logic. Alas, this means students lose the experience of watching a machine show them exactly how rule-based and mechanical this kind of reasoning is.

Automated tools take many forms. Three categories are solvers such as Z3, model-checkers such as TLA+  and CBMC, and languages/tools that automatically verify program contracts such as Dafny.

have argued before that software design is largely about hidden concepts underlying a program that have a many-many relationship with the actual code, and therefore are not directly derivable from the code. It follows that the push-button tools are limited in the depth of properties they can discuss, and therefore also in their relevance to actual design. Of these, the least push-button is TLA+, which I’ve already written about, concluding there were only a couple things with generalizable software design insights, though just learning to write abstracted models can be useful as a thinking exercise.

In short, learning mechanized verification can be quite deep and insightful, but is also a rabbit hole. Automated verification tools are easier to use, but are less deep in the questions they can ask, and less insightful unless one is actually trying to verify a program.

I’ve thought about trying to create a “pedagogical theorem prover” in which programmers can try to prove statements like “This function is never called unless the system has been initialized” without having to give a complicated logical formula describing how “being initialized” corresponds to an exact setting of bits. Until then, I’m still on the lookout for instruction materials that will provide a nice on-ramp to these insights without spending weeks learning about Coq’s difference between propositions and types.


Category Theory

Category theory is a branch of mathematics which attempts to find commonalities between many other branches of mathematics by turning concepts from disparate fields into common objects called categories, which are essentially graphs with a composition rule. In the last decade, category theory in programming has escaped the ivory tower and become a buzzword, to the point where a talk titled “Categories for the Working Hacker” can receives tens of thousands of views. And at MIT last year, David Spivak taught a 1-month category theory course and had over 100 people show up, primarily undergrads but also several local software engineers.


I studied a lot of category theory 8 years ago, but have only found it somewhat useful, even though my research in generic programming touches a lot of topics heavily steeped in category theory. The chief place it comes up when teaching software engineering is a unit on turning programs into equivalent alternatives, such as why a function with a boolean parameter is equivalent to two functions, and how the laws justifying this look identical to rules taught in high school algebra. The reason for this comes from category theory (functions are “exponentials,” booleans are a “sum”), but it doesn’t take category theory to understand.

Lately, I’ve soured on category theory as a useful topic of study, and I became fully disillusioned last year after studying game semantics. Game semantics are a way of defining the meaning of logical statements. Imagine trying to prove a universally-quantified statement like “Scruffles is the cutest dog in the world.” Traditional model-based semantics would say this statement is true if, for all dogs in the world, that dog is either Scruffles or is less cute than Scruffles. Game semantics casts this as a game between you (the “Verifier”) and another party (the “Falsifier”). It goes like this: They present you another dog. If you can show the dog is less cute than Scruffles (or rather, they are unable to find a dog for which you cannot do so), then the statement is true and you win. Otherwise, the statement is false.

(I admit that having this kind of conversation has been my main application of game semantics thus far.)

I started reading the original game semantics paper. The first few sections explained pretty much what I just told you in semi-rigorous terms, and were enlightening. The next section gave category-theoretic definitions of the key concepts. I found this section extremely hard to follow; it would have been easier had they laid it out directly rather than shoehorning it into a category. And while a chief benefit of category theory is the use of universal constructions that allow insights to be transported across disciplines, the definitions here were far too specialized for that to be plausible.

There’s a style of programming called point-free programming which involves coding without variables. So, instead of writing an absolute value function as if x > 0 then x else -x, you write it as sqrt . square, where the . operator is function composition. Category theory is like doing everything in the point-free style. It can sometimes lead to beautifully short definitions that enable a lot of insight, but it can also serve to obscure needlessly.

I’m still open to the idea that there may be a lot of potential in learning category theory. David Spivak told me “A lot of what people do in databases is really Kan extensions” and said his collaborators were able to create an extremely powerful database engine in a measly 5000 lines of Java. Someday, I’ll read his book  and find out. Until then, I don’t recommend programmers study category theory unless they like learning math for its own sake.



Conclusion

So, here’s the overall tally of fields and their usefulness in terms of lessons for software design:

  • Type theory: Very useful
  • Program Analysis: Not useful, other than the definition of “abstraction.”
  • Program Synthesis: Old-school derivational synthesis is useful; modern approaches less so.
  • Formal Verification: Mechanized verification is very useful; automated not so much.
  • Category theory: Not useful, except a small subset which requires no category theory to explain.

So, we have a win for type theory, a win for the part of verification that intersects type theory (by dealing with types so fancy that they become theorems), and a wash for everything else. So, go type theory, I guess.

In conclusion, you can improve your software engineering skills a lot by studying theoretical topics. But most of the benefit comes from the disciplines that study how programs are constructed, not those that focus on how to build tools.


Appendix: Learning Type Theory

Some of you will read the above and think "Great," and then rush to Amazon to order a type theory book.

I unfortunately cannot endorse that, namely because I can't endorse reading textbooks as a good way to learn type theory.

The two main intro type theory textbooks are Practical Foundations for Programming Languages (PFPL) and Types and Programming Languages (TAPL). I'm more familiar with PFPL, but I regard them as pretty similar. Basically, both present the subject as pretty dry, consisting of a list of rules, although they try to add excitement by discussing their significance. The rules have a way of coming alive when you try to come up with them yourself, or come up with rules for a variant of an idea found in a specific programming language. This is how I tutor my undergrads, but it's hard to communicate in book format.

So, what's the right way to learn type theory on your own? I don't have one. The best I can share right now is the general advice that following a course website that uses a book is often better than using the book itself. The best way for practitioners to learn type theory has yet to be built.

Wednesday, September 30, 2020

Book Review: Elements of Programming

The C++ STL may be the most impressive achievement in language standard libraries. Where most programmers are stuck complaining that their language’s default strings aren’t performant enough, about every standard C++ function for strings actually runs on arbitrary character sequences. Design your own container? std::find_if works just as well as for the built-ins. And it does this while often being more performant than the code you’d write yourself.

Alex Stepanov is the man who made this happen, and Elements of Programming (EOP) is his 200-page paean to his method for writing generic code. He’s a believer that programming can be turned from an art to a rigorous discipline based on mathematics, and I’ve long admired him for his deep knowledge and impact. Indeed, he’s spent years at #1 on my list of people I’d like to have lunch with. (Hey readers — can anyone help?)

And now, I’m about to ruin all that by writing a negative review.

EOP has a strong beginning and a strong finish. The first chapter explains core programming language concepts such as types and state — using non-standard terminology, but probably intentionally, judging by the explanation’s lucidity. This culminates in his big idea of being able to write programs on any type that offers a bundle of operations and properties called a concept, explained in this book over a decade before they entered the C++ standard. The afterword is a few pages of reflection on the power of this approach.

Between them is 11 chapters where he plays the same game: define a new abstraction and then show a bunch of functions that can be written using it. And unfortunately, these functions and abstractions are largely not very interesting.

There’s a famous site called Project Euler, where users write code to solve mathy problems such as “In a modified version of the board game Monopoly, on what three squares is a player most likely to land?” My former programming-contest coach advocated against using it to practice, because “It’s not really programming, and it’s not really math.”

I think this is an apt description of EOP, particularly the first half. This starts from Chapter 2, which is about cool algorithms that involve applying some function to itself repeatedly (iteration). One of my favorite lectures in undergrad was on this topic, and yet I still couldn’t enjoy this chapter, as I know no application of these algorithms outside of niche mathematical topics. This gets taken up another notch in Chapter 5, where, in about 5 pages, he goes from explaining commutativity to defining the algebraic structures of monoids, groups, and rings, all the way up to algebraic modules (no relation to software modules). I cannot fathom these explanations being useful to someone who does not already know these concepts, and certainly not to someone who already does. And while I do know many uses of groups in software engineering — as an abstraction of the idea of an invertible operation — he actually spends the remainder of this chapter considering generalizations of the greatest common divisor function.

I slogged through these chapters, excited for the second half of the book, which focused largely on iterators and containers, things more relevant to typical software engineering. Yet after encountering endless listings of variations of list-copy functions, I found myself no more fulfilled, and soon regressed to skimming through the pages.

Aside from my long-term goal to find all the good writing on software design, I had a short-term goal when reading this book: a student wanted me to teach a lesson based on it. But, halfway through the book, my deadline was fast approaching, and I hadn’t found any material useful enough for a software design lesson for experienced engineers.

I then noticed all the chapters were generated by the deeper principle of coming up with a good abstraction to write generic functions against. I got the idea that maybe I could use EOP as a problem book, telling them to look at the descriptions of generic functions, and then come up with both the code and the abstractions they can be written against. Alas, the topic selection is not suitable for this purpose. One section of the book, for example, deals with computing integer sequences by matrix exponentiation. Asking students to come up with this themselves would be too familiar for many who have taken a linear algebra course, and impossible for those who haven’t. I did design a lesson where students come up with their own abstractions for generic programming problems, but I used examples completely unrelated to the book.

I asked the friend who recommended EOP what he got out of the book, and his first answer was a technique for elegantly expressing state machines using goto’s. I similarly loved that part, but, alas, that was the only concrete thing I got out of this book. I’ll explain it at the end of this review and spare you the other 200 pages.

I have an undergraduate degree in mathematics and have authored several papers on generic programming, so I knew I was reading it for others’ benefit. Still, I don’t think my opinion would be changed were this not the case, and I’d really like help understanding the viewpoint of the many readers who did thoroughly enjoy it. Instead, I find myself agreeing with this Amazon reviewer, although I have too much admiration for Stepanov to contemplate a 1-star rating:

If you've ever written a generic function, you already know that the type parameters must obey a set of preconditions. This book lays out a big pile of definitions for types, numbers, algebraic structures, iterators, and such, so as to bamboozle people easily impressed by mathematical notation. It does so sloppily and writes some trivial generic algorithms in C++. To whatever extent one might accomplish something interesting with this topic, this book doesn't. Avoid.

As I read other material by Stepanov, I mourn for the book that could have been. Stepanov clearly cares about these abstractions and algorithms, to the point where he wrote a second book on largely the same material, with a chattier exposition and chapters more explicitly focusing on pure math. How different would it be had he managed to transmit this appreciation to me? The day after finishing, I watched this talk by a close colleague of Stepanov. “In this menu, you can select a bunch of rows and drag them somewhere else,” he explained over animated slides. “How many of you could implement this in one line?” It made me want to open section 10.4 on “rotation algorithms” again.

I’ve started watching a seminar he gave at Amazon. I’m only a few lectures in, but I’m already enthralled by his high teaching ability. I feel like I’m there with him working through problems. I feel like I’ve learned a great secret as he tells the story of how he invented “regular types,” something used throughout EOP but never motivated. To be honest, I still don’t know what this lecture series is about, but nonetheless expect to recommend it when I’m done.

In short, Stepanov has given many gifts to the world of programming, and EOP is not one of them.

Overall rating: Not recommended

With a smattering of exceptions, EOP neither teaches abstractions useful in everyday programming, nor teaches you the skills to invent your own.

Addendum: State machines by goto’s

“The fastest way to go from one place in code to another is goto.”
Alexander Stepanov

Many iterative algorithms can be described as state machines: first it looks for an X, then it does Y with it, then it looks for another X, and so on. Rather than trying to massage the cycles in the state machine into structured loops, Stepanov advocates a style using goto’s, with one goto-label per machine state.

In searching for an example to best illustrate this, I wanted something where the code was under 40 lines (which ruled out Stepanov’s examples), understandable with little context or knowledge of C++, and which was not equivalent to a regular expression. And so:

In my defunctionalization talk, I showed that many state machines are derived from recursive functions, being turned into iterative traversals by creating a state for each point in the program between recursive calls. For that talk, I demonstrated this in full for the example of printing a binary tree. It turns out that adding parentheses makes this derivation substantially harder, as an arbitrary number of close-parentheses may need to be printed after processing a node. And that difficulty comes from trying to massage a state machine into a loop.

In this case, seeing as I came up with this example by starting with a recursive function, the recursive version is quite simple:

void print_tree_rec(tree *t) {
  if (t != NULL) {
    printf("(");
    print_tree_rec(t->left);
    printf(" %d ", t->element);
    print_tree_rec(t->right);
    printf(")");
  }
}

But, for other state machines, the recursive version is not so easy. For example, Dijkstra created the “shunting yard” algorithm for parsing an arithmetic expression all the way back in 1961, yet I’m not aware of the recursive equivalent being discovered until 2007, using the technique of refunctionalization.

Here’s the state machine:

A confession: the first time I thought about how to make this recursive function function iterative, I didn’t get it, and had to look it up. The solution is to merge the “Next from stack” state in the diagram with its successors, resulting in a solution with two nested while-loops, at the cost of some duplicated code.

However, the version based on goto’s reads off this diagram rather nicely. One C++-ism in this code to note: while the stack s is initialized to NULL, the push() and pop() calls can actually change it.

void print_tree(tree *t) {
  tree *cur = t;
  stack *s = NULL;
  
begin_print_node:
  if (cur == NULL) {
    goto dispatch_stack_frame;
  } else {
    printf("(");
    push(LEFT_CHILD, cur, s);
    cur = cur->left;
    goto begin_print_node;
  }
  
print_element:
  printf(" %d ", cur->element);
  push(RIGHT_CHILD, cur, s);
  cur = cur->right;
  goto begin_print_node;
    
finish_print_node:
  printf(")");
  goto dispatch_stack_frame;

dispatch_stack_frame:
  std::pair<DIR, tree*> top_frame;
  if (s == NULL) goto end;
    
  top_frame = pop(s);
  cur = top_frame.second;
  if (top_frame.first == LEFT_CHILD)
    goto print_element;
  else
    goto finish_print_node;
  
  
end:
  return;
}

For the full code, including data declarations, go here

Neither I nor the friend who recommended this book have had a chance to use this technique since reading it. But for the times when you are implementing a state machine, it can be a nice trick — and a delightful surprise for those who grew up learning "Goto Considered Harmful."

Further reading: http://www.cs.rpi.edu/~musser/gsd/notes-on-programming-2006-10-13.pdf, page 191

Update: Reader Greg Jorgensen writes in to share an article he saved on this topic from the Computer Language magazine in 1991. "I’ve written a few FSMs in this style with successful results. One of them is still in use in the guts of a graphics terminal emulation program. That emulator is still used at big companies like British Telecom." Article

Saturday, December 14, 2019

My Interview on CoRecursive: Advanced Software Design with Jimmy Koppel


A few months ago, I sat down with Adam Gordon-Bell of CoRecursive to share my thoughts on software design and self-improvement.

It's actually a bit funny how I found him. "Corecursion" is an advanced programming term that refers to recursive or iterative programs which are structured around the shape of the output, rather than the shape of the input.  It's connected to the theory of data vs. codata, which explains why interactive programs like an HTTP server look very different from batch programs like a compiler. The canonical example of codata is infinite streams, which need very different programming techniques than in-memory lists.

So, I had been preparing to teach a lesson on codata to a group of my advanced students, when I discovered a podcast by that name, which would be an excellent fit for the topics I write about. And indeed it was. Adam was a skilled podcast interviewer, and managed to elicit a lot of information from me on topics ranging from program synthesis to deliberate practice.

Original podcast link is here. I've created a transcript below.

Topics Discussed

  • Why is Designing Software Hard?
  • Software Intentions
  • Building non-fragile software using assumptions
  • Program Synthesis
  • Program Search Space and Abstraction Barriers
  • Logical Level
  • All Possible Changes to Code
  • Code reviews and Stable interfaces
  • Better Refactoring
  • Don't Don't Repeat Yourself
  • Names matter
  • Getting stuck in a design
  • Code quality is not about code
  • Understand the code that is not there
  • Proof level
  • Software Maintenance
  • Cross Language Refactoring
  • 20 Under 20
  • Benjamin Franklin Method of Programming
  • TOPGUN Programming And Deliberate Practice

Transcript


Jimmy:
I'd say that learning to read this logical level, it's kind of like being able to detect pitch in music, in that everything of software design requires seeing it.

Adam:
Hello, this is Adam Gordon Bell. Join me as I learn about building software. This is CoRecursive.

Adam:
That was Jimmy Koppel. He's a super interesting person. He's working on his Ph. D. in the field of program synthesis at MIT. He was previously paid a hundred thousand dollars to drop out of university by Peter Thiel, but he still graduated somehow.

Adam:
The most interesting thing to me is Jimmy's working hard to teach the world how to design better software. And I think he has a really interesting perspective on this. I guess my summary is after spending a lot of time writing programs that write programs, which is what program synthesis is about, he developed some unique insights into what makes software good, what makes it bad, and he spends his time trying to teach people these insights.

Adam:
So Jimmy, thanks for joining me on the podcast.

Jimmy:
Yeah, it's good to be here.

Adam:
So, I feel like you have a unique perspective on software design. I'd like to ask you some questions about that. So why is designing software hard?

Jimmy:
Why is designing software hard? Why is doing anything hard? So rather than jump to that, I'll tell you, there are certain paths in software that are automatic, and there's certain ones that require creativity.

Jimmy:
So, imagine you're writing a system that inputs from one thing and outputs to another. Very often the nuts and bolts of that thing can be done automatically. So, you get the data in some format, and there's only so many things you can do with it, so it's kind of automatic. Then you have the raw data elements, you need to write some format. That's kind of automatic.

Jimmy:
The creativity enters when you have things going on in-between. So, the creative parts of software design are typically in building the intermediate representations. So the creativity happens in building a representation, designing your data abstractions, and usually the in-between parts become automatic.

Adam:
So, the in-between parts are sort of the serialization and de-serialization or-

Jimmy:
Yeah, things that break the data down, and then build it up again. Those are the glue, and they tend to follow fairly automatically from the actual shape of the data.

Adam:
That makes sense. So, do you think that this middle part, like the software design aspect of doing the middle, that that is like a well understood field?

Jimmy:
I think it's a lot more understood than most people realize. So, we still are very far from being able to automate it well. There's a lot of things that we do know how to do, and that I do teach. But part of what it makes it hard is that so much of what goes into that process, and this is something you'll hear from me again and again today, is really about the intention of how the thing is going to be used, which shapes all the ways it can evolve. And that is something which is not directly apparent from the code.

Adam:
Is intention, like the intent behind building this piece of software, is that the intention? So, let's say I work on some software that takes in a Docker image, so, binary snapshot of a Linux environment, and then it outputs a list of security problems that it found.

Jimmy:
So, part of the intention comes from how you define a security problem. Then from your definition of what a security problem, that determines what it's going to look for. So if you intend for there to be some kind of filtering where it's, say, going to not report certain boring kinds of things, or things you're not sure are a problem. And that's going to imply a totally different sets of filters and actions and extra checks that you can put in, versus if it was a purely mechanical process.

Jimmy:
So, the work of software design is in translating this high level goal of reporting security problems, down to low level details, like pointer scan for this, check that — and your high level description of this piece of software. Like it takes some Docker container and reports a security problem. That may never change.

Jimmy:
And then for each component, you might have intermediate level descriptions. And it's possible those intermediate descriptions will change as well. But the way you turn those into the actual details will change all the time.

Adam:
So, how would I use my knowledge of software design to build it in a way in which it was not likely to break in the future? I know that's a very open ended question, but-

Jimmy:
I teach my students something called the RAD process, which is actually a bit of a forced acronym, because the actual elements are ratchets, openness and, and assumptions. So, the best I came up with was: reduce assumptions, add openness, diminish complexity ratchets.

Adam:
Reduce assumptions ...

Jimmy:
So, the thing we'll talk about right now is assumptions. Something I teach a lot is about ... So, software has to be modular in order for it to be feasible to write. And that means that you can look at it as a single piece of code out of context, and talk about what it means, and what must be true about the rest of the code in the system for this to work.

Jimmy:
So for instance, every ... Say you have a point class, and you say point.X, that is already placing an assumption on the system, namely that this is a type that has a field called X, which is an integer. It's a very weak assumption. But those are the kind of assumptions you have to learn to identify.

Jimmy:
So, say you're writing your Docker container scanner, and you can look it as a random 10 line snippets, and try to state in plain English, what this is doing? Because that forces you to say something which is closer to the intention than to the details. And you have to think about how can you look at this 10 lines of code, and through whatever configurations, convince yourself, this does what you want it to do. And then those are all the assumptions that it's placing on the rest of the system.

Jimmy:
And obviously some of the assumptions have to be there. Like if I didn't want to do anything with the points, you need to know it has an X and a Y, but I don't need to know that it's stored as a tuple with X first and Y second. That's an incidental assumption. And so those are things that you learn to identify and eliminate.

Adam:
So, you look for assumptions, and then some of them will be not important to the intention of the software?

Jimmy:
Yeah.

Adam:
And those are the design mistakes, would you say?

Jimmy:
Not necessarily mistakes. This is the process you can take, you can take arbitrarily far. But those are the things that you want to keep localized. So, I'm going to run with the point example again because it's easy to talk about for everyone, even though it's kind of stupid, which is: say you have an application which has passed around X, Y points everywhere.

Jimmy:
One day, you realize that you want part of your system to support both 2D and 3D. So, sometimes these points are X, Y, and sometimes are X, Y, Z. So, the parts that actually needs to be 2D stuff, it's fine if they're taking an X, Y pair. But then you'll have ten thousand other parts of your system, like things you could have saving stuff for passing it around. And if you're actually taking them in this X, Y pair, like when you're taking a parameter you write (X, Y), that is spreading this assumption unnecessarily. And so now, when you go to X, Y, Z, that's when you have to change ten thousand places, because this assumption leaked to all this places where it didn't have to.

Jimmy:
So more realistic as an example, of this kind of phenomenon of an assumption leaking... One of my students works at a large public company, which I won't name, other than to say that I've used their software, and a lot of people listening to this have as well. They have a C++ application, and they're just trying to change the representation of strings. Long ago they could have wrapped ... They could put the string in some, behind some abstract interface. And it's C++, so there'd be zero cost of doing so at runtime, but they didn't. So now his team is putting in a hundred man hours into refactoring their list of strings, and there are several other hundred teams like his.

Adam:
Yeah, I like this point example, because the point could be in 2D or 3D space, and you're saying we can write the software so that we ... Only in the places where we actually need to reference these coordinates of the points, know about the internals of it. And I watched this talk that you did at Strange Loop about ... It was called "You are a Program Synthesizer" and it did talk a little bit about this abstraction barrier. So I was wondering what is program synthesis, and how does it relate?

Jimmy:
Yeah, so program synthesis is broadly defined as program that write programs, so, fun anecdotes: Alan Perlis, computing pioneer, he had the quote that anytime a programmer says, "I should just be able to say what I want, and the computer should give me the program," you should give that guy a lollipop. So, in the intro to program synthesis class on the first day, we give the students lollipops.

Jimmy:
Most of program synthesis — I mean, a writing program in general is hard, so most of it is about specific kinds of programs. So like, every PL conference these days, you'll see a synthesis track, and the paper's like "Synthesizing a Web Crawler from a Demonstration," "Synthesizing SQL Queries from Natural Language," "Synthesizing High Performance Container Libraries from a Normal Container Library [plus a lot of annotations]."

Adam:
It seems like writing programs is difficult. Writing programs that write programs must be quite challenging.

Jimmy:
Yeah. It's its own specialty. So, the way that I see that it, that synthesis-influenced software design, is: when you're writing a program, you want to make it easy to read. If you're designing an API, you want it to be easy for humans to use it to write new programs. How do you determine that? Can you peer into people's head?

Jimmy:
So yeah, you can go down the path of psychology of programming. It's very hard, and I don't know how far anyone's gone...but there are certain things about writing programs and looking at code, understanding what it means, which are not only about psychology, but are more fundamental to the code itself.

Jimmy:
And if there is only one..... you can prove, there's only one algorithm for determining whether a piece of code can ever return to a number greater than a hundred, you can show there's only one algorithm for that, which is not the case, but suppose that you could, then you know that human reading that would have to, in their head, be running something similar to that algorithm. And when you can say, like this is what a program means, and these are the only way to employ that meaning, to compose them and get what these two things together mean.

Jimmy:
So, we can learn about algorithms for writing code and reasoning about code, and use that as a proxy for how humans write code and reason out code. And that gives a lot of insights into the nature of programming and software design.

Adam:
What type of insights does it give us? That makes sense to me. Like at a high level, you're saying, "Oh, if we're working on writing programs that write programs, we have to come up with some understanding of program generation." So what insights have you found?

Jimmy:
So, some of the ones have to do with the search space. So, you do things this way, and you've restricted the amount of information available. Now, there are only, like, three operations that you can do in order to put these together and get results. Whereas, if you don't have this abstraction barrier, and you at any point have the freedom to go down to the raw bits, the raw characters of the string, and do all this stuff, then you have a lot more possibilities. So that's something that helps both with verification and with generation.

Adam:
In my mind it seems related to just extensive use of types. Like when people say, "Make invalid states unrepresentable," is that a similar concept?

Jimmy:
So, every type system designer, at least the kinds of types designed from the software engineering side as opposed to the pure type theory side, so, if you read a paper called, you know, like "A Type System for Concurrent blah, blah, blah," "Type System for Guaranteeing that Database Accesses have Correct Permissions," we let any of those sort of papers ... People who write them have come up with them by studying the domains, studying the kinds of programs people want to write, and usually determining something fundamental about the nature of the problem.

Adam:
Let's say that I want to get better at building software. How do I even know if I'm getting better?

Jimmy:
Say you're playing an instrument, how do you know if you're getting better at the instrument? You have to really cultivate an ear, so you have to be able to listen to yourself in slow motion and notice the little things, listen to masterful recordings. First you'll have a teacher correct you, and eventually you want to correct yourself.

Jimmy:
So, the same thing goes for any skill you're trying to cultivate to high level, and in software design I teach people all about ... Earlier, we talked about the process of turning your intentions, your high level intentions into code. And then there's also the other direction of looking at code. Discovering what facts it implies about a certain point in that program, and then what those facts apply about higher level pieces of data, and eventually all the way up to abstract statements about what the program is doing and what do you want to happen in the world.

Jimmy:
So all those extra layers between these high level intentions and the raw code, it's what I call the logical layer. I used to use the term the third level, but I decided it wasn't descriptive enough. So, I'd say that's learning to read this larger level. It's kind of like being able to detect pitched music, in that everything of software design requires seeing it.

Jimmy:
So every programmer beyond intro can look at some code, and then tell you in plain English what it's doing, at least if they wrote the code. Well, there's all these extra details about being able to say what has to be true at this point, what assumption is it making about the rest of code? Those are the extra ... What else can we change about the system that would make this not work? Those are all the extra details of what you see in the complexity of your code.

Adam:
So what is the logical level of your code or of your design? How would you phrase it? There's the logical level part of your code?

Jimmy:
No, and that's very much not, because for a given piece of code, you can give it many meanings, and they might be very different meanings, like do we say if X is greater than 65, is that a retirement age check? Or is it checking for if something's an ASCII letter? So, very different meanings ascribed to the same code. Or they might be slightly different meanings.

Jimmy:
So, say I have a robot, you know in one arm it makes coffee, and the other arm it opens my mailbox. Well, it does one of those half a second before the other. Is that part of the behavior, or is it just coincidence? So, whether you say it's part of behavior or not, those are two similar but different meanings given to the same concrete behavior.

Jimmy:
So, because there's this many-to-many relation between implementation and interface, you know a single interface can have many implementations — that people get, though people are less keenly aware that a single implementation may correspond to many interfaces. Because of that, it is, in general, impossible to determine the interface from by looking at the code.

Jimmy:
And when I say the interface, I don't mean like, in Java, you say, "Oh, I have a list, and it has an interface.... is the five methods." I mean actual facts, actual logical facts of the behavior. Like if I put something in and then get from that index, I get the thing I just put in. That's a natural fact about all possible sequences of calls.

Adam:
So, the logical level, it's above the code. It's interesting. It makes me think of sometimes if you work on an old software system, and you need to make changes to it, sometimes you have to do something like software anthropology where you have to interview the people who worked on it.

Jimmy:
Software anthropology is software archaeology. You can Google software archaeology, you'll get a ton of hits. And it's because this process about how the high level intentions are turned into high level design, and then to low level design, and then to the code....those processes are either lost or at best living in their head, or in a place you can't find in documentation. And that's what you're extracting.

Jimmy:
So, that's part of the logical level. There's couple of literary places this concept has appeared. So say you put the bits in something, what is flipping a bit doing? The most obvious thing, most concrete appearance, say, I changed a 1 to a 0 or 0 to 1, but then that has a meaning. It's like, "Oh, I changed the permissions of this file." Or I set a fact that I've already visited this square, I've already done this thing. So that's an actual correspondence, which it puts on top of that raw physical action.

Jimmy:
So this is discussed in Gödel, Escher, Bach by Douglas Hofstadter, as...we halve these numbers, how has changing this number done anything? It's because there's an isomorphism between these numbers and something more meaningful. And it's also talked about in "The Art of Motorcycle Maintenance," where you have an action like: I'm turning this screw, and that's what he calls the romantic view. In the classical view, it's that the screw is part of this engine, which is part of this drive train, which blah, blah, blah. And you have these extra layers built in your head of how these parts of the motorcycle link together, and when you're turning a screw you're really acting on those mental layers. And so the same goes for when you write your everyday low-level operations in code.

Adam:
So, is the logical layer synonymous with your earlier comments about intention? Like is the logical layer the intention of the piece of code?

Jimmy:
I'd say the intention is part of the logical layer. Really, when I talk about logical layer, it's kind of a catch-all term for a bunch of related concepts about all these things that code is doing, all these ways describe it's meaning, which are not directly apparent in the code itself. So, when I talk about the logical layer versus the code layer, it's really more of a rhetorical device. It can help stop people from talking past each other.

Jimmy:
So one person says, "I made ... I'm looking at your code, this variable is global, make it not global, that's bad." The person says, "Well, it will do the same thing, whether it's global or not, the way it's written. What can go wrong by leaving it global?" The Person A, maybe they'll just say, "Oh well it's bad." Or they'll concoct some scenario in which something actually bad happens because that person said, "We're not going to do that." They could adopt another scenario and says, "I can do that," and so on.

Jimmy:
And it's because one person is talking about the code as it is now in concrete implementations, and the other is talking about all possible modifications to code, and that's ... They're quantifying over all changes to code. And that's a higher level logical statement. Not very high level, but still ... What all of the things in the logical layer have in common is that they involve quantifying over all possible changes to code. So, a logical statement ... I run x = 1, and then X equals 1 at this point, or X is equivalent to the ID of the stdin or stdout, whichever one, or X is a positive number.

Jimmy:
Those are all logical statements, but they all correspond to infinitely many pieces of code, and those encapsulate the facts need to be true at that point for the code after that to run. So even very low level things, like I set X equal to 1 and then X equals 1. That is talking about all possible changes to code, all possible changes that preserve this fact.

Adam:
Interesting. So, do you think that as a profession, software developers ... Do we put too much focus on reviewing the code and not enough on looking at what these intentions are at the logical level?

Jimmy:
Something I write about in my booklet, the "Seven Mistakes that Cause Fragile Code" is that more junior developers will tend to review things like white space and variable names. Or even a little bit better, maybe they'll say, "You should use this function instead of that function." Whereas more senior developers are more likely to tolerate small variations in that. They're more likely to look much harder at what assumptions are being made by a piece of code.

Jimmy:
It's like this thing only works if you call this thing in the exact point of the program. That is not a stable guarantee. "Stable guarantee": so, that's a word, when I hear it, I know I'm talking to someone senior or becoming senior. Or they might say, "This function takes five parameters, and chances are, we're going to add this thing, and then we'll take seven parameters, and we'll take eight parameters." You need to generalize it somehow. Those are the issues more focused on by a senior engineer doing good review.

Jimmy:
And you know that all of those are not about the current code, but possible changes. It's not a stable guarantee that A happened before B, you make it so that B still works, even if A is called after B. That is about a possible change in the program.

Adam:
So, we should focus on the interfaces, is that a way you might summarize that?

Jimmy:
Yeah, I would say good code review focuses very hard on the interfaces.

Adam:
Yeah. So, I read your pamphlet "Seven Mistakes that Make Fragile Code." I think this one you're discussing, you called "Don't Overthink the Interior." So, I guess the idea is that maybe the method body is less important than actual inputs and outputs?

Jimmy:
Yes.

Adam:
And you also had this concept called "Refactoring is Not Boxing." What does that mean?

Jimmy:
So, one of my friends puts it nicely. Refactoring means changing the factor graph of your code. So, what is not refactoring? If you have a mess on your floor, then there is throwing stuff into a box where it stands, or there's actually figuring out what should go where. And some refactoring that I've seen is very much in this line of boxing.

Jimmy:
So, I was actually discussing this the other week on ... Actually no, just like yesterday on Twitter. The common thing that people do, which is not refactoring, but they call refactoring, goes by the name anti-unification. Not many people know the word anti-unification, but a lot of people do it without knowing that term.

Jimmy:
Anti-unification means, say, you have two piece of code that's different in only one place or two places. So, you wrap those into a function, and you make those two place difference into a parameter. That might be a very good operation, but what does that function mean? Can you give it a clear description independent, other than "I just found a common pattern between these two things?"

Jimmy:
If you can't, then there's really is no way to understand what the thing does, other than, "When I fill in these two parameters, it does this. When I fill in those two parameters, it does that." So that is boxing. You made your mess smaller, but the complexity is still there.

Adam:
So, some people would say, "Don't repeat yourself." It's like the golden rule of software creation. So, if I have two things that are repeated, except they vary on this one parameter, that actually I'm improving the code by boxing those up. Do you?

Jimmy:
Yeah, and that's a very shallow metric, because it's very easy to write a program that does exactly that. And I've been talking about how much software design is about these high level facts about, all changes to code, and it just depends on the global intentions, on the high level intentions, and how all of that stuff is not directly derivable from the code because of this many-to-many relationship between design and implementation.

Jimmy:
And so, if you think you can improve the design of code by doing something that trivial to automate, then that's a red flag. So, "Don't repeat yourself," is really like a simplified version of the real lesson, which is to not repeat concepts. And that will cause you to not repeat code.

Jimmy:
But if you have two different concepts or one concept that appears in two ways, but you very cleverly smush these two ways together, then you still need to understand them independently. And you haven't really improved anything, and you've quite possibly made things worse, because now if you have more indirection your code, you'll need to actually jump around between method definitions to figure out what's going on. If you can't actually give a clean description of what this does, it doesn't require you look at these methods.

Adam:
So, would I be able to tell that just by language usage? Is there a word for these two things put together? Is that a valid case? Or is that ... That sounds shallow to me.

Jimmy:
If I have created a function for these two things, can you tell if it's a good thing if there's a simple way to describe it?

Adam:
Yeah.

Jimmy:
It's not a foolproof indicator, but I think it's a very strong one, in that English has already evolved to try to hide details from things, and we already have a strong intuition about it. So why not reuse it? And I think eventually you should learn how to talk logically about, "Can I give a compact, but precise, description of what this thing does," but using some high level concepts, each of which have their own formal definitions, but I don't have to care about them right now."

Jimmy:
That's the real thing that goes on when you try to formally reason about software, and I think we should turn into our intuition as well. But when you're bootstrapping that process, you can approximate it, just by using the machinery you already have in your brain for using English to abstract details.

Adam:
Interesting. So, actually whether there's a good name for it can be a great guide?

Jimmy:
Yeah. In my review of "The Philosophy of Software Design" by John Ousterhout, I think I highlighted something quite close to that as the best line of the book. It as a line like, "If you're having trouble naming a concept, then maybe the concept's not well defined."

Adam:
Yeah, that's interesting. I mean, the very short version of some domain-driven design course I took a long, long time ago, it was like find all your nouns, and those are the important things. I guess it seemed trivial, but I guess there's some deep truth there.

Jimmy:
Yeah. So, when I took my first computer science class in high school, the teacher had us do something like that, and I thought it was silly, and I ignored it for many years. And now I'm coming back to something close to that, except you're trying to think a little bit harder about "What is your concept," rather than just "What are the nouns?" So, some of your concepts might be verbs, and some of them might be groups of nouns, and some of the nouns might be mere details, but it has a lot in common with that very simple trivial process.

Adam:
Interesting. Yeah. I mean, I guess ultimately there is some intention, as you were saying, some specification that we're working towards. And that is going to be expressed in English, I guess, or some natural language. So, there's some correspondence there.

Jimmy:
Well, I like to think of them as being expressed in formal logic, but you should be able to translate between English and logic.

Adam:
Yeah. Actually, I want to get back to that topic at some point, to ask you about formal proofs. But you know, we were talking about refactoring, boxing things, unboxing things, and I think it's a great vein. You also have this point about getting stuck in an old design, and it seems somewhat related. Could you expand on that point or explain it?

Jimmy:
A very common failure mode of junior engineers is: you're given this one piece of the system and told to implement it, but then the pieces that you're using don't really support what they're doing very well. So they have to hack around it. And in a coaching call just yesterday, someone was talking about how he's writing a trading system, and using some library that wanted information about a trade in a string. And he's like, "Oh, I guess I need to do everything in strings now." And you know, I rolled with it, but for a more experienced person, what I'd want to tell them is, "You should go to whoever made that library and tell them it's stupid and get them to change it."

Adam:
That's funny.

Jimmy:
Yeah. So being stuck in the rut of "Here's the APIs I'm using, here's libraries I'm given, I'm just going to do things the way they do, to do my problem, even it's very awkward," as opposed to, "I'm going to understand how the parts around me work, and then wherever it's not supporting what I want to do, see if I can find a better way that both supports what it already does what I want to do better, so that my code is cleaner, so the existing code is cleaner, or not more ugly, and then go out into other parts of the system and change that. Talk to people working with those, get the dependencies changed."

Jimmy:
It's usually not so easy, but part of going from being a junior to a senior is getting more comfortable doing that. Your responsibility goes from your piece to also understanding the pieces around it and getting pushback on designs, adjacent parts of the system.

Adam:
Another statement that I have written down here from you is, "Code quality is not about code." And so what does that mean? It's definitely catchy.

Jimmy:
Yeah. So, it's what I've been saying all along about the logical layer, in that a common thing that kind of goes along with things like the "Don't repeat yourself" principle is to attach yourself to very superficial measures of code complexity. And that can lead you to all kinds of things that don't change the logic of the program, but seem to make the code shorter. And usually that happens by taking a lot of things that used to be explicit and making them implicit.

Jimmy:
So, one of the worst cases I've seen of this was, actually, I think it was on the official MongoDB blog, where they showed one example using some kind of schema database. Like, look, you have this code that defines a schema, then this code that's maybe generated....there's still this code that accesses it and writes to it. Versus, like you have this MongoDB thing and just put stuff into our giant hash table, therefore it's shorter and has less technical debt and blah, blah, blah. And no, it's, and it's simpler ... And no, it's not simpler. It doesn't have less technical debt. You've just taken this explicit schema and made it implicit in the way that you're using this giant MongoDB hash table thing.

Adam:
So I totally agree, by the way. Yeah. Like we've thrown out our schema. Now it's just implied knowledge that has to travel around, and it's not enforced-

Jimmy:
But by a very superficial read, the code has become simpler. But the logic has not changed.

Adam:
So, we have to understand the code that's not there. That sounds very difficult.

Jimmy:
I actually like the way you're putting it. And in fact, there is a concept in verification, which I think I want to talk more about a future blog post called "ghost code," which is ... And it's kind of like what I was saying earlier when you're turning a screw on the motorcycle, but really you're, in fact, modifying this abstract concept of a drive train. Ghost code is basically: you have some extra variables that exist in your head or in your proof system, but not the actual executable code, which is like a variable for the drive train. And then when you turn the screw, you'll find the code that says "Turn the screw" and you put extra code next to it saying "And correspondingly modify the drive train in this way." So, you say the code that's not there, but that's actually a very serious concept studied and programming language theory.

Adam:
Yeah. So in the Mongo case, there's all this schema stuff that just implied. I can think of other cases, like type system. So, I was working on some code, and it determined if an element is greater or less than an under value, and it does this with an integer. So it returns, like if they're equal, it returns 0. If the first one's greater than the other one, it returns 1. And then if it's the reverse, it's negative 1, right?

Adam:
And so, there's something implied schema there, right? Where although it's an int, the only valid states are these three. Probably a problem, to be honest, that it's represented using an integer.

Jimmy:
Yes. And I have never been able to remember which is which.

Adam:
Yeah, totally, right? There's a ghost schema there as well, I guess, right?

Jimmy:
Yeah. So there's an extra constraint in the output data. From a verification perspective, you'd say it's not only returning a 1, it's also returning a proof that if this thing returns 1, then the left thing is greater than the right thing. And so, in your code, when you write ... When you're checking it for the 1, you have to make this corresponding logical transformation that because it entered the branch where this returning value is 1...to make the extra logical inference that because it returned 1, therefore X is greater than Y.

Adam:
Yeah. And it sounds, to me, like this logical level is actually the proof level that we aren't writing. Is that true?

Jimmy:
Yeah, I'd say so. And in fact, a lot of stuff I'm saying, you go from this returns a 1 to this extra step of because it returns a 1, I apply some fact about this function, therefore X is greater than Y. If you were to do a formal proof in a proof system like Coq or Isabelle or Lean, that would be an explicit step that you'd have to type in.

Jimmy:
Something that I've been interested in a while is trying to see if there is a way where I can teach ordinary programmers to do this kind of thing without having to learn Coq, because you either learn Coq or Isabelle — actually, I haven't used Isabelle, it might be better; I think there's some reason to believe it. If you learn Coq, then you have to do all this very persnickety stuff that has absolutely no relevance to normal programming, and for all these hours and hours and hours, and...If you've ever gotten really sucked into a programming problem, and then find yourself addicted, and then it's 6:00 AM, Coq is three times better at doing that to you. It just makes hours disappear, because you get this constant feedback from the system. But it is a time sink.

Jimmy:
So, you're doing all these things with no relevance to normal programming, but then you also ... You get this intuition of like, "Oh, no, from the fact that it's returned 1, there's an extra step to making the inference that X is greater than Y." That's an insight that I want to be able to give to people without having them have to spend all these hours in Coq. And I haven't figured out how to do that yet other than by having these conversations, and giving a lot of examples.

Adam:
I found this quote on your website. It says, "Since the age of 18, I've been obsessed with the problem of automating software maintenance." So, why is software maintenance important to you?

Jimmy:
Software maintenance is important because the world runs on software, and changing the world means changing the software.

Adam:
So, do you think that maintenance is overly hard at this juncture?

Jimmy:
Yeah. And... Most obvious place this happens is the Y2K problem. So you have who knows how many hundreds of billions being spent to change to, two-digit thing to four-digit thing or some generalized version. And that's a time when all the companies of the world had the same problem at once, but things like that happen all the time. Like earlier I mentioned the string thing with that public company. There's hundreds of hours per team to change their string representation. That kind of thing happens all the time, but it's usually not a case the entire world has the same problem the same time.

Adam:
So, how can we improve software maintenance?

Jimmy:
So, my life goal is to improve software maintenance by a factor of 100. And I think we can get the first 10x just by exactly what we're doing right now, telling developers to think about the logic of the code, to track what assumptions each piece of code is making, and then limit them.

Jimmy:
In the case of the string representation, is they could have very simply done well known super plus things to wrap the string in an abstract interface and this problem...when they wanted to change their representation, it would have taken five minutes instead of ten thousand hours. So averaging that with cases where it's not so dramatic, you maybe can get 10x. The other 10x is going to have to come from tools. So if we do have the ability to try to look at a code and infer what its design is even though there could be more than one answer, then we have to be able to make tools to do the same thing. And then from there be able to make tools that modify the design.

Adam:
So, do these tools exist? Or, I guess, your life's goal is to build them?

Jimmy:
To a limited extent, they do. One very exciting project I've been a little privy to is from Semantic Designs, which is a company in Texas that I spent a few months at a few years ago. They've been doing some work with a large chemical company to try to do design recovery of chemical plants.

Jimmy:
So, they have about a thousand chemical plants around the world. And they're running this very obsolete language. And so, they have all these controllers like, this sensor reads this, then flip the switch, blah, blah, blah. So, they had a high level macro language, which changed over time, which they wrote these in. Then they generated some low level code. Then they changed the low level code, which means the high level code is no longer relevant.

Jimmy:
Now it's 50 years later, and the hardware is failing at an increasing rate. So, we've got to switch these over before that happens. And now you have this giant gob of goo, and you need to figure out what it does.

Jimmy:
So, because you know that this low level code has been generated from these high level libraries, they change over time, and they're modified, you can do smart things to try to recover ... To try to do this clever decompilation, discover what high level operation things correspond to, and then translate it into our little form, then from there translate to Python. And they've been doing that. Then they're already ported ... I don't know what the number is. By now, it's probably over a hundred chemical plants.

Adam:
So, you work, you have a tool that ... I remember the specific line that said, "Programs that write programs that write programs." It took me a little bit to understand this, actually. You've built a tool that allows for cross language refactorings? Is that-

Jimmy:
Okay, so, you're mixing up several different things there. So, my research interests ... So, I was talking about tools like the thing used for the chemical plants. And my goal was to make software maintenance better. Part of that is making tools like that better and more common, tools like the chemical plant thingy.

Jimmy:
But there already are a lot of people in software engineering research who do know how to build those tools. But a reason why those aren't very common is that they're all very specific and hard to build. So for instance, if you had a program that did all its calculations in Fahrenheit, and you wanted to change the code to use Celsius internally, rather than Fahrenheit, that might be a very tough change. And even if you could pay a million dollars to an expert, or a hundred thousand dollars to an expert, and get them to write a tool that does it, you probably wouldn't.

Jimmy:
Even if there's a hundred people, or a thousand people also in the world doing the same thing, you couldn't find them. And their tool would probably only be for one language, and you'd probably have to rewrite it even from Scala to Java, even for similar language. So I think that by reducing the cost and making these tools, we can see a much larger number of these advanced whole program refactoring things built. So that is my goal in research.

Jimmy:
And so, I'm interested in finding better ways to build these kinds of tools, to make them more general and to automate parts of building them. And since I'm interested in automating parts of building these tools, that has the cheeky thing "programs that write programs that write programs."

Adam:
And this is...Cubix is your framework?

Jimmy:
Yeah. So, Cubix is, it's a stretch, to put Cubix literally under the tagline "programs that write programs or write programs." But the paper I just submitted to POPL is on something called Mandate, which is much more fitting with that moniker.

Adam:
Cool. So what is the end goal here in terms of software maintenance?

Jimmy:
The big dream is that if you wanted to write your thing, which takes a program that does stuff with Fahrenheit, changes its internal code to do stuff in Celsius, then I should be able to describe it at a pretty high level, and then get that tool, and then I should be able to only change a few lines, and they get the tool for Java, and then get the tool for C and they get the tool for Rust.

Adam:
That makes sense. So, the reason it's "programs that write programs the write programs" is the Top level program that you're working on could be fed a program about how to change program in a loose sense. Change Fahrenheit to Celsius, and then it will generate programs for a myriad of languages that can do that source transformation to those languages.

Adam:
So, I understand you're doing a training course on some of the principles we've gone over today.

Jimmy:
Yes. My have passion and livelihood is on teaching a lot of the things I talked with Adam earlier about how to see the logic of your programs and write things in ways that are easy to read and also need less time debugging, and they'll be easier to add features to, except in a much more concrete way with a lot more examples, a lot of practice with very good feedback on the practice. That's my advanced software design web course. The next run is starting on August 23rd, I'll be taking 15 people.

Adam:
One other question I wanted to ask you. I've been looking into you, preparing for this interview. So, you currently are at MIT, working on your Ph. D., to my understanding?

Jimmy:
Yes.

Adam:
But according to the Internet, Peter Thiel paid you to not go to university.

Jimmy:
Yep.

Adam:
So, could you describe the program you were a part of and how it was?

Jimmy:
Yes. So, I was part of the Thiel...."20 Under 20" Thiel Fellowship. I entered in 2012, and got a hundred thousand dollars over two years. So, at the time I had been a junior at Carnegie Mellon, but I did not have a senior year. I just left. Fortunately, I still graduated with two degrees, because I had worked really hard in those three years. I was planning to do a master's, but I dropped out.

Jimmy:
So, for two years, I worked ... For one year and a half, I worked on a company, trying to write a program that fixed bugs in other programs, and I talked more about that elsewhere, as such, in my interview with Future of Coding, but that wound up not going so well. Both because the technology was still in its infancy, because I was not an expert in the technology at the time, I was learning as I went, and also because I learned the hard way that a good product idea is not a good business idea. Whoever tells you that good business ideas are a dime a dozen, it's totally wrong.

Adam:
Oh, interesting. How so?

Jimmy:
Good product ideas are dime a dozen. Good product ideas, that...where you can find the audience, where you can convince the buyer the value, that are protected against competition, how we can market it, where you can hire people to work on it, where you can sell add-ons and grow. When you add in all those other factors needed to turn a product into business idea, they become very few and far between.

Adam:
That's interesting because, you've kind of carved out a distinction for me that I haven't thought of, which is that people's business ideas are often just, "Hey, I whatever. I built a better mouse trap or something," but that's not really a business idea. That's just a product. So, did you find the program ... Your two years useful? Was it a good experience?

Jimmy:
Oh yeah. Very. So, it certainly laid the foundations for what I'm doing now, and that is was from a process building this company that I gained the insight into realizing I need to go one middle level up into making these kinds of tools easier to build. But also, managed to become very connected to the industrial side of things, which is why I'm able to do what I do now. And my teachings about software design are very much bridging practice and theory.

Adam:
Yeah. You don't see many people who are working in academia also doing kind of a nuts and bolts advanced software design course.

Jimmy:
You don't see many people anywhere doing in advanced software design courses. In fact, there are zero books written today that I've found — and I've looked — that I would call an advanced software design book. There's a large number of beginner books in software design, and a small handful of intermediate-level ones, but none that I would call advanced. That's because no one knows how to teach it, and my work has changed that.

Adam:
That's interesting. You had this post called ... On your blog called ... Oh, it was about Benjamin Franklin method for programming books.

Jimmy:
Yes.

Adam:
Could you explain that?

Jimmy:
Yeah. It's the whole genre of programming books, as distinguished from computer science books. Usually, they are like "Postgres 7" or "Writing Games on iOS" or "Intro to Rails," and a core feature of them is they have a lot of code snippets. They're walking you through these examples of what to do with these code snippets. Do you type them in? Do you download them from the book's website? How do you learn from these code snippets?

Jimmy:
So, from multiple sources, I learned about the Benjamin Franklin method, which is... he wrote in his autobiography, this method he had for being better at writing without a teacher. He found these articles that he liked that were totally better than his. And he didn't understand how he could produce articles like them. So, he would take notes in these articles, compressed them into like a few sentences. A few minutes later, he would come back, look at his descriptions and try to reproduce an article like it from the descriptions.

Jimmy:
And then that would give him a laser eye to focus on all the little tricks, the rhetorical flares that the professional writers did that made their article better than his. And so he learned. And that's what's a lot of artists in other fields do. So, visual artists might try to copy a painting, and in doing so will discover all the techniques used by the original painter.

Jimmy:
And so, I realized that a modification in that idea can be applied very easily to reading programming books. The most faithful version would be to try to write a description of what this program does that you have the snippet for, come back a few days later, and then try to reproduce it. And you can do that. The feedback cycle is kind of slow, and you might not actually want to care about reproducing the exact same program. It's not as important to wait a few days, forget everything you knew before.

Jimmy:
You can get a very cheap version of this, which is very effective, just by looking at the page, flipping over, looking away, and then trying to write the program. And even though that might only add a few seconds versus copying from the page, in those few seconds, you'll forget so much about the syntax, unless you can compress it into a better form. And that gives you more learning.

Adam:
Yeah, I like this method. I haven't tried it, but I think that we've read the same book, because I saw a reference to Anders Ericcsson, I think, the deliberate practice guy.

Jimmy:
Yeah. So, my summer in Texas, when I was at Semantic Designs. I had a car for the first time since I left California, so I was listening to a lot of audio books. And so within a short period of time, I read both Anders Ericsson's "Peak," and also Josh Waitzkin's "The Art of Learning," both incredible books. And also "Mindsight" by Daniel Siegel. And I think all three of those, or at least two of the three, all mentioned this technique.

Adam:
Yeah, I love it. Well, so I haven't tried it, but when I saw you reference it, I was like, "That totally makes sense." So Anders Ericsson, right, is the deliberate practice guy?

Jimmy:
Yes.

Adam:
We touched on this a little before. He spent a lot of time studying how people become experts. So, he has a number of techniques in his book for working in a field that's less defined, right? Because he studied chess, right? And in chess, there's very defined progression, and I think violin as well.

Adam:
But, as you were saying, okay, here we are, software development. There's no advanced books, right? Everything's surface level, how to learn X. So, another technique that he had that I liked was ... He called it the TOPGUN method from the TOPGUN, the school. Do you remember this technique?

Jimmy:
I remember him talking about TOPGUN Academy. I don't remember the generalized version.

Adam:
So, the idea was ... I think it was the Navy pilots were getting basically schooled by the Russians at some point. And if I'm murder the story, you know, write me a letter, listeners. But this is my approximation. So, they knew they were failing at whatever, dog fighting, right? But they didn't know how to get better. So, they started this TOPGUN school where they just fought against each other. So, they didn't have a teacher, but they managed to, as a group, teach themselves how to be better by competing against each other.

Adam:
I've often thought about this in some ways to do with how a team could get better at software design by just critiquing each other. That was basically their approach, except much more adversarial, like they're pretending to shoot each other down. But they managed to learn by feedback.

Jimmy:
I think it kind of boils down to a bigger thing, which is very often the way to get better is to recognize the difference between practice and performance, between just doing something, versus trying to get better. And so, all too often, in these kinds of fields, the way to get ahead is simply to try to get ahead. Like other people are just programming but not really deliberately focusing on what they did and what they could be doing better, then that's all you need.

Jimmy:
I want to share one story that's that's similar to the TOPGUN academy, but more programming related. An old story of programming folklore, but not super well known. And that is a story of the IBM black team. I think this is sixties, seventies, IBM had many teams with their testers. So, they decided just to take the top 10 percent of testers and put them on their own team.

Jimmy:
And they thought, "Okay, maybe this team will be 25 percent better than the other testing teams." And that's not what happened. They actually became 10x better than the other testing teams, in that now you have a whole group of people who are more motivated, and they formed their personalities around being good at breaking other people's software, and they fed off each other. So, they would learn all these things, share them with each other at lunch, and pretty soon they took to dressing all black, and then wearing mustaches that they would twirl as they try to break someone's piece of software.

Adam:
That's awesome. So, you mentioned that you think that practice is different than performance. How so?

Jimmy:
This is something which is talked about, by K. Anders Ericsson a lot. So, you know something that you're good at, like maybe running, or sailing, or playing an instrument, or certain kinds of programming. And you get into this zone, and it feels pleasurable, and it's really comfortable, pleased with how much stuff you can do. But that is performance. That is not practice.

Jimmy:
You know you're growing because you're pushing yourself to your limits, because you're stretching your brain till it's uncomfortable.

Adam:
When we talk about software design, I think a standard idea is you just have to build a bunch of software, right? If you got enough experience under your belt, then you'll be good at software design.

Jimmy:
Yeah.

Adam:
What do you think of that?

Jimmy:
If you're writing software, you can't help but occasionally doing things you've never done before, but there's nothing deliberate about that. You've heard the line forever, "Practice makes perfect." Maybe you've heard the more modern line, "Perfect practice makes perfect." Practice does not make perfect. Perfect practice makes perfect.

Jimmy:
We can find issue with that line as well, but it's closer to the truth. And so, similar in software. Just doing it, does not necessarily make you better if you're doing the same things over and over. You need to do it better. Do things you haven't done before.

Adam:
And what does practice look like? If I want to know how to be better at designing software systems, how do I practice?

Jimmy:
So it's really a few counterintuitive things. One is I teach people about ways that they can lock down their design, not only making valid states on representable, but also make it so valid states can only be represented in one way. No junk, no fluff.

Jimmy:
So, you learn all these techniques, it becomes much harder to write the wrong program, easier to write the right program. And then they go and take it too far, and try to fill up all these kind of really elaborate type stems that force you to do things a very certain way or otherwise everything will break. They write 20 thousand different types. And they're taking it too far? Yeah, totally. And I say go for it. Everyone needs to go through that phase of taking it too far before they learn how to do it naturally.

Adam:
Interesting. So, when I learn something, I should take that thing, and really see how far I can stretch that piece of that concept.

Jimmy:
Not necessarily, but you have to run towards things that are harder than what you've done before, not run away from them. That doesn't always mean you check that code into production, but you have to experience it.

Adam:
Yeah. I think that I could learn a lot about how databases work by building my own toy database system, but I should never actually use it. It's more of a learning experience.

Jimmy:
And that's the case, like....even if there are specific tasks where you can write ... Where you think you can write something that's better than Postgres for one very specific purpose, and maybe your program can be better if you do that; it's very unlikely you will encounter a case where doing that is actually a trade-off. And so you might never learn....you might never get a chance to exercise that skill in the wild.

Jimmy:
That's what I'm saying. The practice, not the performance. So, you can't always lean on saying, "If I do this enough, then I'll get better at every point I want to get better at." You have to aside time, saying, I'm going to do this thing. This is practice. This is not going to produce an artifact I can use. It's just to help me get better. And I'm okay with that."

Adam:
So, would you advocate building a web scraper over and over and over again? Like different ways, different languages, abusing type systems, having no type systems, doing it lazily, doing it like ... I dunno. Is that a valid way to practice?

Jimmy:
You will certainly learn more doing that than if you just wrote different web scrapers the same way over and over.

Adam:
I'm just, I'm grasping at how we can practice. I feel like, okay, if you're a baseball player, it's pretty clear the difference between practice and a game. But I don't know how clear it is as a software developer.

Jimmy:
Yeah, I don't either. But the beauty is that because there's such a weak concept of practice, that means you do anything...you find any of these things that are questionable practice, even if it's questionable how good they are...you're still doing better than everyone else.

Adam:
Because nobody else practices. They're all just playing the games. So any practice-

Jimmy:
Yes. And this is why I say that while there are a lot of good programmers alive today, a lot of master programmers, but I don't think anyone alive today deserves the title "Grandmaster Programmer."

Adam:
Interesting. Yeah. I wish there was, right? Like if you wanted to become a chess prodigy, I think there's a certain ladder that you need to follow, right? At some point, it involves working with a grand master and spending all of your waking hours doing things. But they know what those things are. I don't feel we do know what those things are. That's not a question. I'm just thinking out loud here. You got my gears turning.

Adam:
Well, I think that I went through basically every question I had for you at this juncture. So, this has been a lot of fun. So, thank you so much for joining me, Jimmy. It's been great.

Jimmy:
You too.

Adam:
That was the interview. I think Jimmy's concept of the logical level in software is one I'm going to keep thinking about for quite awhile. Programs that write programs that write programs. Jimmy is coming at things from a whole different level.

Adam:
If you're listening to this now, say hello. I'd love to hear what you thought of the episode. I know I could have asked more questions about program synthesis, but I didn't want to miss Jimmy's insights on software design.

Adam:
Thanks to everyone in the Slack channel as always, and thanks to those leaving comments on the website. Eli and Chris Buena in particular. The website comments go into a moderation queue if they have a link, but should otherwise appear right away, and if I miss some link that should have been in the show notes for an episode, dropping it via a blog comment is a great idea. Timmy did this for the Cory Doctorow episode, and that was great. Tim has been requesting some closure episodes, so I think I should probably get on that.

Adam:
Until next time. Thank you so much for listening.