The Practice is not the Performance: Why project-based learning fails

Last night, I encountered an old post by Zach Holman where he pushes the idea that traditional school-based CS is useless; project-based learning is the way to go. I’ve heard this idea repeatedly over the last 10 years, and know at least one person who’s started an education company with that premise.

I don’t want to debate the current way universities do things (I found my undergrad quite useful, thank you), but I do want to dispel the idea that everything would be better if only we switched to project-based learning. The opposite is closer to true. Project-based learning is not a superior new idea. Project-based learning is the most inefficient form of learning that still works.

To see why, we actually have to sit down and think about the learning process. I’ve learned a number of models of learning over the course of my teaching training, but the one I’ve found most useful is taught by the Center for Applied Rationality. It goes like this:

1. Break down a skill into small components.
2. Drill each component rapidly, with immediate feedback
3. Integrate the pieces together into larger components; drill these similarly

In that light, project-based learning definitely has something going for it: if you do a project, you will practice all the skills needed to do projects.

There’s something missing, though: it completely gives up on trying to think about what components it’s trying to teach.

Here’s how project-based learning might play out: After finishing his first two Android apps, Bob decides he wants to learn network programming, and generally get better at working on a software team. He and Alice pair up and decide to build a chat app for sending cat gifs, with a distributed replicated backend. They decide to make a spreadsheet of tasks and claim them. Alice also wants to learn network programming, and she swoops in and takes most of the server components; Bob gets left with most of the client work. Over the next three weeks, Bob spends a lot of time building the GUI for the app, which he already knows how to do, and only a couple hours implementing the client protocol. They start writing tests because the teacher said they had to. Bob has trouble writing the tests, and realizes a couple ways he could have made his code differently to make testing easier. He also discovers that two of his tests were too fragile, and needed to be changed when he updated the code.

It’s now one month later. What has he gotten from the experience? He's learned he needed to be more proactive about taking on tasks that challenge him and grow his skills. He’s learned a smidgeon about network programming, and a couple ideas about how to write better tests. These are good lessons, sure, but expensive. Isn’t there a way for Bob to learn more in that month?

False Temptations

What are the common arguments in favor of project-based learning? Here are two of the main ones.

Real skills

The first big reason for project-based learning is that it teaches real skills used in industry. Why do many schools teach much of their curriculum in Haskell, not in the Tiobe Top 10, or even SML or OCaml, not even in the top 50? Wouldn’t they serve their graduates better teaching Node and React?

The first counterargument is that industrial technologies come and go. Proponents acknowledge this, sure, but still call CS departments “out of date” for not following trends. What really drove home the futility of this argument for me was this essay by software-engineering pioneer Mary Shaw. Had she followed that advice in the 60s, she points out, her students would have spent their time studying JCL, the language used to schedule jobs on IBM mainframes.

The second and bigger counterargument: learning concepts is much more important than learning applications, and the best environment to learn a concept is rarely the one in industrial demand.

People often ask me what’s the best language to learn to study software design. I ask them what’s the best instrument to learn to study music theory. Everyone answers piano. In piano, you can see the chords in a way that you can’t in, say, trombone. We see something similar in other domains. In The Art of Learning, Josh Waitzkin recounts how, unlike others, he started studying chess at the fundamentals, in situations with few pieces on the board. He ultimately beat competitors who studied many times harder. In The Art of Game Design: A Book of Lenses, Jesse Schell advocates looking past modern video games and instead studying the concepts in board games, dice games, playground games.

So, for programming, we need to (1) figure out the core concepts to teach, and (2) pick languages that make the concepts readily available. C and Java --- indeed, all top languages until Swift arrived --- lack elegant ways of expressing the idea of a sum, a value that can be one of many alternatives. Thus, when explaining why adding a new alternative sometimes breaks existing code and sometimes doesn’t, I find myself having to explain using the clumsy manifestations as unions and subclasses. The first time I read Bob Harper’s explanation of why they chose SML for the CMU undergraduate curriculum, I thought he was just rationalizing snobbery. Now, I quite agree.

More like real jobs

The second big argument for project-based learning is that it more closely resembles what students will actually do on the job. This, in turn, is based on the idea that the best way to practice an activity is to do it.

This is false. “The only way to improve at X is to do it” is the advice you give when you actually have no idea how to improve. When you do know, you isolate subskills and drill them. Martial artists punch bags and do kata, and fight with extra constraints like no dodging. Musicians play scales, and practice a single measure over and over. Mathematicians rederive theorems from the book. And to condition a shot-putter, the best way is not to put weights in their hands and have them mimic the throwing motion, but rather to train the body’s ability to produce power, using squats and lots of heavy exercises that don’t even resemble shot-putting.¹

Drilling programming

So what am I advocating? I’m advocating that you actually think about what you’re trying to teach, and design drills for it. The first drills should focus on one thing at a time.

So, for Bob trying to learn network programming and the general software engineering skills of being on a team project, here’s my 5-minute attempt to come up with an alternative way to teach these skills:

1. Writing just the networking component of a larger system.
2. Being asked to write the test cases for a small program given to you. The program is deliberately designed to tempt you into all the classic mistakes of test-writing.
3. A simulation where you “coordinate” with the TAs to build something, committing pseudocode to version control. They troll you by deliberately misunderstanding things you say and seeing if their misunderstanding can go undetected.
4. You and several teammates are assigned a small project. You’re asked to divide it amongst yourselves into unrealistically small units of work, and write test cases for each other’s components. (Integrates the skills of 2 and 3.)

These total much less time than the chat app, and teach much more. If students find they’re not optimal for learning, I as the instructor have much more room for experimenting. And if the students do choose to do a full project afterwards, they’ll be much more prepared.

I don’t teach network programming and haven’t tested these specific ideas. But, in my specialty of software design and code quality, I use exercises built on similar principles all the time. So I may teach a client some of the pitfalls of a naive understanding of information hiding, and then show them code from a real project that has that problem and ask them how they’d solve it. Or I’ll ask them to give just the type definitions and method signatures for a Minesweeper game; if they violate any design principles, then I can give them a feature request they can’t handle, or show why the implementation will be prone to bugs.

Is it better than just assigning projects? That’s the wrong question to ask because project-based learning is incredibly easy to beat. My clients are mostly working professional software engineers; they’re already doing “project-based learning” every day. On my website, I claim.

Programmers learn by making bad design decisions, working on the codebase for a year, and then find themselves wishing they could go back and do things differently. I can give you that experience in an hour. 

Does this sound like a bold statement about my teaching prowess? It’s not. In fact, piano teachers put that claim to shame. You can spend hundreds of hours practicing a piece using too many muscles on every key press. If your body awareness isn’t great, you might not find out until your hand cramps up right before your performance. A couple seconds to catch that mistake, a couple minutes to tell you a story of that happening to others, and the piano teacher’s just saved you months.

(As an aside, this is why I believe in finding a competent private coach for every skill you really care about improving in.)

Replacing a traditional CS education with a “software engineering BFA,” like Spoelsky and Atwood suggest, is no longer a hypothetical exercise. We’ve tried it. And now dev bootcamps are going bankrupt. Instead of substituting for a traditional degree, recruiters are calling bootcamps jokes. Olin College of Engineering is famous for its project-based curriculum, but one student reports that she learned much more from traditional classes.

It’s time to stop looking for panaceas and shortcuts and realize that deliberate learning and deliberate practice --- as a separate activity from the everyday doing --- is the only way to mastery. As famed gymnastics coach Chris Sommer puts it, the fastest way to learn is to do things the slow way. Studying the fundamentals may seem like a distraction keeping you from getting your hands dirty making a Rails app using the Google Maps and Twilio APIs, but when you do get there, you’ll find there is less to learn if you’ve already compressed the knowledge into concepts.

Shameless Plug

My Advanced Software Design Web Course starts next week. It’s based on a lot of the learning principles I mentioned in this post, starting each concept with isolated drills and progressing to case studies from real software, and comes with personalized feedback from me on every assignment.

Disclaimer:

Acknowledgments

Thanks to Elliott Jin for comments on earlier drafts of this post.

1 When I was searching for a personal trainer, I asked about this to help screen candidates.

My favorite principle for code quality

Programming seems to become more about memorization every day, with advocates pushing for memorizing lists of design patterns and refactorings and the difference between “parameter coupling” and “invocation coupling.”

Forget all about that. There’s much more to gain by having general principles which apply to many situations. In this post, I’m going to show you one of my favorite software design principles in action, the Embedded Design principle. The Embedded Design principle, which I briefly introduced in a previous post, states that you should always code in a way that makes the design apparent. It’s like pixie dust in its ability to make code more readable, loosely coupled, and impervious to bugs. It’s also hard to grasp, because programmers rarely see a program’s design in first-class form. So, in this post, I’ll show you an example.

Conveniently, a principle can be used as a “machine” for generating examples that violate it. In software design, these examples always end up being code that looks reasonable to the untrained eye, but will spell doom and gloom — often in the form of hilarious bugs — for any team that tries to build on it. Last week, one of my coaching clients asked me how this principle applies to caching. I turned on my machine and it produced this:

The Problem

Meet Bob. Bob is an engineer at AwesomeSauce.com. He’s only been there a few months, but he has a project that will affect everyone: he’s going to create the website’s stats page! Now everyone can see just how engaged AwesomeSauce’s users are.

Bob starts coding. There are three data items, and each one will print either a computed or cached value. Bob ended up using one chunk of code for each data item.

public void displayStats() {
  if (lastCachedTime <= lastMidnight()) {
    numUsers = countUsers();
    lastCachedTime = Time.now();
    print(“Total Users: “ + numUsers);
  } else {
    print(“Total Users: “ + numUsers);
  }

  if (lastCachedTime <= lastMidnight()) {
    numArticles = countArticles();
    lastCachedTime = Time.now();
    print("Articles written: " + numArticles);
  } else {
    print("Articles written: " + numArticles);
  }

  if (lastCachedTime <= lastMidnight()) {
    numWords = countWords();
    lastCachedTime = Time.now();
    print("Words written: " + numWords);
  } else {
    print("Words written: " + numWords);
  }
}

So simple! As the stats page gets bigger, it will be easy to add more by repeating the pattern. Bob sends it to Charlie for review.

Now, one thing about Charlie is that he’s not me. You see, I’d probably flip out and tell Bob about all the bad things that will happen because of this code, and give him a lecture on the Embedded Design Principle and a link to this blog post (which conveniently already has this code). Charlie, however, just takes a sip of coconut water, and says “There’s a bug over there because you re-used lastCachedTime. Otherwise, looks good. Ship it!”

- if (lastCachedTime <= lastMidnight()) {
+ if (lastCachedTimeUsers <= lastMidnight()) {
     numUsers = countUsers();
-    lastCachedTime = Time.now();
+    lastCachedTimeUsers = Time.now();


< similar for articles and words >

It’s 6 months later. AwesomeSauce.com has been growing faster than the national debt. AwesomeSauce needs more frequent updates to the stats page to show this. Team lead Denise decides to double the refresh rate, and Charlie does it.

- if (lastCachedTimeUsers <= lastMidnight()) {
+ if (lastCachedTimeUsers <= lastMidnight() || lastCachedTimeUsers <= lastNoon()) {
    numUsers = countUsers();

Bob is displeased. They had previously decided all stats should refresh simultaneously; see how the mockup just has one “Last updated” time. “Oops,” says Charlie, and together they merge the if-statements to prevent a future mistake.

public void displayDashboard() {
  if (lastCachedTime <= lastMidnight() || lastCachedTime <= lastNoon()) {
    numUsers = countUsers();
    numArticles = countArticles();
    numWords = countWords();
    lastCachedTime = Time.now();
  }
  print(numContent);
  print(numArticles);
  print(numWords);
}

Another year passes. AwesomeSauce has been taking off like Bitcoin. Their stats page is now an information feast, with dozens of items. Time to start simplifying! Surveys show that no-one is actually wondering about the count of words, so they take it down:

- print(numWords);

AwesomeSauce has hit a jackpot! Writers are making 10,000 posts per day. One day, an AwesomeSauce engineer notices they have a big slowdown twice a day, at noon and midnight. After a full day of sleuthing, he discovers that it has something to do with how, at cache refresh time, the stats page is taking a full three minutes to load. With horror, he discovers the call to countWords(), still living, though its results are unused.

This was not merely an instance of “mistakes happen.” This was a heavy price paid from badly-designed code. Everything from the first bug to the site slowdown could have been prevented by structuring it better, and the way to structure it better is a simple application of the Embedded Design Principle.

Nipping it in the Bud

Standard disclaimer: When reading software design advice, always imagine the examples given are 10x longer. Overengineering is bad. But, if you’re not sure whether applying a technique to your code would be overengineering, error on the side of doing it. Abstract early.

Update, 9/14/2018: Some people attacked this, saying "abstract early" is terrible advice. This is a tangent to the rest of the post, and if I interpreted it the same way they did, then I'd certainly agree. But please read the linked Jessitron blog post to see what this is actually saying.

In the first version of the code, the computation, caching, and display of each statistic were all independent knobs, 3*N in total, where N is the number of stats to display. Merging the if-statements combined the N caching behaviors into one knob, bringing the total down to 2*N+1 knobs. Nonetheless, this still made it possible to compute a stat without displaying it. This was the rope that hung AwesomeSauce with the twice-daily slowdown.

This example was extreme, but ones like it are real. A static analysis researcher I know once visited eBay. She was shocked to discover that the checker that excited the engineers the most was one of the most shallow: detecting unused variables. Each unused computation eliminated was a database call saved, and a nibble taken out of server costs.

Conceptually, the set of stats and their caching behavior should be only N+1 knobs: each stat as a whole can be turned on or off, and the caching behavior can be changed. Achieving this will be a non-local change, and won’t be obvious by just thinking about what the code does, thinking at Level 2. Instead, we have to look at the design:

This diagram gives a rough picture of concepts used in the design of the program, showing how the concept of a dashboard stat is instantiated multiple times, and how the concept of each stat is built into the code. This diagram is far from complete; it omits, e.g.: the choice of the English language. But, with it, we can see several things.

The same caching behavior is implemented multiple times, which made the bug in Bob’s first version possible.

The code is indistinguishable from an alternate design where each stat has its own caching policy. This is another example of why design recovery is in general impossible.

And finally, each dashboard stat gets expressed in multiple independent lines of code, each its own “knob.” Hence, the slowdown that plagued AwesomeSauce.

All can be fixed — and the code rendered beautiful — if we take all those many arrows emanating from those boxes on the right, and reduce them down to one per box, one line of code per low-level concept.

This is the true version of the “Don’t Repeat Yourself” (DRY) principle.

Forms

In Zen and the Art of Motorcycle Maintenance, Robert Piersig illustrates these higher-level concepts, which he calls forms, in the world of mechanics. A motorcycle is a power assembly and a running assembly; the power assembly can be divided into the engine and the power-delivery assembly; and so on. A mechanic that’s testing the spark plugs or adjusting a tappet is really working on these higher-level forms. And yet you cannot walk into a motorcycle parts shop and ask for a power assembly: these concepts only exist in the mechanic’s head.

But we’re programmers, and we have tools that translate our thoughts into CPU instructions. To a certain extent, we can actually build these forms directly into our code. Let’s create a value representing a DashboardStat.

public interface DashboardStatComputation {
  public Object compute(); // in real code, use a more specific type
}

public class DashboardStat {
  private DashboardStatComputation computation;
  private String textLabel;
  // constructors and getters
}

In the design, we already had a concept of a dashboard stat: it’s a computation and a label. Now that concept is in the code too. We can go one level further, and also put the collection of dashboard stats into the code too:

public class Dashboard {
  private List<DashboardStat> stats;
  private Time lastComputedTime;
  private Map<DashboardStat, Object> curValues;
  // constructors and getters
  /** Recomputes all statistics if cache is stale */
  public Map<DashboardStat, Object> getCurrentValues() { … }
}

With these higher-level concepts turned into code, Bob’s original dashboard page can now be written declaratively. This code is just too simple to fail.

// Initialization (slightly simplified from actual Java)
dash = new Dashboard(list( new DashboardStat(countUsers,  “Total Users”)
                         , new DashboardStat(countArticles, “Articles written”)
                         , new DashboardStat(countWords,  “Words written”));

// Use
public void displayDashboard() {
  print(dash.getCurrentValues());
}

Beautiful.

More (or less) Embedded Design

You will never turn your program into a pure expression of the design, at least not unless you’re a descendant of Zohar Manna. You are always making tradeoffs on how far to go down this spectrum. Here are some other choices for applying the Embedded Design principle to this example:

• If this example refactoring was too much, there’s a simpler compromise. Don’t have a DashboardStat value; just have a DashboardData which hardcodes each of the computations. This is much less overhead than the way I did it above, but it still dictates that all stats should be computed in an all-or-nothing fashion. 
• Extract the check for whether to recompute into a shouldUpdate method. Now your decisions about cache timing are reified in a separate chunk of code. To go one step further, the caching policy could be parameterized. 
• Might you one day want to serve the website in multiple languages? Wrap each string in a call to an internationalization function. For now, this can just be the identity function, but doing so makes the choice of language explicit in your code, and marks all the text in your code which corresponds to a “translatable string” concept. (Another example of how, at the design level, two different identity functions are not identical.) 

Wrapping Up

Many programmers will get a sense that something was off about the first example. Every programmer I’ve shown this example to knew they should merge the if-statements and factor out the prints. But the ones who knew they should refactor out a “dashboard stat” value only did so by flash of insight, and couldn’t articulate how they came up with it. But by instead thinking about how the code is derived from the design, this refactoring becomes just a straightforward application of the Embedded Design Principle. Finding a good structure for the code was easy once we saw the structure of the design.

So always think beyond the code. Ask yourself about the concepts of your program, and the values that define them. Like Piersig, this is what turns you from the mechanic turning screws into the engineer and artist building something to be proud of.