Friday, February 23, 2018

The Practice is not the Performance: Why project-based learning fails

Last night, I encountered an old post by Zach Holman where he pushes the idea that traditional school-based CS is useless; project-based learning is the way to go. I’ve heard this idea repeatedly over the last 10 years, and know at least one person who’s started an education company with that premise.

I don’t want to debate the current way universities do things (I found my undergrad quite useful, thank you), but I do want to dispel the idea that everything would be better if only we switched to project-based learning. The opposite is closer to true. Project-based learning is not a superior new idea. Project-based learning is the most inefficient form of learning that still works.

To see why, we actually have to sit down and think about the learning process. I’ve learned a number of models of learning over the course of my teaching training, but the one I’ve found most useful is taught by the Center for Applied Rationality. It goes like this:

  1. Break down a skill into small components.
  2. Drill each component rapidly, with immediate feedback
  3. Integrate the pieces together into larger components; drill these similarly

In that light, project-based learning definitely has something going for it: if you do a project, you will practice all the skills needed to do projects.

There’s something missing, though: it completely gives up on trying to think about what components it’s trying to teach.

Here’s how project-based learning might play out: After finishing his first two Android apps, Bob decides he wants to learn network programming, and generally get better at working on a software team. He and Alice pair up and decide to build a chat app for sending cat gifs, with a distributed replicated backend. They decide to make a spreadsheet of tasks and claim them. Alice also wants to learn network programming, and she swoops in and takes most of the server components; Bob gets left with most of the client work. Over the next three weeks, Bob spends a lot of time building the GUI for the app, which he already knows how to do, and only a couple hours implementing the client protocol. They start writing tests because the teacher said they had to. Bob has trouble writing the tests, and realizes a couple ways he could have made his code differently to make testing easier. He also discovers that two of his tests were too fragile, and needed to be changed when he updated the code.

It’s now one month later. What has he gotten from the experience? He's learned he needed to be more proactive about taking on tasks that challenge him and grow his skills. He’s learned a smidgeon about network programming, and a couple ideas about how to write better tests. These are good lessons, sure, but expensive. Isn’t there a way for Bob to learn more in that month?

False Temptations

What are the common arguments in favor of project-based learning? Here are two of the main ones.

Real skills

The first big reason for project-based learning is that it teaches real skills used in industry. Why do many schools teach much of their curriculum in Haskell, not in the Tiobe Top 10, or even SML or OCaml, not even in the top 50? Wouldn’t they serve their graduates better teaching Node and React?

The first counterargument is that industrial technologies come and go. Proponents acknowledge this, sure, but still call CS departments “out of date” for not following trends. What really drove home the futility of this argument for me was this essay by software-engineering pioneer Mary Shaw. Had she followed that advice in the 60s, she points out, her students would have spent their time studying JCL, the language used to schedule jobs on IBM mainframes.

The second and bigger counterargument: learning concepts is much more important than learning applications, and the best environment to learn a concept is rarely the one in industrial demand.

People often ask me what’s the best language to learn to study software design. I ask them what’s the best instrument to learn to study music theory. Everyone answers piano. In piano, you can see the chords in a way that you can’t in, say, trombone. We see something similar in other domains. In The Art of Learning, Josh Waitzkin recounts how, unlike others, he started studying chess at the fundamentals, in situations with few pieces on the board. He ultimately beat competitors who studied many times harder. In The Art of Game Design: A Book of Lenses, Jesse Schell advocates looking past modern video games and instead studying the concepts in board games, dice games, playground games.

So, for programming, we need to (1) figure out the core concepts to teach, and (2) pick languages that make the concepts readily available. C and Java --- indeed, all top languages until Swift arrived --- lack elegant ways of expressing the idea of a sum, a value that can be one of many alternatives. Thus, when explaining why adding a new alternative sometimes breaks existing code and sometimes doesn’t, I find myself having to explain using the clumsy manifestations as unions and subclasses. The first time I read Bob Harper’s explanation of why they chose SML for the CMU undergraduate curriculum, I thought he was just rationalizing snobbery. Now, I quite agree.

More like real jobs

The second big argument for project-based learning is that it more closely resembles what students will actually do on the job. This, in turn, is based on the idea that the best way to practice an activity is to do it.

This is false. “The only way to improve at X is to do it” is the advice you give when you actually have no idea how to improve. When you do know, you isolate subskills and drill them. Martial artists punch bags and do kata, and fight with extra constraints like no dodging. Musicians play scales, and practice a single measure over and over. Mathematicians rederive theorems from the book. And to condition a shot-putter, the best way is not to put weights in their hands and have them mimic the throwing motion, but rather to train the body’s ability to produce power, using squats and lots of heavy exercises that don’t even resemble shot-putting.1

Drilling programming

So what am I advocating? I’m advocating that you actually think about what you’re trying to teach, and design drills for it. The first drills should focus on one thing at a time.

So, for Bob trying to learn network programming and the general software engineering skills of being on a team project, here’s my 5-minute attempt to come up with an alternative way to teach these skills:

  1. Writing just the networking component of a larger system.
  2. Being asked to write the test cases for a small program given to you. The program is deliberately designed to tempt you into all the classic mistakes of test-writing.
  3. A simulation where you “coordinate” with the TAs to build something, committing pseudocode to version control. They troll you by deliberately misunderstanding things you say and seeing if their misunderstanding can go undetected.
  4. You and several teammates are assigned a small project. You’re asked to divide it amongst yourselves into unrealistically small units of work, and write test cases for each other’s components. (Integrates the skills of 2 and 3.)

These total much less time than the chat app, and teach much more. If students find they’re not optimal for learning, I as the instructor have much more room for experimenting. And if the students do choose to do a full project afterwards, they’ll be much more prepared.

I don’t teach network programming and haven’t tested these specific ideas. But, in my specialty of software design and code quality, I use exercises built on similar principles all the time. So I may teach a client some of the pitfalls of a naive understanding of information hiding, and then show them code from a real project that has that problem and ask them how they’d solve it. Or I’ll ask them to give just the type definitions and method signatures for a Minesweeper game; if they violate any design principles, then I can give them a feature request they can’t handle, or show why the implementation will be prone to bugs.

Is it better than just assigning projects? That’s the wrong question to ask because project-based learning is incredibly easy to beat. My clients are mostly working professional software engineers; they’re already doing “project-based learning” every day. On my website, I claim.

Programmers learn by making bad design decisions, working on the codebase for a year, and then find themselves wishing they could go back and do things differently. I can give you that experience in an hour. 

Does this sound like a bold statement about my teaching prowess? It’s not. In fact, piano teachers put that claim to shame. You can spend hundreds of hours practicing a piece using too many muscles on every key press. If your body awareness isn’t great, you might not find out until your hand cramps up right before your performance. A couple seconds to catch that mistake, a couple minutes to tell you a story of that happening to others, and the piano teacher’s just saved you months.

(As an aside, this is why I believe in finding a competent private coach for every skill you really care about improving in.)

Replacing a traditional CS education with a “software engineering BFA,” like Spoelsky and Atwood suggest, is no longer a hypothetical exercise. We’ve tried it. And now dev bootcamps are going bankrupt. Instead of substituting for a traditional degree, recruiters are calling bootcamps jokes. Olin College of Engineering is famous for its project-based curriculum, but one student reports that she learned much more from traditional classes.

It’s time to stop looking for panaceas and shortcuts and realize that deliberate learning and deliberate practice --- as a separate activity from the everyday doing --- is the only way to mastery. As famed gymnastics coach Chris Sommer puts it, the fastest way to learn is to do things the slow way. Studying the fundamentals may seem like a distraction keeping you from getting your hands dirty making a Rails app using the Google Maps and Twilio APIs, but when you do get there, you’ll find there is less to learn if you’ve already compressed the knowledge into concepts.

Shameless Plug

My Advanced Software Design Web Course starts next week. It’s based on a lot of the learning principles I mentioned in this post, starting each concept with isolated drills and progressing to case studies from real software, and comes with personalized feedback from me on every assignment.

Disclaimer:

No, the sum total of knowledge about CS education is not to be found within this post. Yes, I do have some formal training in education; yes, other people have a lot more. Yes, there are a lot of things I didn’t bring up. Yes, the situation with bootcamps is more complicated than a simple referendum on project-based learning. The simple “turbocharging training” model of learning I gave is not a theory of everything. Yes, you need to run into problems in context, find motivation, try things out of order, and even eventually do a full project on a team without guidance. I believe realistic projects do have a place in education, but they still must be coupled with the principles of rapid feedback, and they are a poor substitute for learning the concepts piece-by-piece.

Acknowledgments

Thanks to Elliott Jin for comments on earlier drafts of this post.


1 When I was searching for a personal trainer, I asked about this to help screen candidates.

Liked this post?


Related Articles

12 comments:

  1. Hanon exercises vs Bach's Inventions

    ReplyDelete
  2. I like a lot of your points and I often draw parallels to music instruction for my own teaching (an online electronics course).

    However, my counter argument at a high level is doing something remotely practical and applicable is better than staying abstract. You quoted Mary Shaw, "Had she followed that advice in the 60s, she points out, her students would have spent their time studying JCL, the language used to schedule jobs on IBM mainframes."

    This point is true, but only matters if the student expects to stay on that one language for the remainder of the career...something that is obviously not possible for all of them.

    In my limited experience I have seen the practical ignored many times over. It's done so due to the teacher's laziness or lack of stretching themselves; they don't want to learn the new language or framework or methods. So they hide behind pseudocode and mathematical equations. In my opinion, it is a teacher's job to hold a student's attention and a key way of doing it is showing relevance to the modern world and for things that are directly applicable. Any teacher worth their salt will be able to pull the concepts from that application and show how it might be applied later. I would further venture it'd be better to have a teacher make up a language or an application and treat it as though it was a real world application. To stay in the abstract is much more often a function of the teacher than the importance of higher educational ideals.

    ReplyDelete
    Replies
    1. I agree with what you say about keeping a clear view to something practical. That's not at all in conflict with having structured assignments. If you're teaching music theory to a trombone player, then you will actually analyze the chord structure of the sheet music that they're currently playing. A bit further out there, the first day of my Cost Accounting class, a very dry subject, the professor showed how cost accounting affects current events, such as Trump's claim that he'll donate all profits from foreign dignitaries to the US Government, and how they calculated that profit. None of these require anything near the scope of a "project."

      It turns out that good teaching is a lot of work, no matter how you slice it.

      For your first point: sorry, not sure if I follow. If students are expected to use multiple languages in their career, then they should study JCL in school?

      Delete
  3. This is interesting. Thank you for writing it, you gave me some things to think about.

    I know this isn't the focus of the post, but I feel like you glossed over one argument. Would you mind giving me your thoughts on it?

    That is the argument that CS departments are training Computer Scientists, not Programmers. Because of this, there should be a disparity between the things taught in higher education, and what businesses want.

    ReplyDelete
    Replies
    1. I pretty much wrote this from the perspective that the education provider and the student are aligned in what they want to teach/learn, as you might get from someone signing up for a course a la carte from a provider such as Bradfield CS. Universities tend to also have a bit of a social-engineering aim in their curriculum. What to teach is a pretty separate issue than how.

      Even for universities though, it's not universally true that they are not training programmers. There are software engineering programs like at ANU, and co-op programs like at Waterloo and Northeastern. The University of Cincinatti as a whole even has it in their charter that they are there for industry.

      Delete
  4. I do agree with the deliberate and focused practice over doing projects approach where the boundaries between various concepts and components of the system are unclear. I'm curious if there's a good list of CS exercises in your knowledge? Can be anything ranging from CS fundamentals to software design to testing, etc. Thanks.

    ReplyDelete
    Replies
    1. Well, that's quite a broad request, if I'm understanding correctly. You may have well asked for a list of all good math exercises, to which the answer is probably "Go to a university library and look at the exercises of every textbook."

      The good news is that structured exercises have been the default in both books and courses for a long time, and so they're not very hard to find. Software engineering/design is the main exception that comes to mind (perhaps also game design/development and some business courses like entrepreneurship, along with much of the fine arts), where the choice is typically between "learn by doing on the job" and "learn by doing in a course that resembles a job." That's part of why I've focused so much on developing structured exercises for my software design course. At a lower level than where I teach, some undergraduate software engineering courses do take a structured exercise approach (e.g.: many CMU courses), while others (e.g.: Stanford and University of Houston) take more of a "design studio" approach.

      An example currently on my mind: I'm currently taking MIT's Operating Systems course ( https://pdos.csail.mit.edu/6.828/2018/schedule.html ). It lives the ideas in this post, particularly when contrasted with CMU's OS course (which my undergrad friends took).

      The meat of CMU's course is making you write an OS kernel from scratch, with relatively little guidance. The TAs infamously answer every question with "You need to make a design decision." Finishing that course gives you a metaphorical "hardcore" sticker.

      Comparatively, the MIT course is very structured and guided, especially at the start. While you do eventually write most of a kernel yourself, there
      It's a pretty good example of the ideas in this post: it starts with very structured exercises ("step through this bootloader code in a debugger, look at these memory addresses"), and gives you starter code for dealing with APIs like the protocols of external devices and the format of debug symbols, so that you can focus on more fundamental things at the start. Less hardcore, but probably much efficient pedagogically. And a good example of how, thankfully, the classic approach of structured exercises is still the norm.

      Delete
    2. I'm sorry for the confusion. I should have been more specific.

      So-called "awesome" lists that, usually hosted on GitHub, became quite popular in the recent years. And I was wondering if there's such a curated list of structured CS exercises in your knowledge.

      The closest thing to what I'm describing is a hacker news thread [0] and Teach Yourself CS curriculum [1] (though indirectly). Would be great to collect the greatest structured exercises and categorise them (not implying you should do it or anything).

      [0]: https://news.ycombinator.com/item?id=16591918
      [1]: https://teachyourselfcs.com

      Delete
    3. Aw, I see.

      Interesting. I can tell you that, in my specific area of software design, I collect articles religiously, and have found maybe two that had interesting exercises.

      Probably the closest thing to what you're looking for in this domain: http://nealford.com/katas/

      Delete
    4. Interesting. That means we're onto something here. Seems that the industry is truly lacking proper exercises.

      Re: Katas, here's one more list: http://codekata.com/

      Delete
  5. "Programmers learn by making bad design decisions, working on the codebase for a year, and then find themselves wishing they could go back and do things differently. I can give you that experience in an hour."

    Hi, I'm a software engineering student who started working for a company almost a year ago and I've already experienced this. After this experience I can't emphasize enough on how much easier it would've been since the beginning by applying software architecture knowledge which on a project-based learning is mostly learnt by failing.

    ReplyDelete
  6. The book "How We Learn: The Surprising Truth About When, Where, and Why It Happens" talks a little about drilling. The gist is that repetition can be surprisingly counter-productive, because it fails to teach us how to contextualize problems in different situations. Being able to think about what we know from different angles increases our expertise and fluency, and helps us retain memories longer. So the book advocates for "varied practice."

    > The mixing of items, skills, or concepts during practice, over the longer term, seems to help us not only see the distinctions between them but also to achieve a clearer grasp of each one individually. The hardest part is abandoning our primal faith in repetition.

    The book gives a few examples: about athletes -- you mentioning kata reminded me; about surgeons, but this one is about pilots learning how to quickly react to instruments panel states...

    > The participants saw, on the screen, an instrument panel, below which were the seven choices. If the participant chose the wrong answer—which novices tend to do at the beginning—the screen burped and provided the right one
    > After one hour, even the experienced pilots had improved, becoming faster and more accurate in their reading. The novices’ scores took off: After one hour, they could read the panels as well as pilots with an average of one thousand flying hours. They’d built the same reading skill, at least on ground, in 1/1,000th of the time.
    > “The large improvements attained after modest amounts of training in these aviation PLMs suggest that the approach has promise for accelerating the acquisition of skills in aviation and other training contexts.” Those contexts include any field of study or expertise that involves making distinctions. Is that a rhombus or a trapezoid? An oak tree or a maple? The Chinese symbol for “family” or “house”? A positive sloping line or a negative sloping one?

    The author goes on to argue that obviously you can't replicate a pilot's full skillset this way, but it does illustrate the efficiency of "varied drilling," particularly though active decision-making. If we really do learn most efficiently by actively seeing what we are interested in through many lenses, and contextualizing by making decisions about what we know, then simple repetition isn't exactly an answer to projects.

    It doesn't sound like you're advocating for that -- your examples and drill examples seem quite varied, in line with the book's recommendation -- but I thought you might find this interesting nonetheless.

    ReplyDelete