How Successful Was the MIT Challenge?

By far the most popular project I’ve done publicly has been the MIT Challenge. Starting nearly seven years ago, in the fall of 2011, the idea was to learn the curriculum of MIT’s four year computer science undergrad, evaluate myself by trying to pass the final exams, complete the programming projects and finish in one year.

In simple terms, the project was a success. I did what I set out to do, under the constraints I had imposed upon myself, within my original time limit.

However, behind those simple terms rested a lot of implicit assumptions. Now that I’ve had more time (and hopefully wisdom) between me and the project’s completion, I’d like to dig into some of those assumptions and see how well they’ve held up over time.

Whenever you set a goal in life, there’s a certain simplification of reality. Say you set a goal to lose 15lbs. Implicit, although maybe not articulated, behind that goal are a whole host of assumptions: that you’ll be healthier, look better, that the lost weight will be fat. Even behind those are deeper assumptions you may not even be aware of, such as that being fit is an important value in life.

Therefore, I think any project can be simultaneously evaluated at multiple levels. There is the explicit level of what you stated you were trying to achieve. But beneath that, are many other ideas of success and achievement, some of which you may only realize long after the project has finished.

I believe doing this kind of postmortem (however belatedly) is useful because, while highlighting one’s successes is fun and easy, it’s by looking at how our ideas differed from reality that we get true growth. I also believe that by digging deep and looking at some of my mistakes, I might be able to offer a more comprehensive picture for those who want to do better than I did in the future.

Side note: This is an incredibly self-indulgent post. If you’re not that interested in hearing me rant about myself and my past projects for 2000+ words, you should definitely skip it.

How to Evaluate?

I can think of a couple ways to evaluate the project, beyond simply whether or not I matched my explicit standards and constraints:

1. Knowledge and skill. How did the knowledge and skill I acquired during my project compare with an MIT student. Did my choice of constraints/standards lead to missing things that were important?
2. Rigor and difficulty. How well did my constraints match those of an MIT student? In particular, did my choice of constraints/standards make some things much easier than they should have been, or some of the accomplishments less comparable to the “official” setting?
3. Outside impact. Did my ideas about what the MIT Challenge would mean for other students hold up?

On these three bases, I think the first holds up best, the second is debatable and the third was probably the biggest failure of the project.

How Successful Was The MIT Challenge for Learning Stuff?

The most obvious basis for evaluating the Challenge was to look at how it worked for actually learning the material, and perhaps secondly, how those skills extended beyond the scope of final exams and programming projects of the coursework.

In this sense, I believe the project was mostly a success. There were departures from the normal computer science curriculum. However, these were normally done because to follow the original program strictly would have been impossible. This included things like not having a thesis programming project, doing economics instead of humanities classes and substituting classes which depended on lab work for more theory-based classes.

These deviations likely made my experience different in some ways from an actual MIT student. However, I think for the most part, they didn’t substantially detract from the experience. I still learned roughly the same volume of material, it was just shifted in some ways to make the project practical.

More critically, however, I think are how ways I did the challenge might have impacted my learning of computer science outside of what could easily be measured via my system of doing final exams and programming projects.

I can see a few places where I think my approach to this issue may have created problems:

1. Spacing. By compressing the challenge into one year, it’s likely that I did some damage to my ability to retain the material long term, versus, say, a more traditional four-year schedule. I did my best to mitigate this during the challenge by taking multiple classes simultaneously. However, I think, knowing what I now know about the typical half-life of knowledge learned in formal schooling, I would have been more serious about planning a long-term retention plan, the way I did with my language learning project which had a similar weakness.
2. Lack of peer interaction. During the challenge, I was mostly dismissive of the importance of peers/teachers. I felt this way, rightly or wrongly, because I didn’t find those resources very helpful when I did my actual undergrad. However, I think, owing to the extremely theoretical nature of MIT’s CS program, the peer effects likely are an important part of the education process at a top school, since they allow students to find out about practical projects, tools and opportunities that aren’t taught in classes.
3. Insufficient overlearning of core calculations. I believe my ability to do advanced mathematical calculation and symbol manipulation (algebra, calculus, etc.) was probably weaker than it should have been. Some of this showed up on the exams, with such classes being my worst performing. However, some of it was hidden from that perspective because I was aiming for moderate performance, not overlearning beyond sufficient performance. An actual MIT student, by means of doing assignments, past what is necessary, would likely have better abilities and better long-term retention of those types of abilities than I did.

These critiques don’t mean that I think I could have necessarily done much better, had I to do it all over again. Rather they’re useful to think about in case you wanted to attempt something similar. In my own situation, I think these trade-offs were mostly warranted, however someone learning for a different purpose might disagree.

Trent Fowler recently did his own year-long engineer/computer science program. While it differed from mine in many regards, an interesting way it did so was a much larger emphasis on practical projects. I’m not quite sure how to compare the two, but it does indicate that the space of possibilities for learning is vast.

How Successful Was the MIT Challenge for Matching the Rigor and Difficulty of the Real Thing?

Here my project and the actual degree start to differ substantially. I used a much more limited set of evaluation criteria. I focused on getting a pass (50%+) rather than getting as high a GPA as possible. I graded my own exams, using a solution key, rather than wait for an independent grader to go over them.

If the goal of the Challenge had been to replicate the difficulty of an MIT degree, or to perfectly match the standards of evaluation MIT used, then the project was doomed from the start. This was clearly not my intention, so evaluating it on this basis seems unfair (although, perhaps, probably worth pointing out to many people who casually observed my project and might have been misled to think that I did everything an MIT student would do, and got a degree to go with it!).

However, even within the intention of the project, my goal wasn’t to substantially reduce the difficulty where that was intended to measure my performance on the stuff I had learned. My original idea was that, should I clearly fail an exam according to MIT’s standards, I should have to retake it. I shouldn’t be giving myself an easier time, just because the material was hard.

From this basis, I think I did fairly well, although there are some points of contention. The most major one is how I did grading. While, in almost all cases, I only wrote exams with solution keys I could use to grade later, there often wasn’t clear instructions for how to handle grading. This, like many aspects of the challenge, required assumptions to be made about the proper way to do something.

Generally, my approach with exams was that any instructions provided (say explaining the amount of marks provided for a particular answer, or allowable materials, etc.) must be followed exactly. If information wasn’t provided, I did my best to devise what I thought was an acceptable approach (say docking a certain amount of marks for making a mistake in algebra, let’s say).

For most of the classes, this judgement call wouldn’t have made a difference. However, in some of the classes, particularly those which required long, drawn-out calculations, it could pose a challenge. If an exam had a question worth ten marks, how many points should be deducted because a minus sign wasn’t carried over?

Here opinions vary. In my own undergraduate experience, partial marks were common. Others, however, prefer stricter grading. In that view, any mistake, how matter slight, and all marks should be forfeit (including any follow-up questions that rely on the original result). This was not how I graded myself during the MIT Challenge, nor the basis I used for deciding whether I passed.

In the interest of curiosity, however, I decided recently to go back and see what the outcome would be if I applied the extremely strict grading. No part marks, no carry-forward of mistakes from multi-part questions from one part to the other. Under this new, ultra-strict, grading scheme, my exams passed went from 33/33, to 27/33.

I don’t think that the “correct” answer is the ultra-strict approach, and I still think the original way I evaluated the exams was fair (even if it may not have been exactly the same as another grader would have marked it). However, I think it’s worth noting the extent to which these assumptions might change the outcome of certain things.

An interesting, if perhaps unfortunate, consequence of the MIT Challenge was that many people focused on the presumed “impressiveness” of it, rather than (as my original hope) the idea of learning a lot of material, in a short time, with low cost and some benchmark to university standards. If that impressiveness is grounded in the idea that I did something equivalently difficult to a “real” MIT degree, irrespective of the knowledge acquired, I think the project doesn’t really work. This isn’t so much a critique of how I did the project, but perhaps what other people make of it.

How Successful Was the Idea of the MIT Challenge?

My original hope with the Challenge wasn’t to gain fame as the guy who did this strange project. Rather, I sincerely believed that it was only a matter of time until there were many people doing projects like this and I really wanted to be the first.

In retrospect, this is the biggest area where my thinking was at odds with reality. In the time since the MIT Challenge, I’ve met a handful of people who have done vaguely similar projects in terms of scope and ambition. However, this is a far cry from being a stable and recognized alternative to the formal education system.

Technology isn’t the problem. When I did the Challenge, that was before MOOCs, before the wave of online courseware and certificates from real institutions. Indeed, many of the deficiencies of my project were a result of the patchwork way I had to do it. Most classes had only a fraction of the full material available to an MIT student, so I had to make do. Today, doing something like the MIT Challenge is even easier.

I’ll admit, I was an education optimist. I believed that the purpose of school was learning, and if one could get the learning for a fraction of the cost, it would only be a matter of time for institutions and students to follow.

I’m now much more of an education skeptic. I largely agree with Bryan Caplan’s view that formal education is mostly signalling. This is the view that school doesn’t so much teach you valuable skills, but helps filter society by which people are smart, conscientious and conformist enough to put up with it. This filtering can explain why schools seem to teach so much useless stuff, yet are nonetheless a requirement for almost any good job.

That view took a long time for me to come around to, mostly because I really love learning the kinds of things they teach in school. This was particularly true during the MIT Challenge itself. Although it may sound strange to say it, I genuinely enjoyed most of the project. It was an intellectual puzzle talking about big ideas I could do at my own pace and with my own process. My actual university classes I did during my undergrad, I enjoyed a lot less.

However, this signalling explanation seems to fit the data, and my post-experiment experience, much better than any other I’ve heard.

This doesn’t mean I think learning isn’t valuable. It just means that a lot of education isn’t really about learning. The kind of learning which is valuable, both professionally and personally, is often at odds with how school curricula are designed. This is a bittersweet realization because it both affirms the idea of the Challenge (that big gains in potential learning are possible, in principle) and undermines it (because such gains rarely happen because, institutionally, we mostly pay lip-service to the ideals of learning while we really care mostly about filtering people).

As I wrote, shortly after the Challenge, if my goal had solely to acquire computer science knowledge and become a better programmer, I might have taken a radically different approach. Much more programming. Way less theory. Especially less of the abstruse mathematical sub-fields of which I did more than a few classes on.

However, if I had taken the Challenge and optimized it solely for acquiring computer science knowledge, it might not have attracted as much attention as it did. In that sense, it may be that elements that caused it to fail at my stated purpose, ended up allowing it to succeed in my career as a writer. Projects which succeed, not in spite of their flaws, but because of them, are a feature I’ve found in many of the things I’ve done in life, even though their paradoxical nature makes them impossible to plan in advance.

What Should You Think About the Challenge?

One thing I learned early about writing is that people rarely read the words you write, as you intended them. They read themselves in between the lines, and fill it in with their own imagined realities. This can often be frustrating, because people want you to be accountable, not only to what you said, but also to the hidden ideas they had of you.

In that sense, there’s nothing to be done. For some people, they’ll like the idea of learning without going to school. Others see a possibility of doing a lot more by rethinking their methods and constraints for taking on a project. Maybe other people just admire the boldness and strangeness of someone who would push himself to grind through classwork and exams full-time for a year without any credentials at the end.

For myself, I think the Challenge offered me many opportunities, both in my profession as a writer, and personally, in changing how I think about accomplishing big goals and learning new things. However, despite the unambiguously good things that came from the Challenge, I think there’s a lot of things that I was wrong about, or could have done better.

In that sense, I hope not that people get inspired by a project I’ve done or article I’ve written and try to do something similar, but that they can also learn from my mistakes and strive to do better than I could have done.