I also procrastinated enough on this last report that the nice folks at CS50 posted some very good data on the success of their course, which you can view here.
Harvard: MOOC Final Take-aways
Motivation, and the difference between 60% on everything and 60% on each thing:
Around the fifth of the eight problem sets in the course, I had to make a choice that is at the heart of any challenging MOOC. How much work am I willing to put in for 1) learning stuff, and 2) a shiny certificate, from this unrequired MOOC? This point comes as the Difficulty and time requirement in the course surpasses the Perceived Benefit that the knowledge gained will provide to my life. (Shown in handy graph here) In short, if I don’t think the depth of the course will be useful, should I do it anyway? If I was paying for college credit, then yes. But for a free MOOC???
I calculated that if I skipped a couple of problem sets but continued with the course and the final project, I would be able to pass the course with well over the required 60% and get both the knowledge that I felt I needed, and the shiny certificate. But upon a 2nd, 3rd, and 4th reading of the syllabus, I realized that I may have misinterpreted the course requirements, “Score at least 60% on this, this, and this…”). That means on the whole bundle, right? No. I had to score 60% on each problem set. That just wasn’t going to happen with the amount of effort that I was willing and able to put into the course. From the first chart on the CS50 data page, it appears that this was the pivotal point for many of the MOOC participants, because following this area, there was a much lower rate of dropping out. While the goal of the MOOC is to teach students, it doesn’t seem like the reward was motivating enough for roughly 90% of the 10,000 students who stated that their goal was to complete all course requirements.
Shiny certificates do work though: The ego-boosting idea of a Harvard certificate hanging on my office wall kept coming to mind as I decided whether to trudge on or to bail. I’m sure it made a difference to some of the other students facing the same decision, and to many of us who are looking forward to the course’s next incarnation in the fall.
The point of badges: After I realized that I wasn’t going to be successful in this course, I did continue to work on the last problem set and the final project. To me, these last assignments were back in Positive Benefit Land, so I would be wasting opportunity not to learn from them. I am curious how many “dropouts” went back to the course to work on these. Thinking again about motivation and the injustice of a linguistic mistake pulling the rug out from my plan of skating by with a minimum grade, I gathered some thoughts on how a badge grading system could affect courses like this.
- If I was able to score 60% and get a certificate, mine would be as meaningful as a student who aced every piece of the course. That is hardly fair or representative of what each of us had learned. Badges signifying which programming skills each student has mastered would be tangible and understandable information for future employers to see, lending credibility to the learning outcomes of MOOCs and any other course.
- If I knew half-way through that I wasn’t going to be successful with the entire course, why continue? With a badge system, students can fail on one part but understand that there is still something to work toward, as they complete the remaining sections of the course. CS50’s graph showing student completion of each problem set would look less like a continual drop-off, and could show a surge of activity on the later parts of the course that students wanted to continue. Many more final projects might have been completed, showing what the other 9,000 active students got out of the course.
- The goal could be to collect a percentage of the total programming skills, rather than scoring a 60% in each program assignment. Even within one programming challenge, students might master four skills but fail one, and the program doesn’t run well enough to score the necessary 60%. A skill-based badge system would reward a student who is able to obtain 60% of the entire course’s expected learning outcomes, even if part of the course was over his head. The a la carte course would serve students better, according to their own needs and interests. CS50 does this on the top end, by providing the “hacker edition” options on assignments. Why not extend this to the rest of the course? (says the guy who doesn’t have to design a computer grading system capable of determining which skills within a program are mastered and which are not)
A valuable final project: You can’t expect the 150,000 students who signed up for the course, or even the 1,388 who successfully completed the entire course, to care about the same computer programming applications and uses, so the culmination of a course like this should be something chosen by individuals. CS50’s requirement for the final project is that you use skills from the course to make something useful. That’s pretty much it, other than making a short video to show it off. I put more effort into personal projects than in cookie-cutter assignments with questionable utility to me. Self-determined projects do a much better job of letting an instructor see just how much each student has learned and is able to do on their own, which is kind of the point of any class.
Final thoughts on the data: MOOC skeptics will look at the 0.9% rate of success among the 150,000 students who signed up as absurdly low, and proof that MOOCs just aren’t going to hack it when the novelty wears off. But the data analysis page points out that 50,000 of the sign-ups didn’t even log in once. Should these window shoppers be counted in the success/failure rate? Overall, only around 11,000 students even intended to do all of the required work. The vast majority of participants were just trying to learn something new or to kill some free time on Friday nights, and weren’t interested in the certificate of completion. In the end, 1,388 students completed a rigorous course in a potentially life-changing, career-starting set of skills, and without the MOOC they wouldn’t have the opportunity or motivation to do so. I am curious about how many of the successful students were among the nearly 2,000 who reported taking the course “as a refresher,” compared to how many were newbies to programming. That is some data that would really show that a MOOC like this is an effective tool for teaching a difficult subject like programming to complete beginners.