Harvard MOOC story - Part 5
As the MOOC completes what was the first third of the on-campus course, we get to experience our first test. While there is a slight reprieve from problem sets, extending this round into another week, the focus of the lecture sessions and the small-group sections turned toward test prep. So what does a MOOC test look like, at least for CS50?
Week 4 Take-aways:
- List the Do’s and Don’ts – The test started with VERY explicit descriptions of what is and is not acceptable while taking the test. No wiggle room was left for students to justify additional resources not condoned by the CS50 staff. Sometimes instructors forget about the flip side, failing to tell students what is condoned, and only the more daring students might use a resource that greatly benefits them, while unsure students struggle on without it and are hurt by their over-caution regarding the sanctity of The Test. The CS50 test instructions did a good job of explaining both sides. OK: anything from the CS50 course, other than the discussion board. Not OK: anything from anywhere else, or any human interaction.
- Academic rigor? – The Test is likely the limiting factor when determining whether a MOOC has the academic rigor to match that of a “real” course. In this case, there was a lot of factual information that, if I had to, I could have learned (or memorized). But I didn’t have to, so I didn’t. I knew that with the open-note test, I could easily search the posted lecture notes on the CS50 site and search my own notes files (taken with Evernote). The on-campus students, though, had to actually stuff this information into their brains for rote recall later. So the searchability of an open-note test somewhat cheapens the MOOC experience, compared to the “real” course. But most of the questions were based on logic and the experiences gained through the first 4 problem sets, which is where the MOOC regains some of its lost rigor. In the age of Google it is less important whether you can remember a set of facts, since it’s not that tough to look them up. Assessments in MOOCS and other online courses should make a significant shift toward the higher-order thinking problems that prove mastery of a subject far beyond knowing the definition of a term. The CS50 test was tough.
- Location, Location, Location – It seems like an obvious tip, and one that I’ve stated before, but students need to be able to find things. Especially in an asynchronous course in which students are left to make their own way through material. If I were designing a course in which students took a quiz between Week 4 and Week 5, that is where I would put the link for the test. I would not locate the link on the opening page of the course, next to the syllabus, which hasn’t been viewed in 4 weeks.
- Re-design on the fly -- Keeping with the previous situation, if you are monitoring discussion board posts and see that many students are having trouble finding a quiz link, after posting the hyperlink in their discussion threads, so ahead and scatter it a few more places in the course. If the previous students had trouble finding it, the remaining students will have trouble finding it. You probably didn’t design your course perfectly, so look at each wave of confusion as the opportunity to make it better for the future, now.
- Estimated timing – It’s helpful to know how long the test is before we start it. Help us plan by giving us a time estimate or at least the number of questions. Online students may be working around tight schedules or on a time-limited library computer somewhere.
- Encouraging retakes – The CS50 test can be retaken as many times as a student wants. Even if a student passes, retaking a test is another intense exposure to the most difficult material. To encourage students to try a retake, their answers are emailed to them after their test is submitted. Not the questions, just the answers. Now if they want to retest, they wouldn’t be shooting in the dark as much, and would feel like they are being given an extra tool for success. Instructors should consider this as a good strategy to increase long-term learning in students who try to master information, not just leave it all on their bubble sheets. Or computer screens.
- Don’t forget the review – Especially in something like computer programming, where a lack of understanding now can lead to major problems down the road, it is important to remember that students need feedback on their feedback. An overall grade and a check mark on a particular question don’t really do much to help a student learn. The instructors took some time to review the most contentious questions in the small-group section following the test. While the MOOC students aren’t getting one-on-one feedback from instructors, seeing them work with the small-group students is an extremely effective way for us to review our own understanding of the test material. What if you don’t have an on-campus class to drive your post-test analysis? How do you know what to discuss with the faceless masses and their innumerable differences of mastery? If your test questions and decoy answers are crafted in an intentional manner, the incorrect answer that a student chooses should tell you something about their misunderstandings. Your video review can address students in a way that explains what their assumptions probably were, and help anyone who answered ‘B’ know what they need to fix, which is different from anyone who answered ‘C,’ because they were making a different assumption to reach that answer.
Stay tuned for more online design tips from my exploration of Harvard's CS50 MOOC.