Martin Weller has a great blog post up about course design responses to MOOC completion rates. He starts by arguing that, while completion rates are not everything in MOOCs, they are not nothing either. A lot depends on whether you think completion is an important metric to meet the course goals because, for example, the course is designed to help remedial students pass into a non-remedial track, or whether having students explore the content in a non-comprehensive way accomplishes your course goal. (Martin brings up an analogy by Stephen Downes that nobody complains about the low newspaper completion rates, which I have never heard before and which I love.)
This is good stuff, but it starts us down the path toward a more radical re-examination of how we think about course design. Because while Martin is focusing primarily on course goals and how those should determine metrics, he’s beginning to raise the question of how individual learner goals should influence course design. And once you start asking that question, it changes everything.
The Myth of the Unified Course Goal
Pretty much all popular course design methodologies that I can think of start with the assumption that the goal of the course is to be able to certify that students in the class have learned a well-defined set of knowledge and skills (or, at least, 70% of that set, which is generally enough to pass the course). The truth of the matter is that it is never that simple. Students always have different reasons for taking the class and therefore different goals, different support needs for achieving those goals, and different behaviors that they adopt in order to achieve their goals. When I sign up for a class, I might be signing up for it because I am a major in the subject, want to pursue a related career, and need to learn everything I can about what is being taught. Or I might be taking it because I have never taken a course in the subject before and am curious. Maybe I have a passion for one of the subtopics covered in the syllabus—a particular poet, for example, or a theory that is related to a different discipline which is the one IÂ really am passionate about. Or maybe I’m just trying to fill a prerequisite on my way to getting a diploma. Or I heard that the professor is great. Or easy. Or I have a crush on one of my classmates. Or I have a crush on the professor. Each of these motivations will impact my definition of success for the class and therefore my behavior in class.
We are able to paper over these differences, in part, because of the high barrier to entry for traditional university schooling. Courses cost money and take time. Whatever other motivations I may have as a student, the odds are pretty good that if I am willing to spend the time and the money then one of my  goals is to get credit for the course. So in our design activities, we often pretend that this is the only or, at least, the most important goal that the students have. To the degree that good teachers distinguish among the different goals of their individual students, they generally don’t do so through course design. Most of the time, they adjust their teaching styles and work their personal relationships with those students. At design time, we assume homogeneity of goals, even as we (hopefully) work hard to account for heterogeneity of abilities. This is a convenient fiction because our tools for designing courses for diverse student goals are pretty limited in a traditional class. For one thing, the teacher can only be in one place at a time. Most traditional face-to-face course designs are pretty much single-threaded (although there are some exceptions). For another, the whole system of charging tuition in exchange for credits really pushes us to take seriously our responsibility to provide the certification that the tuition dollars supposedly buy.
But MOOCs explode these constraints, even in courses like the one that Martin describes where remediation of students on a for-credit path is the primary goal of the course design. Because the barrier to entry is so low—zero cost, zero travel or scheduling demands, and zero consequence for dropping out in the middle—you will get students in the class with substantially different goals, including many that do not care about certification at all. This is something that Bob Hoar from the University of Wisconsin taught me at the recent MOOC conference when we chatted about a remedial math MOOC that was similar to the one that Martin wrote about in his post. I made a comment to the effect that his MOOC would probably attract a much more traditional and homogenous group of students than, say, a MOOC about the science of cooking. Oh no, he replied. Actually, some of their registrants included parents of students taking the course who wanted to help their kids. Others were adults who were not UW students but had always struggled with math and wanted to finally get it right.
In an ideal world, we would design our courses to explicitly support goals like these. And we would design our analytics to account for the different goals when we try to measure the “success” of the course. The good news is that massive, technology-enabled courses not only enable us to create different paths for different students; they almost force us to do so. One of the most transformative aspects of distance education in general and MOOCs in particular is that these modalities challenge us to look for pedagogically effective alternatives to the control that faculty can assert in a face-to-face class and that they can’t assert online. But in order to really learn from this forcing function, we need to go beyond designing solely for course goals and explicitly design for student goals.
Differentiated Engagement
To start with, we need some common language to talk about this new design ethic. Mike Caulfield put me onto a term from feminist pedagogy called “differential participation.” While I found the language to be provocative, once I dug into the details behind the idea I found that the motivation for it is different from mine. Differential participation is about the different power relationships among participants. I’m more interested in talking about the goals of different learners. So the term I’m playing with now is “differentiated engagement.” For starters, I like “differentiated” rather than “differential” because this isn’t about more or less, better or worse. It’s about personalization. And I prefer “engagement” over “participation” for similar reasons. I’m not interested in how much or how well a student participates. I’m interested in how and why a student is engaged. We should have a course design methodology that creates courses which invite students to engage with the content and activities in ways that are consistent with their personal goals. Such a design should also, at least in theory, enable us to distinguish between a student whose non-participation is consistent with her goals for the course and one whose non-participation is in conflict with her goals for the course. Differentiated engagement.
The next thing we need is a methodology that enables us to do this sort of design work. And it turns out that we have a pretty good model in software design. Software designers often create personas representing particular types of users as one of the first steps in their product designs. But we need to use these tools correctly. While I have seen personas used in course design exercises before, they generally are deployed in order to identify what students need in order to achieve the course goals rather than to refine our notion of what the student’s goals are:
Dmitri is an avid soccer player who struggles with math and would prefer being on the field to doing his homework. He wants to complete his homework as quickly as possible, and is often a little sloppy about getting it done in his eagerness to get back outside.
While this little snapshot does tell us something about Dmitri’s goals, the focus is on getting him to do what he doesn’t want to do rather than starting by figuring out what he does want to do that is directly related to why he is in the class. We should start instead by identifying the student’s affirmative goals for the course, which entails acknowledging those goals as legitimate in some important sense and our responsibility, in part, as helping Dmitri to meet these goals. If his goal is to just get through the class with as little pain as possible so he can graduate, then whatever else we try to do for Dmitri, we should help him to get through the class with as little pain as possible so that he can graduate.
This does not mean we should abandon the idea of overarching course goals or the responsibilities that those goals are intended to meet, but it does mean that we should stop relying on the crutch of course credit to force students to embrace our course goals as their own. Phil and I have been thinking about this challenge a lot as we design e-Literate TV, which is somewhat MOOC-like in its ambitions. Our goal is provoke conversations on campus that will lead to better, more consensus-driven decisions about how to deploy technology in the service of improving education. That said, while we don’t expect a lot of people to disagree with that goal in principle, we also don’t expect it to be the immediate motivator in a lot of cases. The immediate motivator is more likely to be the Board of Trustees telling the President that he has to get on the MOOC bandwagon (or get off it), or a faculty member with a labor concern preparing for a conversation at a faculty senate meeting, or a CET director feeling cut out of the decision loop. In other words, everybody has their own problems to solve. We can’t give course credit for e-Literate TV. The only way that we can bring people in is to provide them with something that will hopefully help them meet their goals. And then, through our course design, we hope to show them that the best, most satisfying way to meet their goals is to embrace ours as well.
I believe that we should be employing the same approach in traditional course designs. It seems obvious to say that students who actually have intrinsic motivation to engage with our courses will tend to learn better than the ones who are doing the work primarily because they are being told that they have to eat their vegetables or they can’t have any dessert. On a personal level, many teachers do work very hard to engage their students through their day-to-day interactions with them. But that work often stops where course design begins. There will always be a certain “eat your vegetables” aspect of schooling as long as schools are in the business of certifying knowledge, which is one reason why open courses are such an important addition to our tool set. But even within the bounds of a traditional college education, we can do much better at accomplishing our goals for our students by helping them to accomplish their goals for themselves.
Thanks for the link Michael. You are quite right, the mindset of course goals we have is quite difficult to overcome. I really like the idea of differentiated engagement. The 2nd, non-linear model might go some way to addressing this – the learner can take as many of the themes as they like in whatever order. Of course, there is an issue in that many topics do need to be taught linearly – you can’t understand Y until you’ve understood X. And as we know, learners aren’t always in the best position to know what it is they need to know (Meno’s paradox). But we try to introduce some of this design thinking in our learning design approach. Yishay Mor’s OLDS MOOC is good on this too.
You’re bringing up the old learning object modularity problem. We want to make content infinitely re-usable, but we find that context sets boundaries on re-usability. A lot of the old learning object talk was about re-use for course design, but it also pertained to re-use in different learner-defined paths through content. We try to chunk up the content as finely as we can so that we don’t force choices on learners. The OER twist on this is to let people modify the resources to meet their needs, although it is often a more author- and teacher-oriented strategy than a student-oriented one.
Interestingly, this is sort of the opposite of the approach that good software designers often take. In software design, the emphasis days is on trying to understand the needs of a specific class of users in as much detail as you can and to keep testing until you know that you have met their needs. Then you expand your product vision to address the needs of a different class of users.
Sophisticated design generally requires a balance between creating something that opens up possibilities for the users to choose themselves and understanding the users well enough to anticipate their needs. It’s a both/and rather than an either/or. But in course design, the heritage of theorists like John Dewey has led to pedagogical philosophies such as Constructivism and Connectivism being positioned in opposition to prescriptive approaches, which tends to lead the conversation away from approaches that may be prescriptive while also being personalized.
I should add that both “learning styles” (whether real or mythical) and adaptive learning software have brought differentiated student paths into the conversation at various times. But neither really addresses differentiated engagement, at least in the ways that they have been commonly applied.
You know, I think a recent MOOC I took from U of Edinburgh (#edcmooc) does this differentiated engagement thing. First off, all content available from day 1. You are free to read and watch whichever of the readings/videos you like. You can use twitter, discussion forums, blogs, etc. to engage – your choice. There were google Hangouts and live twitter chats for folks who prefer synchronous interaction. The final peer-reviewed digital artefact project gave lots of freedom within loose guidelines related to the course.
On another note, here is the link i posted on Twitter about curriculum theory (was not able to post it on the blog y/day). Basically, most people are used to designing curricula/courses as content or product… Whereas a focus on learner engagement is more of a process approach (often taken by individual teachers but almost never insitutionally). Your idea of “differentiated engagement” siunded to me like another level of process curriculum than what is described here (link below) and moves a little bit towards a critical approach because it recognizes differences among learners and how it can affect their learning process. Anyway, here is the link (in case someone wants to see it but has not seen it on Twitter):
http://infed.org/mobi/curriculum-theory-and-practice/
Maha, part of what I’m saying here is that giving learners lots of options and letting them choose the kind of engagement that they want is *not* the same as designing for differentiated engagement. What I’m advocating for is research into what the goals of the students might be in advance of the course design and creating features of the course experience that are specifically designed to support those goals, as well as metrics that can differentiate students by goals and track their “success” in the course based on those differentiated goals.
Use cases, imagine that!
Seriously, though, I think you’ve really channeled the discussion here in a useful direction. I’ve been fighting for four or five years to get people to realize that designing things that are reused is not the same thing as designing for reuse (and fighting for about two years to get people to realize that designing MOOCs that are reused for blended is not the same as designing MOOCs to be reused for blended). But I think this nicely dovetails that problem with the assessment problem.
Meaning this — I get so frustrated when people say the dropout problem “doesn’t matter”. And I get so frustrated when people bash MOOCs about their dropout rates, saying they *do* matter. And the reason I do is what you’ve nicely put forward here — none of it is grounded in any idea of intentional design. People will defend dropout rates and bring up an anecdote of this person or that person. But an anecdote is not a use case (or in agile dev a “user story”). Use cases guide design, they don’t happen after the fact.You test the system against them, and evaluate the system against them.
So when people say — hey, dropout rates don’t matter, they are alluding (in the better cases) to the problem of differentiated engagement. But they are not taking it into the heart of the design process, so it remains a lousy excuse, not an insight or principle. THAT’s what we need to push — if we get that in place, then the classroom reuse piece just becomes another set of user stories/use cases that can actually GUIDE DEVELOPMENT. And it’s a principle which intersects quite a bit with software design, so maybe it will be heard above the din?
I think there is a fairly good chance that, in the long run, course design techniques will catch some of the wind from Agile and Lean product development techniques. And one of the drivers may turn out to be the relative failure of learning analytics. At some point, it’s going to dawn on people that all these expensive systems that are being sold to them provide relatively low value because they can’t measure some of the things that really matter–not only because some of those things are fundamentally unmeasurable (although some of them are), but also, and perhaps mostly, because we are not asking the software the right questions or giving it the right inputs.
We’ve also pushed product testing down to the wrong level of the system. Teachers should be keeping broad measures of student performance, of course. But product producers have done a relatively lousy job of testing products and techniques. We would be much better off with fewer, better designed tests that tested outcomes at an expensive level of detail over years than the 20,000 crappy experiments we seem to be running.