Where xMOOCs and Adaptive Analytics Both Fail (For Now)

No, this isn’t just an attempt to cram as many sexy keywords into one post title as possible. xMOOCs and adaptive analytics share an ambition: They both are at least partially motivated by a desire to teach at scale. With MOOCs, the goal is obvious. With adaptive analytics, less so, partly because there are multiple motivations and maybe because the desire for scale is not something that is polite to talk about due to a certain amount of discomfort with it. But the motivation is definitely there if you look closely, as I’ll get into in a bit.

The problem is that both of these approaches, in their current incarnations, miss one absolutely critical element of the teaching process. As you can probably guess from all the qualifiers I am using in my language, I don’t think it’s an inherent or permanent failing. But I worry that it’s a failing due to a deep cultural blind spot that we have about what education is, and that it therefore will be challenging to address.

The Case of the Missing Viola Player

There has been a lot of talk recently about the problem of Baumol’s Disease in education. William J. Baumol and his colleague William G. Bowen wrote about the problem of business sectors where costs go up but productivity doesn’t. They used the example of a Beethoven string quintet. You can’t play it with less than five musicians, but musician’s salaries go up because musicians always have the option to get a better paying job in another business. The music industry has to keep raising salaries at least enough to keep qualified musicians performing. So the cost of the performance continues to rise while the income per performance does not. Once a concert hall is full, it’s full. In that case, either ticket prices have to go up or the concert hall will go out of business.

Baumol’s Disease is an imperfect diagnosis of the cost problem in education because there is no evidence of wage inflation among faculty. The AAUP released a study this year that shows, among other things, that tuition at public two-year colleges have risen by nearly 45% above the rate of inflation over the past decade while faculty salaries have declined by 2.5 percent in real terms. The gap between change in tuition and change in faculty salary is even worse at four-year public universities. The smallest gap was at private four-year universities, but even there, tuition rose by about 29% above inflation while faculty salary increases ranged from 1.9 to 7.7 percent.1

Clearly, the aspect of Baumol’s Disease that people who are interested in education latch onto is the notion of the limit on productivity, which in this case is pretty much the same thing as the limitation on class size. One teacher can only teach so many students at a time. If you could increase the number of tuition-paying students, the theory goes, then rising income would keep up with rising costs and tuition wouldn’t have to go up. and if you can increase the number of tuition-paying students even more, then tuition could actually go down.

The first problem with this logic is that, as I just pointed out, faculty salary inflation does not appear to be the cause tuition inflation. It is possible that other scaling limitations would still make the analogy valid. For example, it may be that one of the principal causes of inflation is maintenance of the dormitories, classrooms, dining halls, etc., which, in turn, limit the number of tuition-paying students you can have. In that case, you could have a Baumol-like syndrome, by which I mean that there is a limit on productivity which is also the cause of a rise in cost. On the other hand, the cost increase could also be related to things like investment in research labs, upgrades to “student life” provisions (like better food in the dining halls or a climbing wall in the gym), or increased cost of administrative software. These problems would not be productivity-related.

But let’s leave that concern aside for the moment. Regardless of the ultimate cause of tuition inflation, it would be a Good Thing to figure out how to reach more students (cost-)effectively. And it’s certainly true that, regardless of the cause of tuition inflation, the ability to teach more students could be part of the solution. This is where MOOCs typically get invoked. It’s the whole “massive” thing. If you have 160,000 students in a class (as Sebastian Thrun and Peter Norvig did in their first course), then you don’t need to collect a lot of tuition per student at all to cover an awful lot of cost. Ten dollars per student gives you $1.6 million. From one class. We have solved the productivity problem. Hurrah!

Alas. If only it were that easy. There is a little bit of South Park underpants gnomes logic here:

The aspect of Baumol’s Disease that we tend to ignore is ironically the most obvious one: The reason that you can’t play a Beethoven quintet with four musicians is because there are five parts. While the cMOOC community has at least attempted to provide a theory of pedagogy to explain how they can achieve the “M” in “MOOC,” I have yet to see a serious account by any of the xMOOC providers to tell us how they think they can play a quintet with only four musicians. What happened to that viola player?

In Search of the Missing Melody

Let’s break the class experience down and see if we can find which parts transfer. First, there’s the content transmission. In the traditional model of the classroom, that’s lectures and readings. Readings scale. Traditionally, lectures have not. The biggest lesson that Sal Khan has taught the world is how incredibly cost-effective it can be to scale the lecture with the tools we have today, particularly if we don’t let ourselves get hung up on production values that have nothing to do with learning.

Second, there’s assessment. This is harder to scale. It’s only a little harder if you’re content with multiple choice questions, moderately harder if you want to see students actually solve problems that have cut-and-dry right answers, and a lot harder if you’re trying to assess the student’s thought process and writing skills. xMOOCs are definitely doing assessment. There’s an open question about how well they are doing assessment—I think it varies from subject to subject and class to class—but they are doing it.

Third, there’s remediation.

Ah.

Almost every student gets stuck from time to time. Individual students will get stuck more or less often, at different places, over different things. When I think back on the best teachers I had when I was in school, the ones that leap to mind first and the ones who identified exactly what I was struggling with and helped me past the road block. “Remediation” isn’t quite the right word; the student could be trying to work through a problem that is far in advance of where the rest of the class is. The point is that the zone of proximal development is a real thing. There are places in every student’s learning path where she can learn things with help that she can’t learn without help. These places are individual to the student at that particular moment in the student’s intellectual growth. Therefore, one highly critical element of good teaching is being able to provide highly individualized attention to the student at critical moments.

This is very hard to scale for several reasons. First, by definition, you can’t just give the same thing to every student. Second, you need to be able to identify not only what a student is getting wrong on an assessment but why she is getting it wrong. The field of Intelligent Tutoring Systems (ITSs) has been giving this challenge a lot of attention for the last several decades now. I wrote about it in my post about LearnLab at Carnegie Mellon University. (The “cognitive tutors” I described are a subspecies of ITSs.) In Kurt VanLehn’s outstanding overview of best practices in ITS design, he describes them as having outer loops and inner loops. The outer loop is more or less the same that you get from any testing system. Did the student get the correct answer? The inner loop looks at the steps that the students took to get the answer and responds to them step-by-step, catching them at the moment they make a mistake and providing feedback that is directly relevant to that mistake. In other words, they do what a teacher or tutor would do.

Well-designed ITSs do work, but they are notoriously difficult to build, typically requiring both a great deal of expertise and a great deal of time. There are a couple of tools, like CTAT and Assistments, that attempt to lower the bar, but it is not yet clear that anyone has come close to cracking the nut of scaling and democratizing production of ITSs. And even if they do, ITSs are mostly proven for problems that require relatively clear-cut procedural solutions. Assessing and remediating the flaws in a student’s analysis in a history paper is another matter entirely.

So here’s an area in which a an entire sub-discipline dedicated to solving this problem is still struggling with how to scale production after many years of trying. Has the xMOOC contingent even addressed the remediation problem? A little bit, around the edges. At different times I have heard both Coursera’s Daphne Koller and Udacity’s Peter Norvig talk about sometimes being able to deduce where the student went wrong based on the specific wrong answer she gave. There’s something to that, but I’m not convinced that it’s more than an edge case that works from time to time for a handful of disciplines.

Update: I should say a word here about Coursera’s peer review tool, which does attempt to provide remediation as well as assessment. I think that calibrated peer review is an interesting approach that shows much promise for scaling remediation. But feedback on Coursera’s tool when used in actual courses has been mixed at best so far. Designing an effective peer review solution, with quality rubrics, the right incentives for students, etc., is hard, and there is a lot we don’t know yet about what works and what doesn’t. But to be fair, I should acknowledge Coursera’s significant effort to support remediation.

I’m not being entirely fair to the xMOOC providers. Later in this post, I will talk about an approach that some schools are using with xMOOCs to address the inner loop problem. But in all the MOOC mania, it’s astonishing that this largely unsolved problem is often being papered over. How is that possible?

The Viola Player Is Dead. But Who Killed Him?

If we are honest with ourselves, we have to admit that the viola part has been missing since long before MOOCs showed up on the scene—so long, in fact, that the music scores handed out to our young musicians as they take their roles in the faculty quinquartets are actually missing the part. Young instructors are never told that they need a viola player. They are never given any training in pedagogy. And they certainly aren’t given time or resources to provide individualized help. When I was in college in the 1980s, I sat in my share of 300-person lectures. Since then, the pressure to have larger classes and to teach more classes per semester has only risen, particularly in public colleges and universities. My guess is that college professor productivity has risen in the last decade, if all you mean by “productivity” is number of butts in classroom seats per teacher. The cost has been less time to respond to individual student needs.

In some ways I’m stretching the quintet analogy to the breaking point. The fact is that the lecture, sage-on-the-stage model of teaching is probably as old as college education itself, and maybe even older. This is one reason why I worry that fixing this problem as we attempt to scale will be difficult. It’s been a problem for so long that a lot of folks in academia can’t even see it as a problem. It’s just normal for them. And they’ve been aided and abetted in seeing the world this way by another industry player that has been around a lot longer than the xMOOC providers. I’m talking about the textbook industry.

As the time pressure on faculty increased, the textbook industry stepped in to solve their problems for them. Don’t have time to gather readings for your class? No problem; we’ll give you set of curated and editing readings that you can use out-of-the-box. Don’t have time to prepare lectures? No problem; we have slides that are consistent with the readings, notes on the slides, and questions you can ask your students during class. Don’t have time to make and grade tests? No problem; we’ll provide you with a test engine. Every time the pressure on faculty increased, the textbook industry was there to say, “Don’t worry about that part. We’ll do it for you.” A lot of what has shaped the textbook products in the U.S. higher education market over the last decade or more is essentially this unspoken agreement to hand off increasing amounts of the work involved with teaching to the publishers.

Let me be clear: I don’t see anything wrong or immoral about the idea of an industry that exists to solve teacher problems. The issue is that we haven’t had the right incentive structures to create an industry that exists to solve student problems. The solutions I listed above might solve student problems, but they are designed to solve teacher problems because teachers select the textbooks. And the main teacher problem that textbook publishers in the U.S. higher education market are paid to solve is increasing the throughput of the number of students seen by one faculty member in a semester.

This dynamic is beginning to change, for two reasons. First, students have become very clever about avoiding buying expensive textbooks, so publishers are having to work harder to create value that can’t be easily replaced by Wikipedia or Khan Academy (or the video lectures in a MOOC). Second, college and university funding is increasingly tied to outcomes, which means that a textbook publishers who can show that students learn better with their materials will be more likely to win the adoption. The transition is slow, but it’s happening. This is probably one reason for the blossoming of “adaptive learning” or “personalized learning” products. Unfortunately, I don’t see much evidence that many of these new products have an inner loop. It’s one thing to be able to say, “I see you’re still having troubles with fractions, so I’m not going to move you on to percentages yet.” It’s quite another to be able to say, “I see where you’re having trouble with fractions, so I’m going to give you instruction specific to the misconception you have about fractions that’s getting in your way.” There simply is no magic technology bullet to solve that problem yet. So I worry that, as good as the adaptive products may be at what they do well, we’re not going to notice that the music they are playing is simply not that missing viola part.

That said, all is not lost.

“It just so happens that your friend here is only mostly dead.”

Phil Hill recently pointed me to this video testimonial regarding San José State University’s experiment doing a flipped classroom using an edX course. (Sorry; it’s not embeddable. You’ll have to follow the link to see it.) As you might expect, the class basically used the MOOC for the lectures and assessments and did the remediation in a physical classroom. That’s the point of a flipped class. A traditional classroom forces the professor to put most of the work into the stuff that’s easy to scale and leaves little time for stuff that’s hard to scale. The flipped classroom…uh…flips that. When students run into problems, they are not alone. They are with an expert who can help them. They are also with their peers, who often can help them too. And a lot of the time, they don’t need help. In a lecture, the teacher spends the entire class period focusing equal amounts of attention on those students who need help and those who don’t. In a flipped classroom, the teacher spends most of her time with students who need her help and very little time with the students who don’t. This is a big adjustment for some instructors to make, but it can be done. In the SJSU video, I was particularly heartened to hear the professor speak so enthusiastically about his first experience with this kind of teaching and the new role he found himself in as coach rather than lecturer.

I don’t think it’s much of an exaggeration to say that the fate of higher education in the United States may depend on the ability of faculty to embrace this change. In the music of the classroom, the four parts that we are listening to today are increasingly playable by robots. If you look at this through the lens of disruptive innovation, the textbook publishers provide the…uh…textbook case of a potential disruptor. They have been working their way up the classroom food chain, progressively taking over the parts of the work that faculty don’t have time to do (or don’t want to do). MOOCs are rapidly covering similar ground. At some point, somebody is going to ask, “What do we need the instructor for?” Faculty could be replaceable in the quartet.

But that fifth part is a bitch for the robots. In my opinion, it also happens to be one of the most valuable aspects of the teaching process. I remember teachers I had who gave fantastic lectures. I also remember teachers who helped me to grow beyond what I could have gotten to by listening to lectures, no matter how fantastic they were. When I ask myself which of these two types of teachers had a bigger impact on my life, it’s not even a close contest. For the good of their students as well as their own career security, I very much hope that more faculty will pick up that viola and start practicing.