I almost never quote a blog post in its entirety, but this one from Dan Meyer is so good that I just can’t bear to cut a single word:
Stephanie Simon, reporting for Reuters on inBloom and SXSWedu:
Does Johnny have trouble converting decimals to fractions? The database will have recorded that – and may have recorded as well that he finds textbooks boring, adores animation and plays baseball after school. Personalized learning software can use that data to serve up a tailor-made math lesson, perhaps an animated game that uses baseball statistics to teach decimals.
Three observations:
One, it shouldn’t cost $100 million to figure out that Johnny thinks textbooks are boring.
Two, nowhere in this scenario do we find out why Johnny struggles to convert decimals to fractions. A qualified teacher could resolve that issue in a few minutes with a conversation, a few exercises, and a follow-up assessment. The computer, meanwhile, has a red x where the row labeled “Johnny” intersects the column labeled “Converting Decimals to Fractions.” It struggles to capture conceptual nuance.
Three, “adores” protests a little too much. “Adores” represents the hopes and dreams of the educational technology industry. The purveyors of math educational technology understand that Johnny hates their lecture videos, selected response questions, and behaviorist video games. They hope they can sprinkle some metadata across those experiences — ie. Johnny likes baseball; Johnny adores animation — and transform them.
But our efforts at personalization in math education have led all of our students to the same buffet line. Every station features the same horrible gruel but at its final station you can select your preferred seasoning for that gruel. Paprika, cumin, whatever, it’s yours. It may be the same gruel for Johnny afterwards, but Johnnyadores paprika.
Dan captures most of what I was trying to get at with my rant on the big data hype, but much more clearly and succinctly. Points two and three are the most salient here. First of all, the sort of surface-level analysis we can get from applying machine learning techniques to the current data we have from digital education system is insufficient to do some of the most important diagnostic work that real human teachers do. Think about the math classes is which you had to show your work on your homework. Why was that important? Because the teacher needs to see not only what you got wrong but why you got it wrong. Teachers generally don’t just say, “You got three out of five problems involving converting decimals to fractions wrong. Go study some more.” They sit down and work through the problems with the student to find the source of the errors. It’s really hard to get computers to do this well, even with highly procedural domains like math. (Forget about, say, literary analysis.) So in the vast majority of cases, we don’t even try to design systems where students show their work. And without the step-by-step data, no fancy algorithm is going to teach Johnny.
Second, if the problem is that your content isn’t what the student needs, no fancy algorithm is going to fix that either. Videos are a prime example. I know of one textbook publisher whose teacher customers report that students won’t watch the publishers’ videos, but they can and do find videos on the same topic on YouTube and share them with each other. Think about that. Video-based pedagogical support is valuable enough to the students that they will expend energy searching for videos and sharing them. But they reject the expensive, carefully crafted videos from the publisher that are served up to them on a silver platter. It’s not that the publisher-supplied videos are necessarily “bad” in the sense that they have poor production qualities or are unclear or factually inaccurate. But the students have a particular use in mind for the videos. Maybe they’re struggling with a particular homework problem and just need a quick walk-through of a technique so that they can see the step that they are missing, for example. If the video doesn’t fit their needs—both utilitarian and aesthetic—then it won’t get used. Serving it up adaptively isn’t going to help that problem.
That said, it’s worth taking a little time to break down the different types of adaptive learning analytics into a couple of categories and see just what we should and should not reasonably hope to gain from them.