What is a retention early warning system? What is it good for? What are its limitations? And how are its failings representative of the unfulfilled potential of so many ed tech products? You’ve got questions, we’ve got answers.
Course Signals
68 Percent of Statistics Are Meaningless, Purdue University Edition
Purdue University has been happy to take credit for essentially inventing the retention early warning analytics category. But are they willing to take responsibility for putting out a flawed efficacy study? Eh, not so much.
Purdue University Has an Ethics Problem
It’s fair to say that Purdue University has sparked several important conversations in ed tech through their work on Course Signals. First, they pretty much put the retention early warning system as a product category on the map, conducting ground-breaking research and building a system that several major ed tech players have either licensed or […]
Comments from a Researcher on the Course Signals Kerfluffle
Doug Clow of the Open University has published a thoughtful and detailed blog post in response to the Course Signals effectiveness controversy. He covers far too much ground for me to attempt to summarize here, but I think there are some common themes emerging from the commentary so far: The concerns over the one study have […]
Purdue’s Non-Answer on Course Signals
Inside Higher Ed’s Carl Straumsheim has some reporting on the Course Signals data controversy. He was able to get Purdue research scientist Matt Pistilli on record about it. Here is the sum total of the quotes from the article: Pistilli defended the claims about Signals’ ability to increase retention — with the caveat that more […]
Course Signals Effectiveness Data Appears to be Meaningless (and Why You Should Care)
My father likes to say, “If you stick your head in the freezer and your feet in the oven, on average you’ll be comfortable.” Behind this pithy saying is an insight that is a little different from the “three kinds of lies” saying about statistics. It suggests that certain types of analysis produce a false […]
Digging into the Purdue Course Signals Results
Update: Mike has written another post clarifying the intuitions behind his math. The spectacular Mike Caulfield casts a skeptical eye on the Course Signals data: Only a portion of Purdue’s classes are Course Signals classes, so the chance any course a freshman takes is a Course Signals course can be expressed as a percentage, say 25%. […]