e-Literate

Present is Prologue

Category: Learning Analytics

  • Yes, I did say that Knewton is “selling snake oil”

    My comments in today’s NPR article about Knewton are getting some attention on Twitter. One comment in particular, actually. The one where I accused Knewton CEO Jose Ferriera of selling snake oil. To understand the basis of the comment, you have only to look at the quote from Jose earlier in that selfsame article:

    We think of it like a robot tutor in the sky that can semi-read your mind and figure out what your strengths and weaknesses are, down to the percentile.

    Can it read what I’m thinking now?

    And then there’s this:

    We can take the combined data power of millions of students — all the people who are just like you — [who] had to learn a particular concept before, that you have to learn today — to find the best pieces of content, proven most effective for people just like you, and give that to you every single time.

    Really? Every single time?

    Objectively speaking, I don’t know much about Knewton’s platform or the value that it adds. The only efficacy studies I have seen are for Pearson products, and those studies do not differentiate between Knewton features and Pearson-native features in terms of impact. If you dig deep enough on their site, you can find some information on the techniques that they use. For example, this overview paper, while not perfect, is a credible attempt to explain the techniques that the platform employs and the benefits to the users in layperson’s terms. If Knewton were doing more of this, I wouldn’t be as critical. There are still questions about the value of the product, but they are on the same level with many other products in the adaptive learning category. A company like Knewton should be working hard to demystify what their product does and providing hard, reproducible evidence that they add value.

    But much of what Jose says, at least to the media, is the opposite. No responsible educator or parent should adopt a product—even if it is free—from a company whose CEO describes it as a “robot tutor in the sky that can semi-read your mind” and give you content “proven most effective for people like you every single time.” I’m sorry, but this sort of quasi-mystical garbage debases the very notion of education and harms Knewton’s brand in the process.

    If you want to sell me a product that helps students to learn, then don’t insult my intelligence. Explain what the damned thing does in clear, concrete, and straightforward language, with real-world examples when possible. I may not be a data scientist, but I’m not an idiot either. If you can translate the technical mumbo jumbo into sensible teaching insights for me, then you just might make a sale. But most people wouldn’t buy a used car from somebody who describes their product using language as hyperbolic as “robot tutors in the sky.” The same principle should hold for educational technology.

  • 68 Percent of Statistics Are Meaningless, Purdue University Edition

    I don’t know of any other way to put this. Purdue University is harming higher education by knowingly peddling questionable research for the purpose of institutional self-aggrandizement. Purdue leadership should issue a retraction and an apology.

    We have covered Purdue’s Course Signals extensively here at e-Literate. It is a pioneering program, and evidence does suggest that it helps at-risk students pass courses. That said, Purdue came out with a later study that is suspect. The study in question claimed that students who used Course Signals in consecutive classes were more likely to see improved performance over time, even in courses that did not use the tool. Mike Caulfield looked at the results and had an intuition that the result of the study was actually caused by selection bias. Students who stuck around to take courses in consecutive semesters were more likely to…stick around and take more courses in consecutive semesters. So students who stuck around to take more Course Signals courses in consecutive semesters would, like their peers, be more likely to stick around and take more courses. Al Essa did a mathematical simulation and proved Mike’s intuition that Purdue’s results could be the result of selection bias. Mike wrote up a great explainer here on e-Literate that goes into all the details. If there was indeed a mistake in the research, it was almost certainly an honest one. Nevertheless, there was an obligation on Purdue’s part to re-examine the research in light of the new critique. After all, the school was getting positive press from the research and had licensed the platform to SunGard (now Ellucian). Furthermore, as a pioneering and high-profile foray into learning analytics, Course Signals was getting a lot of attention and influencing future research and product development in the field. We needed a clearer answer regarding the validity of the findings.

    Despite our calls here on the blog, and our efforts to contact Purdue directly, and attention the issue got in the academic press, Purdue chose to remain silent on the issue. Our sources informed us at the time that Purdue leadership was aware of the controversy surrounding the study and made a decision not to respond. Keep in mind that the research was conducted by Purdue staff rather than faculty. As a results, those researchers did not have the cover of academic freedom and were not free to address the study on their own without first getting a green light from their employer. To make matters more complicated, none of the researchers on that project still work at Purdue anymore. So the onus was on the institution to respond. They chose not to do so.

    That was bad enough. Today it became clear that Purdue is actively promoting that questionable research. In a piece published today in Education Dive, Purdue’s “senior communications and marketing specialist” Steve Tally said

    the initial five- and six-year raw data about the impact of Signals showed students who took at least two Signals-enabled courses had graduation rates that were 20% higher. Tally said the program is most effective in freshman and sophomore year classes.

    “We’re changing students’ academic behaviors,” Tally said, “which is why the effect is so much stronger after two courses with Signals rather than one.” A second semester with Signals early on in students’ degree programs could set behaviors for the rest of their academic careers.

    It’s hard to read this as anything other than a reference the study that Mike and Al challenged. Furthermore, the comment about “raw data” suggests that Purdue has made no effort to control for the selection bias in question. Two years after the study was challenged, they have not responded, not looked into it, and continue to use it to promote the image of the university.

    This is unconscionable. If an academic scholar behaved that way, she would be ostracized in her field. And if a big vendor like Pearson or Blackboard behaved that way, it would be broadly vilified in the academic press and academic community. Purdue needs to come clean. They need to defend the basis on which they continue to make claims about their program the same way a scholar applying for tenure at their institution would be expected to be responsible for her claims. Purdue’s peer institutions likewise need to hold the school accountable and let them know that their reputation for integrity and credibility is at stake.

  • Unizin Perspective: Personalized learning’s existence and distance education experience

    By reading the Unizin pitch for the State University System of Florida shared yesterday, we can see quite a few claims about the (potential) benefits to be provided by the consortium. “Make sure that the universities were not cut out of [distance ed] process”; “Secure our foothold in the digital industry”; “Promote greater control and influence over the digital learning ecosystem”; Provide “access to the Canvas LMS at the Unizin price”; Provide “access to tools under development, including a Learning Object Repository and Learning Analytics”; Provide “potential for cooperative relationships to ‘share’ digital instruction within and across the consortium”.

    I want to pick up on University of Florida provost Joe Glover’s further comment on Learning Analytics, however.

    The third goal for Unizin is to acquire, create, or develop learning analytics. Some of the learning management systems have a rather primitive form of learning analytics. Unizin will build on what they have, and this will go from very mechanical types of learning analytics in terms of monitoring student progress and enabling intrusive advising and tutoring; all the way up to personalized learning, which is something that really does not exist yet but is one of the objectives of Unizin.

    (more…)