e-Literate

Present is Prologue

Tag: Matt Pistilli

  • NPR and Missed (Course) Signals

    Anya Kamenetz has a piece up on NPR about learning analytics, highlighting Purdue’s Course Signals as its centerpiece. She does a good job of introducing the topic to a general audience and raising some relevant ethical questions. But she missed one of the biggest ethical questions surrounding Purdue’s product—namely, that some of its research claims are likely false. In particular, she repeats the following claim:

    Course Signals…has been shown to increase the number of students earning A’s and B’s and lower the number of D’s and F’s, and it significantly raises the chances that students will stick with college for an additional year, from 83% to 97%. [Emphasis added.]

    Based on the work of Mike Caulfield and Al Essa summarized in the link above, it looks like that latter claim is probably the result of selection bias rather than a real finding. So who is at fault for this questionable claim being repeated without challenge in a popular venue many months after it has been convincingly challenged?

    For starters, Purdue is. They never responded to the criticism, despite confirmation that they are aware of it—for one thing, they got contacted by us and by Inside Higher Ed—and despite the fact that they apparently continue to make money off the sales of the product through a licensing deal with Ellucian. And the uncorrected paper is still available on their web site. This is unconscionable.

    Anya clearly bears some responsibility too. Although it’s easy to assume from the way the article is written that the dubious claim was repeated to her in an interview by Purdue research Matt Pistilli, she confirmed for me via email that she took the claim from the previously published research paper and did not discuss it with Pistilli. Given that this is her central example of the potential of learning analytics, she should have interrogated this a little more, particularly since she had Matt on the phone. Mike Caulfield also commented to me that any claim of such a dramatic increase in year-to-year retention should automatically be subject to additional scrutiny.

    I have to put some blame on the higher ed press as well. Inside Higher Ed covered the story (and, through them, the Times Higher Education). In fact, Carl Straumsheim actually advanced the story a bit by putting the question to researcher Matt Pistilli (who gave a non-answer). The Chronicle of Higher Education did not cover it, despite having run a puff piece on Purdue’s claims the same day that Mike Caulfield wrote his original piece challenging the results. It is very clear to Phil and me that we are read by the Chronicle staff, in part because they periodically publish stories that have been obviously influenced by our earlier coverage. Sometimes without attribution. I don’t care that much about the credit, but if they thought Purdue’s claims were newsworthy enough to cover in the first place then they should have done their own reporting on the fact that those claims have been called into question. If they had been more aggressive in their coverage then the mainstream press reporters who find Course Signals will be more likely to find the other side(s) of the story as well. Outside of IHE, I’m having trouble finding any coverage, never mind any original reporting, in the higher ed or ed tech press.

    I have a lot of respect for news reporters in general, and I think that most people grossly underestimate how hard the job is. I think highly of Anya as a professional. I like the reporters I interact with most at the Chronicle as well. Nor will I pretend that we are perfect here at e-Literate. We miss important angles and get details wrong our fair share. For example, I doubt that I would have caught the flaw in Purdue’s research if Mike hadn’t brought it to my attention. But collectively, we have to do a better job of providing critical coverage of topics like learning analytics, particularly at a time when so much money is being spent and our entire educational system is starting to be remade on the premise that this stuff will work. And there is absolutely no excuse whatsoever for a research university to not take responsibility for their published research on a topic that is so critical to the future of universities.

  • Purdue’s Non-Answer on Course Signals

    Inside Higher Ed’s Carl Straumsheim has some reporting on the Course Signals data controversy. He was able to get Purdue research scientist Matt Pistilli on record about it. Here is the sum total of the quotes from the article:

    Pistilli defended the claims about Signals’ ability to increase retention — with the caveat that more research needs to be done. “The analysis that we did was just a straightforward analysis of retention rates,” he said. “There’s nothing else to it.”

    To ensure an empirically grounded analysis of Signals, Essa urged Purdue to give researchers access to as much data as possible. Pistilli said he is open to participate in that conversation, but pointed out that granting open access could violate students’ privacy rights.

    With Signals marking its fifth anniversary this year, Pistilli said “it was probably just a matter of time for people to start looking for these pieces and begin to draw conclusions.” In that sense, the discussion about early warning systems resembles that of other ed-tech innovations, like flipping the classroom and massive open online courses, where hype drowns out any serious criticism.

    Now, I understand that interviews necessarily get edited down for news articles, so I am not going to jump to any deep conclusions about Dr. Pistilli’s views or Purdue’s official positions from these three short paragraphs. But I will make two points that do not require a close reading.

    1. Purdue’s credibility is on the line.

    Purdue has made significant noise about their retention claims. They have published articles and made presentations. They have commercially licensed their system for use by other schools, based in part on the strength of those claims. We now have credible analysis calling some of those claims into question. For its own sake, as well as for that of the academic community, Purdue now needs to go on record with a response to the critique, either acknowledging its legitimacy and amending their claims or demonstrating why the analysis is off base. There is nothing in the quote above to indicate that either Dr. Pistilli or the university understand that they have a substantial problem which demands a substantive answer.

    2. Purdue would not have to violate student privacy in order to answer the concerns being raised.

    Course Signals collects substantial fine-grained data about student activity, some of which could be personally identifying. None of which is necessary in order to respond to the questions being raised. In order to test the retention claims, we only need to see gross outcomes, which are easily anonymized. In fact, I would be surprised if Purdue does not already have an anonymized version of the data. But even if they don’t, the amount of effort required to scrub it should be relatively modest and easily proportional to the level of concern being raised here.

    There is no shame in getting research analysis wrong. It happens all the time, even to the very best researchers. This is why academia places such a high value on peer review, whether it takes place before or after publication. We are smarter as a group than we are individually. However, there is shame in using research on student success to promote the brand of an institution, and to make money, and then decline to open that research up to appropriate scrutiny by the academic community. I am not accusing either Purdue or Dr. Pistilli of doing so at this point. The critical analysis has only recently come to public attention, and I imagine that it takes a little time for any academic institution to formulate and approve an appropriate response. But the clock is ticking.

    If Dr. Pistilli or any other appropriate representative of Purdue who can speak to the substance of the research would like to respond, we certainly would give them air time on e-Literate. It doesn’t have to be here; I am sure there are other appropriate forums. But we are happy to give them an opportunity to respond here if that would help cultivate a dialog.