Inside Higher Ed’s Carl Straumsheim has some reporting on the Course Signals data controversy. He was able to get Purdue research scientist Matt Pistilli on record about it. Here is the sum total of the quotes from the article:
Pistilli defended the claims about Signals’ ability to increase retention — with the caveat that more research needs to be done. “The analysis that we did was just a straightforward analysis of retention rates,” he said. “There’s nothing else to it.”
To ensure an empirically grounded analysis of Signals, Essa urged Purdue to give researchers access to as much data as possible. Pistilli said he is open to participate in that conversation, but pointed out that granting open access could violate students’ privacy rights.
With Signals marking its fifth anniversary this year, Pistilli said “it was probably just a matter of time for people to start looking for these pieces and begin to draw conclusions.” In that sense, the discussion about early warning systems resembles that of other ed-tech innovations, like flipping the classroom and massive open online courses, where hype drowns out any serious criticism.
Now, I understand that interviews necessarily get edited down for news articles, so I am not going to jump to any deep conclusions about Dr. Pistilli’s views or Purdue’s official positions from these three short paragraphs. But I will make two points that do not require a close reading.
1. Purdue’s credibility is on the line.
Purdue has made significant noise about their retention claims. They have published articles and made presentations. They have commercially licensed their system for use by other schools, based in part on the strength of those claims. We now have credible analysis calling some of those claims into question. For its own sake, as well as for that of the academic community, Purdue now needs to go on record with a response to the critique, either acknowledging its legitimacy and amending their claims or demonstrating why the analysis is off base. There is nothing in the quote above to indicate that either Dr. Pistilli or the university understand that they have a substantial problem which demands a substantive answer.
2. Purdue would not have to violate student privacy in order to answer the concerns being raised.
Course Signals collects substantial fine-grained data about student activity, some of which could be personally identifying. None of which is necessary in order to respond to the questions being raised. In order to test the retention claims, we only need to see gross outcomes, which are easily anonymized. In fact, I would be surprised if Purdue does not already have an anonymized version of the data. But even if they don’t, the amount of effort required to scrub it should be relatively modest and easily proportional to the level of concern being raised here.
There is no shame in getting research analysis wrong. It happens all the time, even to the very best researchers. This is why academia places such a high value on peer review, whether it takes place before or after publication. We are smarter as a group than we are individually. However, there is shame in using research on student success to promote the brand of an institution, and to make money, and then decline to open that research up to appropriate scrutiny by the academic community. I am not accusing either Purdue or Dr. Pistilli of doing so at this point. The critical analysis has only recently come to public attention, and I imagine that it takes a little time for any academic institution to formulate and approve an appropriate response. But the clock is ticking.
If Dr. Pistilli or any other appropriate representative of Purdue who can speak to the substance of the research would like to respond, we certainly would give them air time on e-Literate. It doesn’t have to be here; I am sure there are other appropriate forums. But we are happy to give them an opportunity to respond here if that would help cultivate a dialog.
