e-Literate

Present is Prologue

Author: Michael Feldstein

  • Comments from a Researcher on the Course Signals Kerfluffle

    Doug Clow of the Open University has published a thoughtful and detailed blog post in response to the Course Signals effectiveness controversy. He covers far too much ground for me to attempt to summarize here, but I think there are some common themes emerging from the commentary so far:

    • The concerns over the one study have not changed the fact that Course Signals and the researchers who have been studying it are generally held in high regard. They have some very strong intra-course results which have not been challenged by current re-analysis. Even on the research that is now being challenged, they made a good faith effort to follow best research practices and are exemplars in some respects. On a personal note, I know Kim Arnold (although I did not realize she was an author on the particular study in question until Doug mentioned it in his post) and, like Doug, I think very highly of her and have learned a lot from her about learning analytics over the years.
    • That said, both the researchers and, especially, Purdue as an institution have an obligation to respond as promptly as is feasible to the challenge, in part because Purdue has chosen to license the technology in question and stands to make money based on these research claims (regardless of whether the researchers’ work was independent of the business deal). To be clear, nobody is accusing anyone of deliberately cooking the books. The point is simply that Purdue has an added ethical obligation as a consequence of the business deal.
    • The larger problem is not so much with the Purdue work itself as it is with the fact that both pre- and post-publication peer review failed. This can happen, even in papers that get a lot more eyeballs than this one did; Mike Caulfield has aptly pointed to the widely influential Reinhart and Rogoff economics paper on the effects of debt on national economies which has been belatedly proven to be in error as an apt analogy. Nevertheless, any such failure should trigger some introspection within a field regarding whether we should be doing more to cultivate robust community-based exploration of these studies, of which peer review is a part.

    I highly recommend reading Doug’s post in full.

  • Purdue’s Non-Answer on Course Signals

    Inside Higher Ed’s Carl Straumsheim has some reporting on the Course Signals data controversy. He was able to get Purdue research scientist Matt Pistilli on record about it. Here is the sum total of the quotes from the article:

    Pistilli defended the claims about Signals’ ability to increase retention — with the caveat that more research needs to be done. “The analysis that we did was just a straightforward analysis of retention rates,” he said. “There’s nothing else to it.”

    To ensure an empirically grounded analysis of Signals, Essa urged Purdue to give researchers access to as much data as possible. Pistilli said he is open to participate in that conversation, but pointed out that granting open access could violate students’ privacy rights.

    With Signals marking its fifth anniversary this year, Pistilli said “it was probably just a matter of time for people to start looking for these pieces and begin to draw conclusions.” In that sense, the discussion about early warning systems resembles that of other ed-tech innovations, like flipping the classroom and massive open online courses, where hype drowns out any serious criticism.

    Now, I understand that interviews necessarily get edited down for news articles, so I am not going to jump to any deep conclusions about Dr. Pistilli’s views or Purdue’s official positions from these three short paragraphs. But I will make two points that do not require a close reading.

    1. Purdue’s credibility is on the line.

    Purdue has made significant noise about their retention claims. They have published articles and made presentations. They have commercially licensed their system for use by other schools, based in part on the strength of those claims. We now have credible analysis calling some of those claims into question. For its own sake, as well as for that of the academic community, Purdue now needs to go on record with a response to the critique, either acknowledging its legitimacy and amending their claims or demonstrating why the analysis is off base. There is nothing in the quote above to indicate that either Dr. Pistilli or the university understand that they have a substantial problem which demands a substantive answer.

    2. Purdue would not have to violate student privacy in order to answer the concerns being raised.

    Course Signals collects substantial fine-grained data about student activity, some of which could be personally identifying. None of which is necessary in order to respond to the questions being raised. In order to test the retention claims, we only need to see gross outcomes, which are easily anonymized. In fact, I would be surprised if Purdue does not already have an anonymized version of the data. But even if they don’t, the amount of effort required to scrub it should be relatively modest and easily proportional to the level of concern being raised here.

    There is no shame in getting research analysis wrong. It happens all the time, even to the very best researchers. This is why academia places such a high value on peer review, whether it takes place before or after publication. We are smarter as a group than we are individually. However, there is shame in using research on student success to promote the brand of an institution, and to make money, and then decline to open that research up to appropriate scrutiny by the academic community. I am not accusing either Purdue or Dr. Pistilli of doing so at this point. The critical analysis has only recently come to public attention, and I imagine that it takes a little time for any academic institution to formulate and approve an appropriate response. But the clock is ticking.

    If Dr. Pistilli or any other appropriate representative of Purdue who can speak to the substance of the research would like to respond, we certainly would give them air time on e-Literate. It doesn’t have to be here; I am sure there are other appropriate forums. But we are happy to give them an opportunity to respond here if that would help cultivate a dialog.

  • Course Signals Effectiveness Data Appears to be Meaningless (and Why You Should Care)

    My father likes to say, “If you stick your head in the freezer and your feet in the oven, on average you’ll be comfortable.” Behind this pithy saying is an insight that is a little different from the “three kinds of lies” saying about statistics. It suggests that certain types of analysis produce a false coherence to the world. We see honestly patterns where there aren’t any. This is what appears to have happened with studies that Purdue University has done regarding Course Signals’ effectiveness.

    The problem was speculatively raised and then re-explained by Mike Caulfield (who we are proud to now have as an e-Literate featured blogger):

    From this desk here, without a stitch of research, I can show that people who have had more car accidents live, on average, longer than people that have had very few car accidents.

    Why? Because each year you live you have a chance of racking up another car accident. In general, the older you live, the more car accidents you are likely to have had.

    If you want to know whether people who have more accidents are more likely to live longer becauseof the car accidents, you have to do something like take 40 year-olds and compare the number of 40 year-olds that make it to 41 in your high and low accident groups (simple check), or use any one of a number of more sophisticated methods to filter out the age-car accident relation.

    The Purdue example is somewhat more contained, because the event of taking a Course Signals class or set of classes happens once per semester.  But what I am asking is whether

    1. the number of classes a student took is controlled, for, and more importantly,
    2. whether first to second year retention is calculated as
      1.  the number of students that started year two / the number of students who started year one (our car accident problem), or
      2.  the number of students who started year two / the number of students who finished year one (our better measure in this case).

    Pointing to the first of these posts, I suggested that a response from the Purdue researchers would be helpful. (It still would be.) Now Al Essa, recently of Desire2Learn and currently of McGraw Hill, has done a more mathematically rigorous analysis showing that Mike’s intuition appears to be correct. He took the Purdue findings, substituted the phrase “were given a chocolate” for “took a class using Course Signals,” and ran a simulation to see whether he could reproduce the Purdue results with no causal connection between the chocolatey intervention and the retention results:

    The following are some results from the simulation. The first row displays retention rates for students who received no chocolates. The second row displays retention rates for students who received at least one chocolate. The last row shows students who received two or more chocolates. Why track students who received two or more chocolates? Because the authors of the study claim that two is the “magic number” where significant retention gains kick in.

    The simulation data shows us that the retention gain for students is not a real gain (i.e. causal) but an artifact of the simple fact that students who stay longer in college are more likely to receive more chocolates. So, the answer to the question we started off with is “No.”. You can’t improve retention rates by giving students chocolates.

    This is a problem that goes well beyond Course Signals itself for several reasons. First, both Desire2Learn and Blackboard have modeled their own retention early warning systems after Purdue’s work. For that matter, I have praised Course Signals up and down and criticized these companies for not modeling their products more closely on that work, largely based on the results of the effectiveness studies. So we don’t know what we thought we knew about effective early warning systems. The fact that the research results appear to be spurious does not mean that systems like Course Signals has no value, but it does mean that we don’t have the proof that we thought we had of their value.

    More generally, we need to work much harder as a community to critically evaluate effectiveness study results. Big decisions are being made based on this research. Products are being designed and bought. Grants are being awarded. Laws are starting to be written. I believe strongly in effectiveness research, but I also believe strongly that effectiveness research is hard. The Purdue results have been around for quite a while now. It is disturbing that they are only now getting critical examination.

    Update: Seeing some of the posts in the comments thread, I feel the need to make a clarification in fairness to Purdue. They have two important sets of research findings, only one of which is being called into question here. Their early findings, that Course Signals can increase student grades and chances of completion within a class, are not being challenged here. Those are important results. The findings that are being questioned are their longitudinal analysis showing that students who Course Signals in one class are more likely to do well in future classes. Even there, it is possible that Course Signals does have some long-term effect. The point is simply that the result the researchers got in their analysis looks suspiciously similar to the results one would get from a fairly straightforward selection bias.

  • Apereo Foundation Reception at EDUCAUSE Tonight

    This is just a quick note on behalf of my friends at the Apereo Foundation to note that their reception will be at 6:30 PM at the Hilton tonight at EDUCAUSE. For some reason, it got left off the program. Details are here. I, unfortunately, will not be able to join, but if you’re interested in talking to good people ding good work in higher ed open source, then drop on by.

  • Announcing e-Literate TV

    A006F5AA-D0A1-4BF3-9123-C0DAC8AE29ED@roc.mn.charter.com

    This week is the week of the big annual EDUCAUSE conference which, among other things, is the world’s largest ed tech fashion show. In the next five days, we will learn that earth tones are the latest style for adaptive personalized learning systems, and that hemlines and license fees are both going way up this year. And by the end of the week, more than one conference-goer will be heard to mutter, “If I have to sit through one more MOOC presentation, I am going to shoot myself.”

    And yet, outside this bubble, the noise dissipates surprisingly quickly. When Phil and I visit campuses, we routinely meet faculty who have literally never heard of MOOCs. When new ed tech products do come to campus, they are often either uncritically embraced as the savior of higher education or reflexively reviled as its destruction—and often the latter by one stakeholder group in quick reaction to the former by another group. We have real and serious challenges in education, some of which can genuinely be helped through the judicious application of technology by thoughtful and concerned educators. But we are not having the right conversations to match problems with solutions.

    That is why Phil and I are delighted and excited to announce a new initiative we’re calling e-Literate TV, in collaboration with a company called In the Telling. Using some of the lessons that we’re learning from the MOOC community about differential engagement, our goal is create multiple entry points into a conversation about the issues. The first entry point is a series of 10-minute video episodes providing overviews of each new topic. One of the two great assets that In the Telling brings to the table is their experience at telling stories in film. With their help, we are crafting segments that we hope will be appealing and informative to those faculty, presidents, provosts, and other important college and university stakeholders who are not ed tech junkies. It will introduce the topics in what we hope will be an engaging and provocative manner. All episodes will be released under a Creative Commons license on YouTube.

    For those who wish to dive deeper, we will be taking advantage of In the Telling’s second asset, which is their Telling Story platform. We can tie content directly to the timeline of each video, bringing up further references, like e-Literate blog posts or relevant scholarly articles, in context. And finally, we’ll be integrating that platform into WordPress, where we will be posing questions that are intended to be discussion starters across campus stakeholder groups. We are particularly interested in community- and conversation-building features and how to add to these capabilities over time. Access to that content will also be free, and in cases where it is e-Literate blog posts, it is already Creative Commons licensed.

    The pilot series will consist of five episodes (not counting the introduction) on the following topics:

    • The big issues in higher education that are driving interest in and need for educational technology
    • The landscape of online learning
    • The state of play with MOOCs
    • The rise of courseware
    • Learning analytics and adaptive learning

    Filming of these episodes has already begun and will be finished up at EDUCAUSE. We’ll have a longer trailer showing off some of this new material soon and expect to launch the pilot series in December or January.

    Beyond that, we are seeking sponsorship to fund e-Literate TV as an ongoing project. We think we can do four series the size of the pilot every year. Truth to tell, each one of the pilot episodes could be a topic of its own series. We’re also interested in digging into topics like the role of open education, how to get beyond the factory model in the classroom, and how the ed tech ecosystem works.

    We’ll be sharing more about the project as it develops. In the meantime, here’s a taste of the video aspect of the series:

    And here’s an overview of the Telling Story platform:

  • The IMS’s New “Caliper” Learning Analytics Interoperability Framework Is Deeply Interesting

    The IMS has announced the initial public release of something they call Caliper, which they characterize as a learning analytics interoperability framework. But it’s actually much, much more than that. In fact, it represents the functional core of something that my SUNY colleagues and I used to refer to as a Learning Management Operating System (LMOS), and is something that I have been hoping to see for eight years, because it promises to resolve the tension between the flexibility of lots of  separately developed, specialized learning tools and the value and convenience of an integrated system.

    Let’s take a peek at the framework to see why I’m so hopeful about this framework. But before we do that, you should fasten your seat belts and strap on your aviator goggles. It’s going to get geeky in here.

    (more…)

  • e-Literate and Conflicts of Interest

    As you may know, Phil and I started a consulting practice in January. Throughout my years of blogging here, I have made it a practice to update readers on how changes in professional life may affect the writing that I (and now we) do on the blog. I don’t believe there is any such thing as “objective” analysis. Analysis is all about perspective, and perspective is derived from your point of view—where you stand, and so on. If we are going to be fortunate enough to continue earning your trust, then it’s important for you to understand our perspective so that you can anticipate and account for any limitations that are inherent in it. As it turns out, running a consulting business creates both some complex ethical challenges as well as some exciting opportunities for the blog. Now that Phil and I have had some time to understand the landscape better, this seems like a good time to give you an update. (more…)