e-Literate

Present is Prologue

Tag: learning analytics

  • NPR and Missed (Course) Signals

    Anya Kamenetz has a piece up on NPR about learning analytics, highlighting Purdue’s Course Signals as its centerpiece. She does a good job of introducing the topic to a general audience and raising some relevant ethical questions. But she missed one of the biggest ethical questions surrounding Purdue’s product—namely, that some of its research claims are likely false. In particular, she repeats the following claim:

    Course Signals…has been shown to increase the number of students earning A’s and B’s and lower the number of D’s and F’s, and it significantly raises the chances that students will stick with college for an additional year, from 83% to 97%. [Emphasis added.]

    Based on the work of Mike Caulfield and Al Essa summarized in the link above, it looks like that latter claim is probably the result of selection bias rather than a real finding. So who is at fault for this questionable claim being repeated without challenge in a popular venue many months after it has been convincingly challenged?

    For starters, Purdue is. They never responded to the criticism, despite confirmation that they are aware of it—for one thing, they got contacted by us and by Inside Higher Ed—and despite the fact that they apparently continue to make money off the sales of the product through a licensing deal with Ellucian. And the uncorrected paper is still available on their web site. This is unconscionable.

    Anya clearly bears some responsibility too. Although it’s easy to assume from the way the article is written that the dubious claim was repeated to her in an interview by Purdue research Matt Pistilli, she confirmed for me via email that she took the claim from the previously published research paper and did not discuss it with Pistilli. Given that this is her central example of the potential of learning analytics, she should have interrogated this a little more, particularly since she had Matt on the phone. Mike Caulfield also commented to me that any claim of such a dramatic increase in year-to-year retention should automatically be subject to additional scrutiny.

    I have to put some blame on the higher ed press as well. Inside Higher Ed covered the story (and, through them, the Times Higher Education). In fact, Carl Straumsheim actually advanced the story a bit by putting the question to researcher Matt Pistilli (who gave a non-answer). The Chronicle of Higher Education did not cover it, despite having run a puff piece on Purdue’s claims the same day that Mike Caulfield wrote his original piece challenging the results. It is very clear to Phil and me that we are read by the Chronicle staff, in part because they periodically publish stories that have been obviously influenced by our earlier coverage. Sometimes without attribution. I don’t care that much about the credit, but if they thought Purdue’s claims were newsworthy enough to cover in the first place then they should have done their own reporting on the fact that those claims have been called into question. If they had been more aggressive in their coverage then the mainstream press reporters who find Course Signals will be more likely to find the other side(s) of the story as well. Outside of IHE, I’m having trouble finding any coverage, never mind any original reporting, in the higher ed or ed tech press.

    I have a lot of respect for news reporters in general, and I think that most people grossly underestimate how hard the job is. I think highly of Anya as a professional. I like the reporters I interact with most at the Chronicle as well. Nor will I pretend that we are perfect here at e-Literate. We miss important angles and get details wrong our fair share. For example, I doubt that I would have caught the flaw in Purdue’s research if Mike hadn’t brought it to my attention. But collectively, we have to do a better job of providing critical coverage of topics like learning analytics, particularly at a time when so much money is being spent and our entire educational system is starting to be remade on the premise that this stuff will work. And there is absolutely no excuse whatsoever for a research university to not take responsibility for their published research on a topic that is so critical to the future of universities.

  • Blackboard’s Perceptis Acquisition Offers Clues into Company’s Strategy

    Yesterday Blackboard announced that they acquired Perceptis, a provider of help desk and financial aid support services for colleges and universities. In and of itself, this is not a huge acquisition. Perceptis has 33 clients, offers services that Blackboard was already offering, and has no substantial new technology. But as we approach BbWorld next week, the move provides some early hints into the strategic direction that the company may highlight at the conference.

    I had the opportunity to talk with Blackboard’s Vice President of Education Services Katie Blot about the move.

    (more…)

  • No, I don’t believe that Harvard or MIT are hiding edX data

    Since my Sunday post What Harvard and MIT could learn from the University of Phoenix about analytics, there have been a few comments with a common theme about Harvard and MIT perhaps withholding any learner-centered analytics data. As a recap, my argument was:

    Beyond data aggregated over the entire course, the Harvard and MIT edX data provides no insight into learner patterns of behavior over time. Did the discussion forum posts increase or decrease over time, did video access change over time, etc? We don’t know. There is some insight we could obtain by looking at the last transaction event and number of chapters accessed, but the insight would be limited. But learner patterns of behavior can provide real insights, and it is here where the University of Phoenix (UoP) could teach Harvard and MIT some lessons on analytics.

    Some of the comments that are worth addressing:

    “Non-aggregated microdata (or a “person-click” dataset, see http://blogs.edweek.org/edweek/edtechresearcher/2013/06/the_person-click_dataset.html ) are much harder (impossible?) to de-identify. So you are being unfair in comparing this public release of data with internal data analytic efforts.”

    “Agreed. The part I don’t understand is how they still don’t realize how useless this all is. Unless they are collecting better data, but just not sharing it openly, hogging it to themselves until it ‘looks good enough for marketing’ or something.”

    “The edX initiative likely has event-level data to analyze. I don’t blame them for not wanting to share that with the world for free though. That would be a very valuable dataset.”

    The common theme seems to be that there must be learner-centered data over time, but Harvard and MIT chose not to release this data either due to privacy or selfish reasons. This is a valid question to raise, but I see no evidence to back up these suppositions.

    (more…)

  • What Harvard and MIT could learn from the University of Phoenix about analytics

    Last week Harvard and MIT released de-identified data from their edX-based MOOCs. Rather than just produce a summary report, the intent of this release was to open up the data and share it publicly. While it is good to see this approach to Open Data, unfortunately the data set is of limited value, and it actually illustrates a key problem with analytics in higher ed. From MIT News description:

    A research team from Harvard University and MIT has released its third and final promised deliverable — the de-identified learning data — relating to an initial study of online learning based on each institution’s first-year courses on the edX platform.

    Specifically, the dataset contains the original learning data from the 16 HarvardX and MITx courses offered in 2012-13 that formed the basis of the first HarvardX and MITx working papers (released in January) and underpin a suite of powerful open-source interactive visualization tools (released in February).

    At first I was eager to explore the data, but I am not sure how much useful insight is possible due to how the data was collected. The data is structured with one student per row for each course they took (taking multiple courses would lead to multiple rows of data). The data columns (pulled from the Person Course Documentation file) are shown below:

    (more…)

  • New e-Literate TV Episode: Adaptive Learning and Learning Analytics

    In our latest episode, the penultimate in the pilot series, we explore the topics that are likely to be moving up the curve of the hype cycle this year—adaptive learning and learning analytics.  Like many of the topics in the pilot series, we could have made an entire series about this one. (And maybe we will at some point.) But since the first adaptive learning product faculty will run into is most likely to be from a textbook publisher, we interviewed McGraw Hill’s Al Essa and Pearson’s Jason Jordan about their respective takes on what this trend is all about.

    Here it is.

  • The IMS’s New “Caliper” Learning Analytics Interoperability Framework Is Deeply Interesting

    The IMS has announced the initial public release of something they call Caliper, which they characterize as a learning analytics interoperability framework. But it’s actually much, much more than that. In fact, it represents the functional core of something that my SUNY colleagues and I used to refer to as a Learning Management Operating System (LMOS), and is something that I have been hoping to see for eight years, because it promises to resolve the tension between the flexibility of lots of  separately developed, specialized learning tools and the value and convenience of an integrated system.

    Let’s take a peek at the framework to see why I’m so hopeful about this framework. But before we do that, you should fasten your seat belts and strap on your aviator goggles. It’s going to get geeky in here.

    (more…)

  • Blackboard Analytics Update

    In my last post, I promised that I would give an update specifically on the state of Blackboard’s learning analytics. Well, here you go. This is a summary of what I learned about their product from a chat with Mark Max, Blackboard’s VP of Learning Analytics and, to a lesser degree, with VP of User Experience Stephanie Weeks. I wrote about Blackboard’s Retention Center product some time ago. That product (or feature set, since it is free in Blackboard) directly competes with Desire2Learn’s Student Success System. This post is more broadly about their Analytics product suite, which is most directly analogous with Desire2Learn’s Insights product, although it is actually much, much broader in scope.

    The short version is this: Blackboard has very solid and reliable technology base from which they are building their learning analytics. It is easily the most mature platform among the LMS providers from that perspective. What they are a little short on is vision. In other words, they are pretty much the mirror image of Desire2Learn.

    (more…)