e-Literate

Present is Prologue

Tag: Al Essa

  • NPR and Missed (Course) Signals

    Anya Kamenetz has a piece up on NPR about learning analytics, highlighting Purdue’s Course Signals as its centerpiece. She does a good job of introducing the topic to a general audience and raising some relevant ethical questions. But she missed one of the biggest ethical questions surrounding Purdue’s product—namely, that some of its research claims are likely false. In particular, she repeats the following claim:

    Course Signals…has been shown to increase the number of students earning A’s and B’s and lower the number of D’s and F’s, and it significantly raises the chances that students will stick with college for an additional year, from 83% to 97%. [Emphasis added.]

    Based on the work of Mike Caulfield and Al Essa summarized in the link above, it looks like that latter claim is probably the result of selection bias rather than a real finding. So who is at fault for this questionable claim being repeated without challenge in a popular venue many months after it has been convincingly challenged?

    For starters, Purdue is. They never responded to the criticism, despite confirmation that they are aware of it—for one thing, they got contacted by us and by Inside Higher Ed—and despite the fact that they apparently continue to make money off the sales of the product through a licensing deal with Ellucian. And the uncorrected paper is still available on their web site. This is unconscionable.

    Anya clearly bears some responsibility too. Although it’s easy to assume from the way the article is written that the dubious claim was repeated to her in an interview by Purdue research Matt Pistilli, she confirmed for me via email that she took the claim from the previously published research paper and did not discuss it with Pistilli. Given that this is her central example of the potential of learning analytics, she should have interrogated this a little more, particularly since she had Matt on the phone. Mike Caulfield also commented to me that any claim of such a dramatic increase in year-to-year retention should automatically be subject to additional scrutiny.

    I have to put some blame on the higher ed press as well. Inside Higher Ed covered the story (and, through them, the Times Higher Education). In fact, Carl Straumsheim actually advanced the story a bit by putting the question to researcher Matt Pistilli (who gave a non-answer). The Chronicle of Higher Education did not cover it, despite having run a puff piece on Purdue’s claims the same day that Mike Caulfield wrote his original piece challenging the results. It is very clear to Phil and me that we are read by the Chronicle staff, in part because they periodically publish stories that have been obviously influenced by our earlier coverage. Sometimes without attribution. I don’t care that much about the credit, but if they thought Purdue’s claims were newsworthy enough to cover in the first place then they should have done their own reporting on the fact that those claims have been called into question. If they had been more aggressive in their coverage then the mainstream press reporters who find Course Signals will be more likely to find the other side(s) of the story as well. Outside of IHE, I’m having trouble finding any coverage, never mind any original reporting, in the higher ed or ed tech press.

    I have a lot of respect for news reporters in general, and I think that most people grossly underestimate how hard the job is. I think highly of Anya as a professional. I like the reporters I interact with most at the Chronicle as well. Nor will I pretend that we are perfect here at e-Literate. We miss important angles and get details wrong our fair share. For example, I doubt that I would have caught the flaw in Purdue’s research if Mike hadn’t brought it to my attention. But collectively, we have to do a better job of providing critical coverage of topics like learning analytics, particularly at a time when so much money is being spent and our entire educational system is starting to be remade on the premise that this stuff will work. And there is absolutely no excuse whatsoever for a research university to not take responsibility for their published research on a topic that is so critical to the future of universities.

  • New e-Literate TV Episode: Adaptive Learning and Learning Analytics

    In our latest episode, the penultimate in the pilot series, we explore the topics that are likely to be moving up the curve of the hype cycle this year—adaptive learning and learning analytics.  Like many of the topics in the pilot series, we could have made an entire series about this one. (And maybe we will at some point.) But since the first adaptive learning product faculty will run into is most likely to be from a textbook publisher, we interviewed McGraw Hill’s Al Essa and Pearson’s Jason Jordan about their respective takes on what this trend is all about.

    Here it is.

  • Some big personnel changes in LMS market

    In just the past week we have had three fairly significant people depart higher ed LMS companies. This really is turning out to be a bumpy ride as the market changes.

    • Ray Henderson announced last night that he is leaving his operational role at Blackboard (President, Academic Platforms and CTO) and is moving into a role with the Board of Directors. More info from Jay Bhatt’s post here and Ray Henderson’s post here. Michael is working on an e-Literate post with more information soon. Bill Flook covered in an article here.
    • Devlin Daley, one of the two founders of Instructure, the company behind the Canvas LMS, is leaving the company as of today. I talked to Instructure rep today who indicated that Devlin is looking to get back in to startup ed tech mode, whereas Instructure is becoming a larger company. I’ll write more of an analysis on this move soon. For now I’ll just say that it is extremely rare for a tech founder to leave a company that might go public within a year or two.
    • Al Essa, the Director of Analytics Research and Strategy for Desire2Learn, has left the company to join McGraw-Hill based on his LinkedIn profile. This is curious timing, given Desire2Learn’s major focus on analytics and the Student Success System this year.

    More to come.

  • Desire 2 Improve

    Phil and I spent some time at Desire2Learn’s FUSION conference the other week, and it was an interesting experience. D2L had a couple of first-ever “analyst days” in which we and a few others got some intensive briefings on their products and strategies. (D2L isn’t the only company that is doing this sort of thing, but they’ve gone significantly further with it than the other LMS companies have so far.) It was a useful complement to the information that we get from walking the floor and talking to customers. And, as I’m sure they intended, we walked away from the experience feeling somewhat more positive about the company than we did going in—not so much because of the dog-and-pony show but because we were able to ask some hard questions to senior managers and, more often than not, get reasonable answers.

    Historically, my take on D2L has been as follows: On the good side, they have coherent product vision and their own take on the LMS space—I have a lot of respect for Ken Chapman as a product guy—and good relationships with their customers. These factors contribute significantly to their very high customer retention rate, even relative to the other LMS providers which, as a group, generally have fairly high retention rates. On the bad side, D2L has not always been particularly good at executing technically difficult projects, and they have not always had a good sense of how well they are performing in that regard—what they have achieved, how long it will take them to deliver functionality, how serious the problems are, and so on. CEO John Baker, whose deeply ingrained sense of optimism enabled him to face down a patent lawsuit against serious odds, can be the worst offender in this regard. He is chronically overly optimistic about the state of the company’s products and the seriousness of any bumps in the road.

    In the last year or two, D2L has begun hiring some outsiders to bolster their technical team. They have benefitted from the hot tech market in Kitchener in general and to the collapse of RIM in particular, snapping up relatively senior Blackberry technical people. But it’s fair to say that they have been actively recruiting tech talent, both inside and outside of Kitchener, for a while now. For example, they hired Al Essa, whom I have known for over a decade since his time as CEO of MIT’s Sloan School of Business and his leadership in the dotLRN open source LMS community, to head up their analytics product line.

    It’s hard to tell for certain whether this hiring trend has accelerated since D2L received their investment, but it certainly appears like that might be the case. I’ll call out the example of Nick Oddson, the new VP of Enterprise Product Engineering, who came from OpenText (a serious enterprise content management company that you probably have never heard of). Nick impressed both Phil and me as a guy who knows his stuff and is a straight shooter about what the company is doing well and what it is not doing well yet. At some point we expect to write more about what Nick in particular is up to, because we think it’s pretty pivotal for the future of the company. For now, the point is that D2L seems to be making hires that have the potential to shore up their historic weaknesses. In the next 12-24 months, we will see whether these new hires are empowered to make a real difference in D2L’s performance.

    I’ll have a follow-up post specifically on their analytics products fairly soon.