e-Literate

Present is Prologue

Tag: MIke Caulfield

  • Can Pearson Solve the Rubric’s Cube?

    Love ’em or hate ’em, it’s hard to dispute that Pearson has an outsized impact on education in America. This huge company—they have a stock market valuation of $18 billion—touches all levels from kindergarten through career education, providing textbooks, homework platforms, high-stakes testing, and even helping to design entire online degree programs. So when they announce a major change in their corporate strategy, it is consequential.

    That is one reason why I think that most everybody who is motivated to read this blog on a regular basis will also find it worthwhile to read Pearson’s startling publication, “The Incomplete Guide to Delivering Learning Outcomes” and, more generally, peruse their new efficacy web site. One of our goals for e-Literate is to explain what the industry is doing, why, and what it might mean for education. Finding the answers to these questions is often an exercise in reading the tea leaves, as Phil ably demonstrated in his recent posts on the Udacity/SJSU pilot and the layoffs at Desire2Learn. But this time is different. In all my years of covering the ed tech industry, I have never seen a company be so explicit and detailed about their strategy as Pearson is being now with their efficacy publications. Yes, there is plenty of marketing speak here. But there is also quite a bit about what they are actually doing as a company internally—details about pilots and quality reviews and hiring processes and M&A criteria. These are the gears that make a company go. The changes that Pearson is making in these areas are the best clues we can possibly have as to what the company really means when they say that they want efficacy to be at the core of their business going forward. And they have published this information for all the world to see.

    These now-public details suggest a hugely ambitious change effort within the company. Phil and I have consulted for a few textbook publishers, including Pearson, and I worked for Cengage for a year and a half. We have a pretty good idea of the magnitude of the change management challenges these companies face right now and the strategies that various publishers are bringing to bear in an effort to meet them. I can say with absolute conviction that what Pearson has announced is no half-hearted attempt or PR window dressing, and I can say with equal conviction that what they are attempting will be enormously difficult to pull off. They are not screwing around. Whatever happens going forward, Pearson is likely to be a business school case study for the ages.

    As if all of this drama weren’t enough, Pearson’s strategy raises another question which should be fascinating for educators; namely, can a rubric transform a multi-billion-dollar company?

    Fair warning: This post is ridiculously long. Even by my standards.

    (more…)

  • Don’t Cry for Me, Argentina

    While it is well hidden, wrapped in a very careful press release, Phil’s sharp eye has caught the details in SJSU’s press release about the next phase in the Udacity pilot that suggest the partnership between the school and the company is winding down. When Carl Straumsheim of Inside Higher Ed asked an SJSU spokesperson point-blank whether Udacity would continue to be involved with the courses, the reply he got was “Good question for Udacity.”

    Ouch.

    The Schadenfreude surrounding Sebastian Thrun’s fall from grace has been intense ever since the Fast Company article quoted the man who the author labeled the “godfather of free online education” as saying that he realized he had a bad product, and noted that his company is “changing course” to focus on corporate training. Mike Caulfield captured the tone of the reaction in ed tech circles rather nicely when he wrote:

    Thrun can’t build a bucket that doesn’t leak, so he’s going to sell sieves….Udacity dithered for a bit on whether it would be accountable for student outcomes. Failures at San José State put an end to that. The move now is to return to the original idea: high failure rates and dropouts are features, not bugs, because they represent a way to thin pools of applicants for potential employers. Thrun is moving to an area where he is unaccountable, because accountability is hard.

    I imagine that it would be easy for somebody running or funding an ed tech startup to draw the wrong lessons from this sad story. Consider this blog post to be an open letter to my friends at ed tech startups with some advice about how to avoid the kind of disdain and ridicule that Thrun is receiving now.

    (more…)

  • Massive, Open, and Course Design

    Martin Weller has a great blog post up about course design responses to MOOC completion rates. He starts by arguing that, while completion rates are not everything in MOOCs, they are not nothing either. A lot depends on whether you think completion is an important metric to meet the course goals because, for example, the course is designed to help remedial students pass into a non-remedial track, or whether having students explore the content in a non-comprehensive way accomplishes your course goal. (Martin brings up an analogy by Stephen Downes that nobody complains about the low newspaper completion rates, which I have never heard before and which I love.)

    This is good stuff, but it starts us down the path toward a more radical re-examination of how we think about course design. Because while Martin is focusing primarily on course goals and how those should determine metrics, he’s beginning to raise the question of how individual learner goals should influence course design. And once you start asking that question, it changes everything.

    (more…)

  • Course Signals Effectiveness Data Appears to be Meaningless (and Why You Should Care)

    My father likes to say, “If you stick your head in the freezer and your feet in the oven, on average you’ll be comfortable.” Behind this pithy saying is an insight that is a little different from the “three kinds of lies” saying about statistics. It suggests that certain types of analysis produce a false coherence to the world. We see honestly patterns where there aren’t any. This is what appears to have happened with studies that Purdue University has done regarding Course Signals’ effectiveness.

    The problem was speculatively raised and then re-explained by Mike Caulfield (who we are proud to now have as an e-Literate featured blogger):

    From this desk here, without a stitch of research, I can show that people who have had more car accidents live, on average, longer than people that have had very few car accidents.

    Why? Because each year you live you have a chance of racking up another car accident. In general, the older you live, the more car accidents you are likely to have had.

    If you want to know whether people who have more accidents are more likely to live longer becauseof the car accidents, you have to do something like take 40 year-olds and compare the number of 40 year-olds that make it to 41 in your high and low accident groups (simple check), or use any one of a number of more sophisticated methods to filter out the age-car accident relation.

    The Purdue example is somewhat more contained, because the event of taking a Course Signals class or set of classes happens once per semester.  But what I am asking is whether

    1. the number of classes a student took is controlled, for, and more importantly,
    2. whether first to second year retention is calculated as
      1.  the number of students that started year two / the number of students who started year one (our car accident problem), or
      2.  the number of students who started year two / the number of students who finished year one (our better measure in this case).

    Pointing to the first of these posts, I suggested that a response from the Purdue researchers would be helpful. (It still would be.) Now Al Essa, recently of Desire2Learn and currently of McGraw Hill, has done a more mathematically rigorous analysis showing that Mike’s intuition appears to be correct. He took the Purdue findings, substituted the phrase “were given a chocolate” for “took a class using Course Signals,” and ran a simulation to see whether he could reproduce the Purdue results with no causal connection between the chocolatey intervention and the retention results:

    The following are some results from the simulation. The first row displays retention rates for students who received no chocolates. The second row displays retention rates for students who received at least one chocolate. The last row shows students who received two or more chocolates. Why track students who received two or more chocolates? Because the authors of the study claim that two is the “magic number” where significant retention gains kick in.

    The simulation data shows us that the retention gain for students is not a real gain (i.e. causal) but an artifact of the simple fact that students who stay longer in college are more likely to receive more chocolates. So, the answer to the question we started off with is “No.”. You can’t improve retention rates by giving students chocolates.

    This is a problem that goes well beyond Course Signals itself for several reasons. First, both Desire2Learn and Blackboard have modeled their own retention early warning systems after Purdue’s work. For that matter, I have praised Course Signals up and down and criticized these companies for not modeling their products more closely on that work, largely based on the results of the effectiveness studies. So we don’t know what we thought we knew about effective early warning systems. The fact that the research results appear to be spurious does not mean that systems like Course Signals has no value, but it does mean that we don’t have the proof that we thought we had of their value.

    More generally, we need to work much harder as a community to critically evaluate effectiveness study results. Big decisions are being made based on this research. Products are being designed and bought. Grants are being awarded. Laws are starting to be written. I believe strongly in effectiveness research, but I also believe strongly that effectiveness research is hard. The Purdue results have been around for quite a while now. It is disturbing that they are only now getting critical examination.

    Update: Seeing some of the posts in the comments thread, I feel the need to make a clarification in fairness to Purdue. They have two important sets of research findings, only one of which is being called into question here. Their early findings, that Course Signals can increase student grades and chances of completion within a class, are not being challenged here. Those are important results. The findings that are being questioned are their longitudinal analysis showing that students who Course Signals in one class are more likely to do well in future classes. Even there, it is possible that Course Signals does have some long-term effect. The point is simply that the result the researchers got in their analysis looks suspiciously similar to the results one would get from a fairly straightforward selection bias.

  • Please Welcome Featured Blogger Mike Caulfield

    Michael and I have been very impressed with the articles from Mike Caulfield, an edublogger who writes at Hapgood. Mike is director of blended and networked learning at Washington State University Vancouver, and he and Michael first met at a Lumen Learning event this summer. In addition to writing at his blog, Mike Caulfield has also written an excellent  EDUCAUSE Review article along with Amy Collier and Sherif Halawa, titled “Rethinking Online Community in MOOCs Used for Blended Learning”.

    I’m happy to say that Mike will be doing a series of posts for us, starting today. Please welcome him.

  • Digging into the Purdue Course Signals Results

    Update: Mike has written another post clarifying the intuitions behind his math.

    The spectacular Mike Caulfield casts a skeptical eye on the Course Signals data:

    Only a portion of Purdue’s classes are Course Signals classes, so the chance any course a freshman takes is a Course Signals course can be expressed as a percentage, say 25%. In an overly dramatic simplification of this model, a freshman who takes four classes the first semester and drops out has a has about a 16% chance of having taken two Course Signals courses (as always, beware my math here, but I think I’m right). Meanwhile they have a 74% chance of having taken 1 or fewer, and a 42% chance of having taken exactly one.

    What about about a student who does *not* drop out first semester, and takes a full load of five courses each semester? Well, the chance of that student having two or more Course Signals courses is 75%. That’s right — just by taking a full load of classes and not dropping out first semester you’re likely to be tagged as a CS 2+ student.

    In other words, each class you take is like an additional coin flip. A lot of what Course Signals “analysis” is measuring is how many classes students are taking.

    Are there predictions this model makes that we can test? Absolutely. As we saw in the above example, at a 25% CS adoption rate, the median dropout has a 42% chance of having taken exactly one CS course. So it’s quite normal for a dropout to have had a CS course. But early on in the program the adoption rate would have much lower. What are the odds of a first semester dropout having a CS course in those early pilots? For the sake of argument let’s say adoption at that point was 5%. In that case, the chance our 4-course semester drop out would have exactly one CS course drops from 42% to 17%. In other words, as adoption grows having had one course in CS will cease to be a useful predictor of first to second-year persistence.

    Is that what we see? Assuming adoption grew between 2007 and 2009, that’s *exactly* what we see.

    I’d like to see somebody at Purdue (or Ellucian) respond to the questions that Mike raises. Matt Pistilli, are you listening?