e-Literate

Present is Prologue

Category: Ed Tech

The “Ed Tech” category includes posts about educational technology products themselves, including LMSs and other learning platforms, adaptive learning and other digital curricular materials products, learning analytics, and educational apps of all types. It also includes technical aspects of ed tech products, especially interoperability.

  • Purdue Course Signals Data Issue Explainer

    The “Course Signals” story originally covered here has recently gone international, with Britain’s prestigious Times Higher Education magazine picking up the Inside Higher Ed story and publishing it as an “Editor’s Pick”. Hopefully this will push  the Course Signals team to answer questions asked of them nearly two months ago, questions that have still not been satisfactorily answered.

    We realize those watching the posts on e-Literate over the past couple weeks may have some questions about what the “Course Signals issue” is, what it isn’t, and why it is so important for the educational technology community to make sure Purdue is accounting for the recent issue discovered with their statistical approach. This explainer should get you up to speed.

    What is Course Signals? Why is it important?

    Course Signals is a software product developed at Purdue University to increase student success through the use of analytics to alert faculty, students, and staff to potential problems. Through using a formula that takes into account a variety of predictors and current behaviors (e.g. previous GPA, attendance, running scores), Course Signals can help spot potential academic problems before traditional methods might. That formula labels student status in given course according to a green-yellow-red scheme that clearly indicates whether students are in danger of the dreaded DWIF (dropping out, withdrawing, getting an incomplete, or failing)

    While the product is used to to improve in-class student performance, the product is most often discussed in in a larger frame, as a product that increases long-term student success. The product has won prestigious awards for its approach to retention, and the product is particularly important in the analytics field, as its reported ability to increase retention by 21% makes it one of the most effective interventions out there, and suggests that technological solutions to student success can significantly outperform more traditional measures.

    What problems were found in the data supporting the retention effects?

    Purdue had been claiming that taking classes using CS technology led to better retention. Several anomalies in the data led to the discovery that the experiment may suffer from a “reverse-causality” problem.

    One such anomaly was an odd “dose-response” curve. With many effective interventions, as exposure to the intervention increases, the desired benefit increases as well. In the recent Purdue data, taking one Course Signal-enhanced course was shown to have a very slight negative benefit, while taking two had a very strong benefit.

    The story became even more complex when older data was examined. Early in the program taking one CS-enhanced course had a very substantial impact on retention, nearly equal to taking two CS-enhanced classes. But as the program expanded over the years, taking one CS-enhanced class started to show no impact at all. This behavior is not consistent with Course Signals causing higher retention.

    I hypothesized a simple model to explain this shift: rather than students taking more CS-courses retaining at a higher rate, what was really happening was that the students who dropped out mid-year were taking less CS classes because they were taking less classes period. In other words, the retention/CS link existed, but not in a meaningful way. Unlike the Purdue model where taking CS-enhanced courses caused retention, this “reverse-causality” model explained why as participation expanded taking one CS-enhanced course might move from being a strong predictor to having no predictive force at all.

    Michael Feldstein picked up on this analysis, and prodded the Purdue team for a response. When no response came, Alfred Essa, head of R & D and Analytics at McGraw-Hill, took my “back-of-the-envelope” model, and built it out into a full-fledged simulation. The simulation confirmed the reverse-causality model explained the data anomalies very well, much better than Purdue’s causal model. Purdue’s response to the simulation did not address the serious issues raised.

    Does this mean Course Signals does not work?

    It depends. Purdue has yet to respond to the new information in any meaningful way, and until they either release revised estimates that control for this effect or release their data for third-party analysis, we don’t know the full story. Additionally, there are some course level effects seen in early Signals testing that will be unaffected by the issue.

    However, Purdue’s recent response to Inside Higher Ed indicates that they did not control for the reverse-causality issue at all. If this is true, then the likelihood is that the retention impact of Course Signals will be positive, but significantly below the 21% they have been claiming.

    But positive impact is good, right?

    Not really. The great insight regarding educational interventions of the past decade or so is what we might term “Hattie’s Law”, after researcher John Hattie. Most educational interventions have some effect. Doing something is usually better than doing nothing. The question that administrators face is not which interventions “work”, but which interventions “work better than average.”

    At a 21% impact on retention, Course Signals was clearly in the “better than average” category, and its unparalleled dominance in that area suggested that the formula and approach embraced by Course Signals formed the best possible path forward.

    Halve that impact and everything changes. Peer coaching models such as InsideTrack have shown impact in the 10-15% range. Increased student aid has shown moderate impact, as has streamlined registration and course access initiatives.

    Additionally, other analytics packages exist that have taken a different route than Course Signals. Up until now, they have lived in the shadow of Purdue’s success. If CS impact is shown to be significantly reduced, it may be time to give those approaches a second look.

    What is unaffected by the new analysis?

    Until Purdue fixes and reruns their analysis, it it hard to know what the effects might be. However, there were a number of claims Purdue made that were not based on longitudinal analysis, and these should stand. For instance, students in Course Signals do tend to get more A’s and less F’s, and that data would be unaffexted by this issue.

    While that’s good, it’s not the major intent of at least some institutions interested in the system. What makes systems like this particularly attractive is their ability to pay for themselves over time by increasing retention.

    There remains a question as to how a system that boosts grades could fail to boost retention. There are a couple potential hypotheses. First of all, it is quite possible that when the numbers are rerun there will still be a significant, though reduced, retention effect, and that reduced effect is still congruent with the better scores.

    Alternately, it could be that students in Course Signals courses score highly in Course Signals-enhanced courses, but at the expense of other courses. My daughter’s math teacher has a very strict policy on math homework which has whipped her into shape in that class, but this means she often delays studying for other things. Students with finite time resources can rearrange their time, but not always expand it.

    Finally, for some nontrivial amount of students, retention problems are not due to grades. Not to push the reverse-causality logic too far, but for some students low grades could be a sign of financial or domestic difficulty; fixing the grade would not address the larger problem.

    What are the larger cultural implications?

    As Michael has outlined in a different post, there are major cultural implications to this error, ones which partially indict the research analytics community’s approach to research. To my knowledge, the study was never peer-reviewed outside of its inclusion in conference proceedings, but it is one of the most referenced studies in learning analytics.

    Technology does move fast enough that old publication cycles do not serve the industry well. But if pre-publication peer-review does not exist, there are a host of things we need to make post-publication review work. We need to release more underlying data, invite more criticism, and separate the PR arm of many organizations from their research arm (or at least insure more autonomy). Additionally, we may need to place more rigorous controls on conference presentation, and make sure that presentations making strong statistical claims undergo a more thorough and profiessional review.

    The cultural implications of an error like this going undetected this long in a community that is supposedly a community of data analysts are also stunning, and will be the subject of a future post. For the moment we are still waiting for Purdue to engage honestly with the critique, and re-run their numbers after controlling for this effect. Hopefully that will happen later this week.

    UPDATE: As Doug notes below, the paper did undergo a full peer review before its inclusion in the LAK conference. I was aware of that, but reading through the post, I realize that is not clear. As I mentioned, we’re looking at putting together a more detailed analysis on how we got here after we know better what the damage is, and will walk through those issues more thoroughly at that time. In the meantime, I’d love to start a conversation about that issue in the comments. Let’s assume that some analytics is sugar water, and some is useful medicine. How do we create a culture and a process that helps us separate one from the other? What’s preventing us from doing that now?

     

     

     

  • State of the Anglosphere’s Higher Education LMS Market: 2013 Edition

    I shared the most recent graphic summarizing the LMS market in September 2012, and thanks to new data sources it’s time for an update. As with all previous versions, the 2005 – 2009 data points are based on the Campus Computing Project, and therefore is based on US adoption from non-profit institutions. This set of longitudinal data provides an anchor for the summary.

    What I’ve been attempting to do lately is to expand the market definition beyond the US. Last year I used some heuristics:

    LMS_MarketShare_20121018-Home

    (more…)

  • Purdue’s Non-Answer on Course Signals

    Inside Higher Ed’s Carl Straumsheim has some reporting on the Course Signals data controversy. He was able to get Purdue research scientist Matt Pistilli on record about it. Here is the sum total of the quotes from the article:

    Pistilli defended the claims about Signals’ ability to increase retention — with the caveat that more research needs to be done. “The analysis that we did was just a straightforward analysis of retention rates,” he said. “There’s nothing else to it.”

    To ensure an empirically grounded analysis of Signals, Essa urged Purdue to give researchers access to as much data as possible. Pistilli said he is open to participate in that conversation, but pointed out that granting open access could violate students’ privacy rights.

    With Signals marking its fifth anniversary this year, Pistilli said “it was probably just a matter of time for people to start looking for these pieces and begin to draw conclusions.” In that sense, the discussion about early warning systems resembles that of other ed-tech innovations, like flipping the classroom and massive open online courses, where hype drowns out any serious criticism.

    Now, I understand that interviews necessarily get edited down for news articles, so I am not going to jump to any deep conclusions about Dr. Pistilli’s views or Purdue’s official positions from these three short paragraphs. But I will make two points that do not require a close reading.

    1. Purdue’s credibility is on the line.

    Purdue has made significant noise about their retention claims. They have published articles and made presentations. They have commercially licensed their system for use by other schools, based in part on the strength of those claims. We now have credible analysis calling some of those claims into question. For its own sake, as well as for that of the academic community, Purdue now needs to go on record with a response to the critique, either acknowledging its legitimacy and amending their claims or demonstrating why the analysis is off base. There is nothing in the quote above to indicate that either Dr. Pistilli or the university understand that they have a substantial problem which demands a substantive answer.

    2. Purdue would not have to violate student privacy in order to answer the concerns being raised.

    Course Signals collects substantial fine-grained data about student activity, some of which could be personally identifying. None of which is necessary in order to respond to the questions being raised. In order to test the retention claims, we only need to see gross outcomes, which are easily anonymized. In fact, I would be surprised if Purdue does not already have an anonymized version of the data. But even if they don’t, the amount of effort required to scrub it should be relatively modest and easily proportional to the level of concern being raised here.

    There is no shame in getting research analysis wrong. It happens all the time, even to the very best researchers. This is why academia places such a high value on peer review, whether it takes place before or after publication. We are smarter as a group than we are individually. However, there is shame in using research on student success to promote the brand of an institution, and to make money, and then decline to open that research up to appropriate scrutiny by the academic community. I am not accusing either Purdue or Dr. Pistilli of doing so at this point. The critical analysis has only recently come to public attention, and I imagine that it takes a little time for any academic institution to formulate and approve an appropriate response. But the clock is ticking.

    If Dr. Pistilli or any other appropriate representative of Purdue who can speak to the substance of the research would like to respond, we certainly would give them air time on e-Literate. It doesn’t have to be here; I am sure there are other appropriate forums. But we are happy to give them an opportunity to respond here if that would help cultivate a dialog.

  • New data available for higher education LMS market

    Despite much talk about the demise of the LMS market, the end is nowhere in sight. Unlike many of the newer learning platform concepts (e.g. MOOCs, free platforms, unbundled learning platforms), the LMS market has an established business model and real revenues. Just today came news of an investment analysis report predicting that total LMS market (higher ed, corporate training, K-12) would triple in revenue by 2018, moving from $2.6B to $7.8B. The LMS ain’t sexy, but it’s still important.

    global_LMS_stacked_by_typeThis is why I have found it surprising how long it is taking to move from survey-based and anecdotal market information to harder data directly measured by the actual LMS implementations. Until recently, that is, when there are at least two very useful sites available.

     

    Edutechnica

    George Kroner, a former engineer at Blackboard who now works for University of Maryland University College (UMUC), has developed what may be the most thorough measurement of LMS adoption in higher education at Edutechnica (OK, he’s better at coding and analysis than site naming). This side project (not affiliated with UMUC) started two months ago based on George’s ambition to unite various learning communities with better data. He said that he was inspired by the Campus Computing Project (CCP) and that Edutechnica should be seen as complementary to the CCP.

    (more…)

  • Course Signals Effectiveness Data Appears to be Meaningless (and Why You Should Care)

    My father likes to say, “If you stick your head in the freezer and your feet in the oven, on average you’ll be comfortable.” Behind this pithy saying is an insight that is a little different from the “three kinds of lies” saying about statistics. It suggests that certain types of analysis produce a false coherence to the world. We see honestly patterns where there aren’t any. This is what appears to have happened with studies that Purdue University has done regarding Course Signals’ effectiveness.

    The problem was speculatively raised and then re-explained by Mike Caulfield (who we are proud to now have as an e-Literate featured blogger):

    From this desk here, without a stitch of research, I can show that people who have had more car accidents live, on average, longer than people that have had very few car accidents.

    Why? Because each year you live you have a chance of racking up another car accident. In general, the older you live, the more car accidents you are likely to have had.

    If you want to know whether people who have more accidents are more likely to live longer becauseof the car accidents, you have to do something like take 40 year-olds and compare the number of 40 year-olds that make it to 41 in your high and low accident groups (simple check), or use any one of a number of more sophisticated methods to filter out the age-car accident relation.

    The Purdue example is somewhat more contained, because the event of taking a Course Signals class or set of classes happens once per semester.  But what I am asking is whether

    1. the number of classes a student took is controlled, for, and more importantly,
    2. whether first to second year retention is calculated as
      1.  the number of students that started year two / the number of students who started year one (our car accident problem), or
      2.  the number of students who started year two / the number of students who finished year one (our better measure in this case).

    Pointing to the first of these posts, I suggested that a response from the Purdue researchers would be helpful. (It still would be.) Now Al Essa, recently of Desire2Learn and currently of McGraw Hill, has done a more mathematically rigorous analysis showing that Mike’s intuition appears to be correct. He took the Purdue findings, substituted the phrase “were given a chocolate” for “took a class using Course Signals,” and ran a simulation to see whether he could reproduce the Purdue results with no causal connection between the chocolatey intervention and the retention results:

    The following are some results from the simulation. The first row displays retention rates for students who received no chocolates. The second row displays retention rates for students who received at least one chocolate. The last row shows students who received two or more chocolates. Why track students who received two or more chocolates? Because the authors of the study claim that two is the “magic number” where significant retention gains kick in.

    The simulation data shows us that the retention gain for students is not a real gain (i.e. causal) but an artifact of the simple fact that students who stay longer in college are more likely to receive more chocolates. So, the answer to the question we started off with is “No.”. You can’t improve retention rates by giving students chocolates.

    This is a problem that goes well beyond Course Signals itself for several reasons. First, both Desire2Learn and Blackboard have modeled their own retention early warning systems after Purdue’s work. For that matter, I have praised Course Signals up and down and criticized these companies for not modeling their products more closely on that work, largely based on the results of the effectiveness studies. So we don’t know what we thought we knew about effective early warning systems. The fact that the research results appear to be spurious does not mean that systems like Course Signals has no value, but it does mean that we don’t have the proof that we thought we had of their value.

    More generally, we need to work much harder as a community to critically evaluate effectiveness study results. Big decisions are being made based on this research. Products are being designed and bought. Grants are being awarded. Laws are starting to be written. I believe strongly in effectiveness research, but I also believe strongly that effectiveness research is hard. The Purdue results have been around for quite a while now. It is disturbing that they are only now getting critical examination.

    Update: Seeing some of the posts in the comments thread, I feel the need to make a clarification in fairness to Purdue. They have two important sets of research findings, only one of which is being called into question here. Their early findings, that Course Signals can increase student grades and chances of completion within a class, are not being challenged here. Those are important results. The findings that are being questioned are their longitudinal analysis showing that students who Course Signals in one class are more likely to do well in future classes. Even there, it is possible that Course Signals does have some long-term effect. The point is simply that the result the researchers got in their analysis looks suspiciously similar to the results one would get from a fairly straightforward selection bias.

  • Comment from member of research team on USA Today flipped classroom article

    Update 10/26: We now have Rachel Levy and Nancy Lape (who was the researcher interviewed by USA Today) both agreeing with Darryl’s comments. That’s three of the four members of the research team. While I do not claim to understand how the reporter developed her story line (I have asked for comment), it is quite clear that the article does not represent the work of the Harvey Mudd research team and has a misleading headline and lede.

    As a follow up to my response to the USA Today article on flipped classroom research, there was a very informative comment one Google+ from Darryl Yong (one of the members of the Harvey Mudd research team). I thought this comment deserved to be seen by a wider audience, so I’m reproducing in full.

    Thanks, Phil, for this post. The USA Today article paints an inaccurate picture of our work and I wanted to try to clarify some things and continue this conversation with you and your readers. (I am only writing on my own behalf and not for my collaborators on this study.)

    My biggest regret is that the article greatly oversimplifies things by portraying our study as an attempt to answer whether flipped classrooms work or not. That kind of research question is too blunt to be useful. Our goal is to better understand the conditions under which flipped classrooms lead to better student outcomes. As Phil and others here point out, there are many different manifestations of what we mean by the term “flipped classroom” and that we should be wary about talking about it as if there is one canonical implementation. How are the benefits of flipped classrooms affected by school and student contexts, and are there other benefits that we haven’t yet characterized? We don’t have any preconceived answers to these research questions. While folks might disagree with us whether these research questions are interesting or not, they should at least know that we’re aware of the good work that others have already done and we’re trying to build on it rather than debunk it.

    (more…)

  • Educational Technology and the Sources of Innovation

    Based on Jon Udell’s excellent post, I spent the weekend getting reacquainted with work of Eric von Hippel, the researcher who pioneered the study of user-driven innovation.

    What’s interesting about von Hippel is that his research hits on the common themes of the open education movement, but does so in a slightly different key.

    Briefly, there are a number of intersecting debates about MOOCs. There is what Reich frames as the Dewey/Thorndike debate about what learning is. There is the centralized/de-centralized debate about what the web does best. There is the debate about about whether MOOCs are disruptive or innovative or neither, and the discussion over how much ability to remix teachers need to make classroom learning work well (answer, probably, is quite a bit).

    But people on both sides of the debates are often driven by a larger question that we are not naming directly enough: “What are the sources of innovation?”

    User Innovation

    This is the question that von Hippel has been investigating for over thirty years now. And if we see innovation not as something that has happened, but as something we want to continue to happen, this may be the most important question of all.

    (more…)