e-Literate

Present is Prologue

Tag: SRI

  • Adaptive Learning Fails to Make the Grade. Or Does It?!

    My latest Chronicle column is up. It analyzes the results of the SRI Education study of the Gates Foundation adaptive learning grantees, some of which we’ve covered in our e-Literate TV case studies. If you’re looking for evidence that adaptive learning is going to deliver on the promise of a robot tutor in the sky, you won’t find it there. But it’s easy to flatten that result into “adaptive learning doesn’t work.” I don’t believe that the SRI study shows any such thing.

    First of all, what is our standard of proof? A good half of my column is devoted to the methodological challenges of doing big meta-studies like this one. It’s really hard to (ethically) control the variables across multiple classrooms well enough to get a clean result. SRI had to throw out most of the data they had for some measures.

    But equally importantly, there’s just a lot that meta-studies can’t get at precisely because the goals of each implementation are different. For example, one of the goals for implementing OLI at UC Davis was getting students more prepared to engage in higher-level critical thinking in class discussion. Here are some Davis faculty talking about their course design goals:

    I’m not sure how one would empirically measure such a result; nor could I see how to incorporate it into a meta-study that also includes, for example, Essex County College’s developmental math course.

    While the Gates Foundation should get credit for bringing in a credible third-party evaluator to review the results of the grants, ((Full disclosure: Our company received grant money from the Gates Foundation to cover personalized learning)) the design parameters for this particular study do not appear to be as useful as they might have been. That said, the larger point is that it’s really hard to do educational research well. Rather than using these studies as Rorchach tests, we should be taking the time to improve our educational research literacy and better understand what each study can and cannot prove.

  • SRI’s Study on Gates Personalized Learning Grants Is Out

    This is almost old news now, but we just haven’t been able to dig into it yet. As part of its Adaptive Learning Market Acceleration Program (ALMAP) program, the Gates Foundation funded SRI to do a study of the results of the grants after two years. I hope to finally clear some time to parse through it this week, but two high-level points jump out at me at first glance (neither of which is a big surprise):

    1. There are no conclusive wins here. This is not a robot-tutor-in-the-sky moment. A few programs did well here and there. A handful produced promising incremental gains. But this is not a report that screams, “Wow, adaptive courseware works!” The most you can say is that adaptive learning looks like it could be another arrow in the quiver that helps out in some situations—like developmental math courses at two-year colleges, for example.
    2. Large-scale educational research is incredibly hard and may actually be impossible to do rigorously for certain kinds of questions. I’ll probably get into this more when I’m ready to really write about the report, but one reason the conclusions are murky is because there so many variables in each class—not just each course subject, not just each course at one university, but even with each section of each class taught by one teacher—that really matter, some of which are impossible to control and others of which are unethical to control. It would be a mistake to overinterpret the study as showing that adaptive learning doesn’t help much. I’ve seen studies with narrower focus that have gotten clearer gains. This brings us back to the arrow-in-the-quiver theory. The wider the scope of the research focus, the more that pockets of real benefit will be obscured by noise and uncontrolled variables.

    The report is here.

  • Adaptive Learning Market Acceleration Program (ALMAP) Summer Meeting Notes

    I recently attended the ALMAP Summer Meeting. ALMAP is a program funded by the Gates Foundation, with the goals described in this RFP webinar presentation from March 2013:

    We believe that well implemented personalized & adaptive learning has the potential to dramatically improve student outcomes

    Our strategy to accelerate the adoption of Adaptive Learning in higher education is to invest in market change drivers… …resulting in strong, healthy market growth

    As the program is in its mid stage (without real results to speak of yet), I’ll summarize Tony Bates style with summary of program and some notes at the end. Consider this my more-than-140-character response to Glenda Morgan:

    Originally planned for 10 institutions, the Gates Foundation funded 14 separate grantees at a level of ~$100,000 each. The courses must run for 3 sequential semesters with greater than 500 students total (per school), and the program will take 24 months total (starting June 2013). The awards were given to the following schools:

    (more…)