e-Literate

Present is Prologue

Category: Outcomes

  • Adaptive Learning Fails to Make the Grade. Or Does It?!

    My latest Chronicle column is up. It analyzes the results of the SRI Education study of the Gates Foundation adaptive learning grantees, some of which we’ve covered in our e-Literate TV case studies. If you’re looking for evidence that adaptive learning is going to deliver on the promise of a robot tutor in the sky, you won’t find it there. But it’s easy to flatten that result into “adaptive learning doesn’t work.” I don’t believe that the SRI study shows any such thing.

    First of all, what is our standard of proof? A good half of my column is devoted to the methodological challenges of doing big meta-studies like this one. It’s really hard to (ethically) control the variables across multiple classrooms well enough to get a clean result. SRI had to throw out most of the data they had for some measures.

    But equally importantly, there’s just a lot that meta-studies can’t get at precisely because the goals of each implementation are different. For example, one of the goals for implementing OLI at UC Davis was getting students more prepared to engage in higher-level critical thinking in class discussion. Here are some Davis faculty talking about their course design goals:

    I’m not sure how one would empirically measure such a result; nor could I see how to incorporate it into a meta-study that also includes, for example, Essex County College’s developmental math course.

    While the Gates Foundation should get credit for bringing in a credible third-party evaluator to review the results of the grants, ((Full disclosure: Our company received grant money from the Gates Foundation to cover personalized learning)) the design parameters for this particular study do not appear to be as useful as they might have been. That said, the larger point is that it’s really hard to do educational research well. Rather than using these studies as Rorchach tests, we should be taking the time to improve our educational research literacy and better understand what each study can and cannot prove.

  • SRI’s Study on Gates Personalized Learning Grants Is Out

    This is almost old news now, but we just haven’t been able to dig into it yet. As part of its Adaptive Learning Market Acceleration Program (ALMAP) program, the Gates Foundation funded SRI to do a study of the results of the grants after two years. I hope to finally clear some time to parse through it this week, but two high-level points jump out at me at first glance (neither of which is a big surprise):

    1. There are no conclusive wins here. This is not a robot-tutor-in-the-sky moment. A few programs did well here and there. A handful produced promising incremental gains. But this is not a report that screams, “Wow, adaptive courseware works!” The most you can say is that adaptive learning looks like it could be another arrow in the quiver that helps out in some situations—like developmental math courses at two-year colleges, for example.
    2. Large-scale educational research is incredibly hard and may actually be impossible to do rigorously for certain kinds of questions. I’ll probably get into this more when I’m ready to really write about the report, but one reason the conclusions are murky is because there so many variables in each class—not just each course subject, not just each course at one university, but even with each section of each class taught by one teacher—that really matter, some of which are impossible to control and others of which are unethical to control. It would be a mistake to overinterpret the study as showing that adaptive learning doesn’t help much. I’ve seen studies with narrower focus that have gotten clearer gains. This brings us back to the arrow-in-the-quiver theory. The wider the scope of the research focus, the more that pockets of real benefit will be obscured by noise and uncontrolled variables.

    The report is here.

  • Student Course Evaluations and Impact on Active Learning

    The Chronicle has an article out today, “Can the Student Course Evaluation Be Redeemed?”, that rightly points out how student course evaluations are often counter-productive to improving teaching and learning. The article refers to a Stanford professor’s call for an instructor completed “inventory of the research-based teaching practices they use”, but most of the article centers on revised course evaluation tool from a Kansas State University spin-off (the IDEA Center). One of the key problems described is that “administrators often take their results as numerical gospel” as well as faculty misapplying the results.

    However they’re used, a lot of course evaluations simply aren’t very good, [IDEA president] Mr. Ryalls says.

    But as flawed as they are, faculty members still turn to them as some gauge of effectiveness in the classroom. About three-quarters of instructors use formal evaluations and informal feedback “quite a bit” or “very much” when altering their courses, according to the Faculty Survey of Student Engagement.

    One limitation of many tools is that they ask students things they don’t really know. A frequent example: Was your instructor knowledgeable about course content?

    There is one additional problem with most student course evaluations that is not explicitly covered in the Chronicle articles – students newly involved in active learning approaches often rate the course and instructor poorly even if they end up learning more effectively. We saw this in our e-Literate TV case study at UC Davis. In a previous post we highlighted how the routine hard work required of students in active learning courses can lead to poor evaluations, but later in the interview student course evaluations came up as a major barrier to improving teaching practices. (more…)

  • Asking What Students Spend On Textbooks Is Very Important, But Insufficient

    Mike Caulfield responded to my post on data usage to understand college textbook expenditures. The core of my argument is a critique of commonly cited College Board data. That data originating from financial aid offices leads to the conclusion that students on average either spend or budget $1,200 per year with that number rising, while there is more reliable data originating from students showing the number to be half that amount and dropping.

    In Mike’s response post yesterday, he generally agreed with the observation but is concerned that “readers of that piece are likely to take away the wrong conclusion from Phil’s figures (even if Phil himself does not)”. There is a risk that people see the lower numbers and conclude the “crisis is overblown”, leading to this observation:

    If we’re looking to find out if prices for some set of goods are too high, then by definition we cannot look at what people are spending as a reliable gauge, because one of the big effects of “prices too high” is that people can’t afford what they need.

    If you don’t pay attention to this you get in all sorts of tautologies.

    In the specific world of textbooks, Mike considers the lower-cost method of renting used textbooks, noting:

    So which figure do we use here? The chances of getting everything you need as a rental are low. Sure, you could be the super-prepared student who knows how to work the system and get them *all* as rentals — but not every student can be first in line at the bookstore. And the ones at the back of the line — guess their socio-economic class and first generation status?

    This is an important issue, and I appreciate Mike’s understanding that I am not arguing that college textbook pricing is an overblown crisis. I agree that the crisis is real and that the hardest-hit are likely low socio-economic class and first generation students.

    But let’s move past these agreements and drop the gloves. (more…)

  • College Scorecard Article Published In Washington Post

    I have written several posts looking at the new College Scorecard and its inherent flaws in the data, often starting with observations from Russ Poulin at WCET. Today Susan Svrluga, education reporter at the Washington Post, posted a new article co-written by me and Russ and titled “Hundreds of colleges missing from Obama’s College Scorecard?”. The gist of the article is taking a holistic view of data problems and why they exist.

    In a nutshell, the College Scorecard combines data from multiple sources – primarily from the Education Department’s own Integrated Postsecondary Education Data System (IPEDS) and National Student Loan Data System (NSLDS) – and publishes the results of both as a consumer-facing Web site and an analyst-friendly data download. The essence of the problem is that throughout this process the data is filtered based on questionable assumptions, leading to the fuzzy lens viewing subsets of the real data.

    Poulin Hill College Scorecard Graphic

  • Personalized Learning is Hard

    Paul Fain has written a really good, nuanced article at IHE covering the update that Essex County College gave of their developmental math adaptive learning pilot at a recent conference in Washington, DC. We did a two–part case study on ECC in our e-Literate TV series). The headline results are as follows:

    • In the first year, the pass rate was worse than  in the traditional classes. (The first semester was “disastrous.”)
    • This year—the second year—the pass rate is coming closer to the traditional class but is still underperforming.
    • The article seems to imply that students who earn a C in the personalized learning class do better than students who earn a C in the traditional class, but the article is not explicit about that.

    There is no magic pill. As Phil and I have been saying all along—most recently in my last post, which mentioned ECC’s use of adaptive learning—the software is, at best, an enabler. It’s the work that the students and teachers do around the software that makes the difference. Or not. In ECC’s case, they are trying to implement a pretty radical change in pedagogy with an at-risk population. It’s worth digging into the details.

    (more…)

  • ED and CBE: Example of higher ed “structural barrier to change” that is out of institutions’ control

    There has been a great conversation going on in the comments to my recent post “Universities As Innovators That Have Difficulty Adopting Their Own Changes” on too many relevant issues to summarize (really, go read the ongoing comment thread). They mostly center on the institution and faculty reward system, yet those are not the only sources of structural barriers to change that lead institutions to this “difficulty adopting their own changes”. Increasingly there are outside forces that both encourage change and resist change, and it is important to recognize the impact of the entire higher education ecosystem.

    Yesterday Amy Laitinen from New America wrote an excellent article titled “Whatever Happened to the Department’s Competency-Based Education Experiments?” highlighting just such an example.

    About this time two years ago, President Obama went on his college affordability bus tour and unveiled his plan to take on the rising costs of higher education in front of thousands of students at SUNY Buffalo. Promoting innovation and competition was a key part of his plan and President Obama held up competency-based education (CBE) up as one of the “innovative new ways to prepare our students for a 21st century economy and maintain a high level of quality without breaking the bank.” The President touted Southern New Hampshire University’s College for America CBE approach. The university “gives course credit based on how well students master the material, not just on how many hours they spend in the classroom,” he explained. “So the idea would be if you’re learning the material faster, you can finish faster, which means you pay less and you save money.” This earned applause from students in the audience as well as from CBE practitioners around the country. [snip]

    The problem is that day was nearly two years ago and the CBE experimental sites are not yet off the ground. (more…)