e-Literate

Present is Prologue

Tag: Intelligent Tutoring Systems

  • Response to Robert Talbert: Pedagogical change is difficult, many need support

    On Monday Robert Talbert, associate professor at Grand Valley State University and author of the Casting Out Nines blog, wrote a provocative and important post titled “Active learning as an ethical issue”. Robert noted:

    The recent Proceedings of the National Academy of Sciences study stands out among these recent studies. It is a meta-study of 225 prior studies on active learning, and the results are bracing: students in these studies who were in classes focused on lecture and direct instruction in the classroom were 55% more likely to fail their courses than their counterparts in active learning focused classes, and scored almost half a standard deviation lower than their active learning counterparts on exams.

    This sentence from the PNAS study stopped me in my tracks when I first read it:

    “If the experiments analyzed here had been conducted as randomized controlled trials of medical interventions, they may have been stopped for benefit—meaning that enrolling patients in the control condition might be discontinued because the treatment being tested was clearly more beneficial.”

    Robert’s central point is that active learning should be thought of as an ethical issue, where it could be considered unethical to withhold treatment. He then asks why faculty might withhold active learning and listed four reasons: self-preservation, laziness, a weird and irrational superiority complex, and legitimate external forces (such as overly controlling school structure).

    The argument is an interesting and compelling one based on the study, and it is worth reading the whole article and his follow-up post. I wish we treated teaching and learning more often as an ethical issue,but I would add one additional reason that the active learning treatment is not more prevalent. This one comes from our discussions with faculty and support staff as part of our e-Literate TV series on personalized learning, and Michael and I summarized the point in the introduction episode. (more…)

  • Promising Research Results On Specific Forms Of Adaptive Learning / ITS

    Recently I described an unpublished study by Dragan Gasevic and team on the use of Knowillage / LeaP adaptive platform. ((When the study started Knowillage was an independent company; mid-way through study D2L bought Knowillage and renamed product as LeaP.)) The context of article was on D2L’s misuse of the results, but the study itself is interesting in terms of its findings that adaptive learning usage (specifically LeaP in addition to Moodle within an Intro to Chemistry course) can improve academic performance. I will share more when and if the results become public.

    If we look to published research reports there are other studies that back up the potential of adaptive approaches, but the most promising results appear to be for a subset of adaptive systems that provide not just content selection but also tutoring. Last year a research team from Simon Fraser University and Washington State University published a meta-analysis on Intelligent Tutoring Systems (ITS) which they described as having origins from 1970 and the development of SCHOLAR. ((I would link to G+ post by George Station here if it were not for the ironic impossibility of searching within that platform.)) The study looked at 107 studies involving 14,321 participants and found:

    The use of ITS was associated with greater achievement in comparison with teacher-led, large-group instruction (g .42), non-ITS computer-based instruction (g .57), and textbooks or workbooks (g .35). There was no significant difference between learning from ITS and learning from individualized human tutoring (g –.11) or small-group instruction (g .05). Significant, positive mean effect sizes were found regardless of whether the ITS was used as the principal means of instruction, a supplement to teacher-led instruction, an integral component of teacher-led instruction, or an aid to homework. Significant, positive effect sizes were found at all levels of education, in almost all subject domains evaluated, and whether or not the ITS provided feedback or modeled student misconceptions. The claim that ITS are relatively effective tools for learning is consistent with our analysis of potential publication bias.

    (more…)

  • No, really, courseware is a thing now

    In the operating plan slide deck that Cengage recently released as a consequence of their bankruptcy proceedings, the executive summary slide says that a key element of their strategy is “driving aggressive digital growth in a course model.” “Course solutions” is mentioned three times in the deck as well. Cengage, as a company, is essentially betting its future on courseware. Not just digital products in general, but courseware in particular.

    But they are hardly the only provider building content in this relatively new category. I thought it might be useful to provide a run-down of who is doing what in this space. It turns out that there is a pretty wide range in terms of approaches to the product category.

    (more…)

  • A Taxonomy of Adaptive Analytics Strategies

    I almost never quote a blog post in its entirety, but this one from Dan Meyer is so good that I just can’t bear to cut a single word:

    Stephanie Simon, reporting for Reuters on inBloom and SXSWedu:

    Does Johnny have trouble converting decimals to fractions? The database will have recorded that – and may have recorded as well that he finds textbooks boring, adores animation and plays baseball after school. Personalized learning software can use that data to serve up a tailor-made math lesson, perhaps an animated game that uses baseball statistics to teach decimals.

    Three observations:

    One, it shouldn’t cost $100 million to figure out that Johnny thinks textbooks are boring.

    Two, nowhere in this scenario do we find out why Johnny struggles to convert decimals to fractions. A qualified teacher could resolve that issue in a few minutes with a conversation, a few exercises, and a follow-up assessment. The computer, meanwhile, has a red x where the row labeled “Johnny” intersects the column labeled “Converting Decimals to Fractions.” It struggles to capture conceptual nuance.

    Three, “adores” protests a little too much. “Adores” represents the hopes and dreams of the educational technology industry. The purveyors of math educational technology understand that Johnny hates their lecture videos, selected response questions, and behaviorist video games. They hope they can sprinkle some metadata across those experiences — ie. Johnny likes baseball; Johnny adores animation — and transform them.

    But our efforts at personalization in math education have led all of our students to the same buffet line. Every station features the same horrible gruel but at its final station you can select your preferred seasoning for that gruel. Paprika, cumin, whatever, it’s yours. It may be the same gruel for Johnny afterwards, but Johnnyadores paprika.

    Dan captures most of what I was trying to get at with my rant on the big data hype, but much more clearly and succinctly. Points two and three are the most salient here. First of all, the sort of surface-level analysis we can get from applying machine learning techniques to the current data we have from digital education system is insufficient to do some of the most important diagnostic work that real human teachers do. Think about the math classes is which you had to show your work on your homework. Why was that important? Because the teacher needs to see not only what you got wrong but why you got it wrong. Teachers generally don’t just say, “You got three out of five problems involving converting decimals to fractions wrong. Go study some more.” They sit down and work through the problems with the student to find the source of the errors. It’s really hard to get computers to do this well, even with highly procedural domains like math. (Forget about, say, literary analysis.) So in the vast majority of cases, we don’t even try to design systems where students show their work. And without the step-by-step data, no fancy algorithm is going to teach Johnny.

    Second, if the problem is that your content isn’t what the student needs, no fancy algorithm is going to fix that either. Videos are a prime example. I know of one textbook publisher whose teacher customers report that students won’t watch the publishers’ videos, but they can and do find videos on the same topic on YouTube and share them with each other. Think about that. Video-based pedagogical support is valuable enough to the students that they will expend energy searching for videos and sharing them. But they reject the expensive, carefully crafted videos from the publisher that are served up to them on a silver platter. It’s not that the publisher-supplied videos are necessarily “bad” in the sense that they have poor production qualities or are unclear or factually inaccurate. But the students have a particular use in mind for the videos. Maybe they’re struggling with a particular homework problem and just need a quick walk-through of a technique so that they can see the step that they are missing, for example. If the video doesn’t fit their needs—both utilitarian and aesthetic—then it won’t get used. Serving it up adaptively isn’t going to help that problem.

    That said, it’s worth taking a little time to break down the different types of adaptive learning analytics into a couple of categories and see just what we should and should not reasonably hope to gain from them.

    (more…)

  • Where xMOOCs and Adaptive Analytics Both Fail (For Now)

    No, this isn’t just an attempt to cram as many sexy keywords into one post title as possible. xMOOCs and adaptive analytics share an ambition: They both are at least partially motivated by a desire to teach at scale. With MOOCs, the goal is obvious. With adaptive analytics, less so, partly because there are multiple motivations and maybe because the desire for scale is not something that is polite to talk about due to a certain amount of discomfort with it. But the motivation is definitely there if you look closely, as I’ll get into in a bit.

    The problem is that both of these approaches, in their current incarnations, miss one absolutely critical element of the teaching process. As you can probably guess from all the qualifiers I am using in my language, I don’t think it’s an inherent or permanent failing. But I worry that it’s a failing due to a deep cultural blind spot that we have about what education is, and that it therefore will be challenging to address.

    (more…)