e-Literate

Present is Prologue

Category: Pedagogy

  • Pearson, Efficacy, and Research

    A while back, I mentioned that MindWires, the consulting company that Phil and I run, had been hired by Pearson in response to a post I wrote a while back expressing concerns about the possibility of the company trying to define “efficacy” in education for educators (or to them) rather than with them. The heart of the engagement was us facilitating conversations with different groups of educators about how they think about learning outcomes—how they define them, how they know whether students are achieving them, how the institution does or doesn’t support achieving them, and so on. As a rule, we don’t blog about our consulting work here on e-Literate. But since we think these conversations have broader implications for education, we asked for and received permission to blog about what we learn under the following conditions:

    • The blogging is not part of the paid engagement. We are not obliged to blog about anything in particular or, for that matter, to blog at all.
    • Pearson has no editorial input or prior review of anything we write.
    • If we write about specific schools or academics who participated in the discussions, we will seek their permission before blogging about them.

    I honestly wasn’t sure what, if anything, would come out of these conversations that would be worth blogging about. But we got some interesting feedback. It seems to me that the aspect I’d like to cover in this post has implications not only for Pearson, and not only for ed tech vendors in general, but for open education and maybe for the future of education in general. It certainly is relevant to my recent post about why the LMS is the way it is and the follow-up post about fostering better campus conversations. It’s about the role of research in educational product design. It’s also about the relationship of faculty to the scholarship of teaching.

    (more…)

  • Why Big Data (Mostly) Can’t Help Improve Teaching

    Here’s a nifty video summary of a doctoral dissertation by Derek Muller that a client pointed out to me:

    The basic gist is that students have pre-conceived notions that are wrong, and it is very hard to dislodge those mistaken notions. If you show them a video with an accurate explanation, the students will say that the video was clear and helpful, but they will misremember it as confirming their (mistaken) preconceived notions. In short, they won’t learn. In contrast, if you show them a video that starts by directly stating and then refuting their misconception, they like the video less and say it is confusing, but they actually learn more. This is a really important pedagogical point to know whether you are giving traditional in-class lectures, writing curricular materials, or creating one of those oh-so-modern video lectures that all the cool kids are into these days.

    It’s also a good example of the kind of insight that big data is completely blind to. And it gives us good reason to be skeptical that taking large lecture courses online, turning them into REALLY large lecture courses (with nice videos), and expecting that new and more effective pedagogies will rise out of the data because, you know, science or something, is more of a hope (or a fantasy) than a plan to improve education.

    (more…)

  • Where xMOOCs and Adaptive Analytics Both Fail (For Now)

    No, this isn’t just an attempt to cram as many sexy keywords into one post title as possible. xMOOCs and adaptive analytics share an ambition: They both are at least partially motivated by a desire to teach at scale. With MOOCs, the goal is obvious. With adaptive analytics, less so, partly because there are multiple motivations and maybe because the desire for scale is not something that is polite to talk about due to a certain amount of discomfort with it. But the motivation is definitely there if you look closely, as I’ll get into in a bit.

    The problem is that both of these approaches, in their current incarnations, miss one absolutely critical element of the teaching process. As you can probably guess from all the qualifiers I am using in my language, I don’t think it’s an inherent or permanent failing. But I worry that it’s a failing due to a deep cultural blind spot that we have about what education is, and that it therefore will be challenging to address.

    (more…)

  • Open Secret: Pittsburgh’s Ed Tech Revolution

    Generally when we talk about goals for educational technology, we talk about one of two things: improving access or improving effectiveness. Rarely do we get an opportunity to talk credibly about an innovation that can move both of those needles at the same time. And yet, I saw just such innovation in Pittsburgh this week at the LearnLab Corporate Partners Meeting. LearnLab, jointly run by Carnegie Mellon University and the University of Pittsburgh, is not widely known in ed tech circles. However, it’s close cousin OLI is. For example, an independent study by ITHAKA S+R made a splash when it showed OLI could cut instructional time in half and still achieve the same level of effectiveness. Two groups of students across six different public universities took the same introductory statistics class. The first group took a traditional version of the class, with an average of 3 hours per week spent in the classroom. The second group took a hybrid version, where they spent one hour a week in the classroom and some time studying independently with the OLI cognitive tutoring software. So that second group spent one-third of the time in the classroom, which amounts to a very significant cost savings and an opportunity to teach more students for less money. Students themselves spent more time on homework in the second group than the first, but even so, they spent 25% less total time on the class than the ones in the traditional class. And yet, both groups achieved essentially the same competence at the end of the class:

    To sum up, OLI was able to teach more students at lower cost and with less time on task from the students yet with the same effectiveness of a traditional class. Another, smaller study with similar traditional vs. cognitive-tutor-hybrid for an introduction to logic course found that while students in the hybrid condition saw a small (though not statistically significant) gap in learning effectiveness relative to their peers, they achieved more than double the completion rate:

    OLI is pretty well known in the broader ed tech community. They show up at conferences. The ITHAKA study got lots of play. I find it odd that we have not had a broader conversation about how they have achieved their results. Because the techniques developed by OLI, their colleagues at LearnLab, and their collaborators in other institutions have serious implications for OER, MOOCs, the textbook industry, and really, the future of education.

    (more…)

  • Going Meta on Khan

    The wonderful Dan Meyers points to this delightful Mystery Science Theater 3000 homage by John Golden and David Coffey as they critique one of Sal Khan’s math videos. Here’s the video:

    Everybody enjoys a good snarkfest, and these guys are particularly good at it. But those who are taking delight in seeing Sal Khan get “taken down a peg” are completely missing the value and innovation here. To me, the important thing is that these guys have adopted one of Khan’s signature moves—short, low-production-value, personal web videos—to critique pedagogical technique. This has great value for math teachers who may miss some of the finer points of math instruction (e.g., consistency in terminology) that Golden and Coffey point out in their review. It’s basically a Khan Academy-style lesson for teachers, and it should be able to scale in production just as Khan Academy itself has. I would love to see many, many more video reviews of instructional materials on the web.

    To their credit, Khan Academy has taken down the video in question, presumably to improve the video based on the critique. One of the premises of OER is that people will improve the quality of materials. The assumption is generally that they will do so themselves by editing, but this is a demonstration that just having the content out in the open where the author can get feedback is of great value too.

  • How to Keynote an Unconference

    A while back, I had the privilege of being the keynote speaker at the NERCOMP LMS Unconference. I had never attended an unconference before, nevermind keynoting one, and I found the prospect to be fascinating and exciting. And nerve-wracking. On the surface, a keynote appears to be the antithesis of the unconference spirit. I needed to do something different than the usual fare in order to make it work. I needed to do an unkeynote. And yet, Stephen Downes had warned me that he, Brian Lamb, and D’Arcy Norman had tried giving an unkeynote before and, in his words, “They almost lynched us. They were not happy to receive an unkeynote.” (D’Arcy’s post-mortem of their effort is definitely worthwhile reading.) So, what to do?

    The approach I tried seemed to work, judging by the feedback I got from the attendees and, to a lesser degree, by the influence of the presentation that I was able to observe on the rest of the unconference. I had intended to blog about the experience a while ago but it fell off my to-do list. However, prompted by the good folks of the NERCOMP LMS SIG, I am now returning to the topic.

    (more…)

  • What Is Machine Learning Good For?

    A few weeks ago, Audrey Watters wrote a great piece on her concerns about robo-grading of essays. (I tend to take a lot of inspiration from the things that annoy Audrey, in part because they usually annoy me too.) Here’s the crux of her argument:

    According to Steve Kolowich’s Inside Higher Ed story, [educational researcher Mark] Shermis “acknowledges that [Automated Essay Scoring] software has not yet been able to replicate human intuition when it comes to identifying creativity. But while fostering original, nuanced expression is a good goal for a creative writing instructor, many instructors might settle for an easier way to make sure their students know how to write direct, effective sentences and paragraphs. ‘If you go to a business school or an engineering school, they’re not looking for creative writers,’ Shermis says. ‘They’re looking for people who can communicate ideas. And that’s what the technology is best at’ evaluating.”

    Why are nuance and originality just the purview of the creative writing department? Why are those things seen here as indirect or ineffective? Why do we think creativity is opposed to communication?  Is writing then just regurgitation?

    What sorts of essays gain high marks among the SAT graders – human now or robot in the future? Are these the sorts of essays that students will be expected to write in college? Is this the sort of writing that a citizen/worker/grown-up will be expected to produce?  Or, for the sake of speed and cost effectiveness, in Vander Ark’s formulation, are we promoting one mode of writing for standardized assessments at the K–12 level, only to realize when students get to college and to the job market that, alas, they still don’t know how to write?

    How can we get students to write more? How can we help them find their voice and hone their craft? How do we create authentic writing assignments and experiences – ones that appeal to real issues and real discourse communities, not just to robot graders? How do we encourage students to find something to say and to write that something well?  Is that by telling them that their work will be assessed by an automaton?

    How do we support the instructors who have to read student papers and offer them thinking and writing guidance? When we talk about saving time and money here, whose bottom line are we really looking out for?  Who’s really interested in this robot grader technology?  And why? [Emphasis added.]

    This is a classic case of a market gone awry. Machine learning is sold as an “efficiency” tool, because there is money in squeezing cost out of education. In and of itself, there’s nothing wrong with wanting education to be cost-effective. David Wiley’s formulation of “standard deviations per dollar” has both a numerator and a denominator. You can attack either number and still affect the ratio. The problem with obsessing over the denominator is that you start forgetting that “cost-effective” has to be effective. If you want to know what the ongoing industrialization of education looks like in the post-industrial world, robo-grading is it. We are reducing the evaluation to the least common denominator, where the denomination is in dollars.

    But it doesn’t have to be that way. What if we looked at machine learning (the technology that makes robo-grading possible) from the perspective of trying to raise the numerator, i.e., effectiveness, while keeping cost the same? How could the technology be used as a force multiplier for good teachers, helping them to focus on what they do best in roughly the same way that flipping the classroom is supposed to do? If the goal is teaching better rather than just teaching cheaper, then what is machine learning good for?

    (more…)