e-Literate

Present is Prologue

Tag: Machine learning

  • Can There Be a Microscope of the Mind?

    Can There Be a Microscope of the Mind?

    In my last post, I made an extended analogy between today’s ed tech and 19th Century medicine. My core argument was that effective ed tech cannot evolve without a trained profession of self-consciously empirical educators any more than effective medication could have evolved without a profession of self-consciously empirical physicians.

    In this post, I’d like to go beyond analogies and look at the actual state of some cutting-edge cognitive science. I want to do this for several reasons. First, a lot of educators are skeptical or even cynical regarding the potential relevance of this work to the ways that they think about teaching. This is completely understandable, particularly given that most educators hear about this sort of research through product commercials or hyperbolic media puff pieces. By exploring the science in some detail, I want to show that having a basic understanding of even foundational research that has no direct classroom applications can stimulate the thinking of classroom educators in useful ways.

    Second, I want to show that even educators with no background in science or math can achieve an empowering level of cognitive science literacy with a reasonable investment of time (like the time it takes to read a long blog post, for example).

    And finally, I want to show that, after we strip away the hype and the ennui it engenders, we can recover a sense of wonder about the science while maintaining a sense of realism about its practical applicability. I have chosen to characterize the methodological paper I’ll be explaining as an attempt to create a “microscope of the mind.” That’s dangerously close to “robot tutor in the sky that can semi-read your mind” territory. I hope to demonstrate that there is a non-hyperbolic sense in which we can believe that my microscope of the mind analogy is a reasonable one.

    Here’s how I’m going to do it:

    I am going to explain a research study on cognitive neuroscience. It’s not a big, sexy paper that gets coverage in outlets like Wired. It’s a methodology study. I’m going to explain enough of the basic underlying concepts in math, physics, and cognitive psychology for you to be able to get the gist of the paper and judge its significance for yourself. I’m going to explain how fMRIs work and what machine learning is. And I’m going to explain the larger context of why the researchers tried this particular experiment and how it is relevant to our larger understanding of how people learn. Along the way, I will touch on topics as diverse as theology, Russian literature, and the miracle of selfies, but always with the goal of showing how the science can be accessible, interesting, and relevant to non-scientists.

    This is not a short read, but I hope that it will reward your effort.

    (more…)

  • More Blogging on Automated Essay Grading

    Sometime guest blogger and friend of e-Literate Elijah Mayfield has another great post up on using machine learning tools in the service of improving student writing over at his company blog. However you may feel about the technology, the exploration that he’s doing raises some important question about what good feedback on writing is. This aspect of educational technology—the fact that it forces us to examine our tacit knowledge of teaching and make it explicit—is one of the things that I value most about the field.

    Give it a read.

  • Blackboard Analytics Update

    In my last post, I promised that I would give an update specifically on the state of Blackboard’s learning analytics. Well, here you go. This is a summary of what I learned about their product from a chat with Mark Max, Blackboard’s VP of Learning Analytics and, to a lesser degree, with VP of User Experience Stephanie Weeks. I wrote about Blackboard’s Retention Center product some time ago. That product (or feature set, since it is free in Blackboard) directly competes with Desire2Learn’s Student Success System. This post is more broadly about their Analytics product suite, which is most directly analogous with Desire2Learn’s Insights product, although it is actually much, much broader in scope.

    The short version is this: Blackboard has very solid and reliable technology base from which they are building their learning analytics. It is easily the most mature platform among the LMS providers from that perspective. What they are a little short on is vision. In other words, they are pretty much the mirror image of Desire2Learn.

    (more…)

  • Getting students useful feedback from machine learning

    Last month, I wrote this narrow defense of automated essay grading, hoping to clear the air on a new and controversial technology. In that post’s prolific comments section, Laura Gibbs made a comment echoing what I’ve heard from every teacher I speak to.

    I am waiting for someone to show me a real example of this “useful supplement” provided by the computer that is responding to natural human language use – I understand what you want it to be, but I would contend that natural human language use is so complex (complex for a computer to apprehend) that trying to give writing mechanics feedback on spontaneously generated student writing will lead only to confusion for the students.

    When we talk about machine learning being used to automatically grade writing, most people don’t know what that looks like. Because they don’t know the technology, they make it up. As far as I can tell, this is based on a combination of decades-old technology like Microsoft Word’s green grammar squiggles, clever new applications like Apple’s Siri personal assistant, and downright fiction, like Tony Stark’s snarky talking suits. What you get from this cross is a weird and incompetent artificial intelligence pointing out commas and giving students high grades for hiding the word “defenestration” in an essay.

    My cofounder at LightSIDE Labs, David Adamson, taught in a high school for six years. If we were endeavoring to build something that was this unhelpful for teachers, he would have walked out a long time ago. In fact, though, David is a researcher in his own right. David’s Ph.D. research isn’t as focused on machine learning and algorithms as my own; instead, his work brings him into Pittsburgh public schools, talking with students and teachers, and putting technology where it can make a difference. In this post, rather than focus on essay evaluation and helping students with writing – which will be the subject of future posts – I’m going to explore the things he’s already doing in classrooms.

    (more…)

  • Six Ways the edX Announcement Gets Automated Essay Grading Wrong

    Six Ways the edX Announcement Gets Automated Essay Grading Wrong

    Last week, edX made a splashy spectacle of an announcement about automated essay grading, leaving educators fuming. Let’s rethink their claims. 

    “Give Professors a break,” the New York Times suggested in this joint press release from edX, Harvard, and MIT. The breathless story weaves a tale of robo-professors taking over the grading process, leaving professors free to kick back their feet and take a nap, and subsequently inviting universities, ever-focused on the bottom-line, to fire all the professors. If I had set out to write an article intentionally provoking fear, uncertainty, and doubt in the minds of teachers and writers, I don’t think I could have done any better than this piece.

    Anyone who’s seen their claims published in science journalism knows that the popular claims bear only the foggiest resemblance to academic results. It’s unclear to me whether the misunderstanding is due to edX intentionally overselling their product for publicity, or if something got lost in translation while writing the story. Whatever the cause, the story was cocksure and forceful about auto-scoring’s role in shaping the future of education.

    I was a participant in last year’s ASAP competition, which served as a benchmark for the industry; the primary result of this, aside from convincing me to found LightSIDE Labs, is that I get email; a lot of email. I’ve been told that automated essay grading is both the salvation of education and the downfall of modern society. Naturally, I have strong opinions about that, based both on my experience with developing the technology and participating in the contest, as well as in the conversations I’ve had since then.

    Before we resign ourselves to burning the AI researchers at the stake, let’s step back for a minute and think about what the technology actually does. Below, I’ve tried to correct the most common fallacies I’ve seen coming both from articles like the edX piece, as well as the incendiary commentary that it provokes. (more…)

  • A Taxonomy of Adaptive Analytics Strategies

    I almost never quote a blog post in its entirety, but this one from Dan Meyer is so good that I just can’t bear to cut a single word:

    Stephanie Simon, reporting for Reuters on inBloom and SXSWedu:

    Does Johnny have trouble converting decimals to fractions? The database will have recorded that – and may have recorded as well that he finds textbooks boring, adores animation and plays baseball after school. Personalized learning software can use that data to serve up a tailor-made math lesson, perhaps an animated game that uses baseball statistics to teach decimals.

    Three observations:

    One, it shouldn’t cost $100 million to figure out that Johnny thinks textbooks are boring.

    Two, nowhere in this scenario do we find out why Johnny struggles to convert decimals to fractions. A qualified teacher could resolve that issue in a few minutes with a conversation, a few exercises, and a follow-up assessment. The computer, meanwhile, has a red x where the row labeled “Johnny” intersects the column labeled “Converting Decimals to Fractions.” It struggles to capture conceptual nuance.

    Three, “adores” protests a little too much. “Adores” represents the hopes and dreams of the educational technology industry. The purveyors of math educational technology understand that Johnny hates their lecture videos, selected response questions, and behaviorist video games. They hope they can sprinkle some metadata across those experiences — ie. Johnny likes baseball; Johnny adores animation — and transform them.

    But our efforts at personalization in math education have led all of our students to the same buffet line. Every station features the same horrible gruel but at its final station you can select your preferred seasoning for that gruel. Paprika, cumin, whatever, it’s yours. It may be the same gruel for Johnny afterwards, but Johnnyadores paprika.

    Dan captures most of what I was trying to get at with my rant on the big data hype, but much more clearly and succinctly. Points two and three are the most salient here. First of all, the sort of surface-level analysis we can get from applying machine learning techniques to the current data we have from digital education system is insufficient to do some of the most important diagnostic work that real human teachers do. Think about the math classes is which you had to show your work on your homework. Why was that important? Because the teacher needs to see not only what you got wrong but why you got it wrong. Teachers generally don’t just say, “You got three out of five problems involving converting decimals to fractions wrong. Go study some more.” They sit down and work through the problems with the student to find the source of the errors. It’s really hard to get computers to do this well, even with highly procedural domains like math. (Forget about, say, literary analysis.) So in the vast majority of cases, we don’t even try to design systems where students show their work. And without the step-by-step data, no fancy algorithm is going to teach Johnny.

    Second, if the problem is that your content isn’t what the student needs, no fancy algorithm is going to fix that either. Videos are a prime example. I know of one textbook publisher whose teacher customers report that students won’t watch the publishers’ videos, but they can and do find videos on the same topic on YouTube and share them with each other. Think about that. Video-based pedagogical support is valuable enough to the students that they will expend energy searching for videos and sharing them. But they reject the expensive, carefully crafted videos from the publisher that are served up to them on a silver platter. It’s not that the publisher-supplied videos are necessarily “bad” in the sense that they have poor production qualities or are unclear or factually inaccurate. But the students have a particular use in mind for the videos. Maybe they’re struggling with a particular homework problem and just need a quick walk-through of a technique so that they can see the step that they are missing, for example. If the video doesn’t fit their needs—both utilitarian and aesthetic—then it won’t get used. Serving it up adaptively isn’t going to help that problem.

    That said, it’s worth taking a little time to break down the different types of adaptive learning analytics into a couple of categories and see just what we should and should not reasonably hope to gain from them.

    (more…)

  • What Is Machine Learning Good For?

    A few weeks ago, Audrey Watters wrote a great piece on her concerns about robo-grading of essays. (I tend to take a lot of inspiration from the things that annoy Audrey, in part because they usually annoy me too.) Here’s the crux of her argument:

    According to Steve Kolowich’s Inside Higher Ed story, [educational researcher Mark] Shermis “acknowledges that [Automated Essay Scoring] software has not yet been able to replicate human intuition when it comes to identifying creativity. But while fostering original, nuanced expression is a good goal for a creative writing instructor, many instructors might settle for an easier way to make sure their students know how to write direct, effective sentences and paragraphs. ‘If you go to a business school or an engineering school, they’re not looking for creative writers,’ Shermis says. ‘They’re looking for people who can communicate ideas. And that’s what the technology is best at’ evaluating.”

    Why are nuance and originality just the purview of the creative writing department? Why are those things seen here as indirect or ineffective? Why do we think creativity is opposed to communication?  Is writing then just regurgitation?

    What sorts of essays gain high marks among the SAT graders – human now or robot in the future? Are these the sorts of essays that students will be expected to write in college? Is this the sort of writing that a citizen/worker/grown-up will be expected to produce?  Or, for the sake of speed and cost effectiveness, in Vander Ark’s formulation, are we promoting one mode of writing for standardized assessments at the K–12 level, only to realize when students get to college and to the job market that, alas, they still don’t know how to write?

    How can we get students to write more? How can we help them find their voice and hone their craft? How do we create authentic writing assignments and experiences – ones that appeal to real issues and real discourse communities, not just to robot graders? How do we encourage students to find something to say and to write that something well?  Is that by telling them that their work will be assessed by an automaton?

    How do we support the instructors who have to read student papers and offer them thinking and writing guidance? When we talk about saving time and money here, whose bottom line are we really looking out for?  Who’s really interested in this robot grader technology?  And why? [Emphasis added.]

    This is a classic case of a market gone awry. Machine learning is sold as an “efficiency” tool, because there is money in squeezing cost out of education. In and of itself, there’s nothing wrong with wanting education to be cost-effective. David Wiley’s formulation of “standard deviations per dollar” has both a numerator and a denominator. You can attack either number and still affect the ratio. The problem with obsessing over the denominator is that you start forgetting that “cost-effective” has to be effective. If you want to know what the ongoing industrialization of education looks like in the post-industrial world, robo-grading is it. We are reducing the evaluation to the least common denominator, where the denomination is in dollars.

    But it doesn’t have to be that way. What if we looked at machine learning (the technology that makes robo-grading possible) from the perspective of trying to raise the numerator, i.e., effectiveness, while keeping cost the same? How could the technology be used as a force multiplier for good teachers, helping them to focus on what they do best in roughly the same way that flipping the classroom is supposed to do? If the goal is teaching better rather than just teaching cheaper, then what is machine learning good for?

    (more…)