e-Literate

Present is Prologue

Author: Michael Feldstein

  • There’s a Layer in Between Learning Content and Learning Analytics

    This post is the third in a series about the collaboration on a new courseware platform between Carnegie Mellon University’s Open Learning Initiative (OLI) and Arizona State University’s Center for Education Through eXploration (ETX). (The web version of this post has a clickable table of contents up at the top.) I’ll be writing about the platform periodically on a variety of topics ranging from nuts-and-bolts questions like what features it will have by when all the way to very broad and deep topics like architecture, interoperability standards, and hoped-for impacts on the educational ecosystem. As a reminder, I’ll also be holding a Blursday Social webinar with OLI’s Norman Bier and ETX’s Ariel Anbar on this Thursday, 2/4 at 4 PM ET.

    Today I want to review a fundamental aspect of the collaboration that I discussed in my previous post in this series from a slightly different angle. Not too long ago, I had a conversation about this angle with a new friend. After we had waded into it for a while, she said to me, “Everybody wants to talk about content and learning analytics. You’re saying that there’s a layer in between.”

    Exactly. She expressed it perfectly.

    And yet, two aspects about that moment surprised me.

    First, even though I have been writing about this idea almost since I started this blog in 2005 and have involved in the design of multiple EdTech systems that rely on this principle, I had never thought about it in quite the way she had put it before she said it. Second, I don’t think she had either. For context, my new friend is a PhD candidate in experimental psychology who is only weeks away from defending her dissertation at Carnegie Mellon University. She has a history in EdTech that is as long as mine. Furthermore, she had to be working with this concept directly and intimately, given the subject of her dissertation. And yet, her formulation of this idea also seemed novel to her as she expressed it to me in that moment.

    If I am still finding new ways to wrap my head around this concept, and she is too, then chances are good, Dear Reader, that previous blog posts I’ve written about the topic have failed to fully communicate the idea to you as well. So I’m going to take another go at it in this post. First I’m going to talk about what, generally speaking, we mean by “in between” and then about why it’s important for us to understand the layer that’s “in between” learning content and learning analytics.

    Meaning is the meat of the sandwich

    Imagine I handed you a piece of paper with “8429971043” written on it. Take a moment and try to imagine contexts in which that act would make sense to you. What might that number mean? Why would I be giving it to you? Sketch out a few stories in your head.

    Now instead, imagine I handed you a piece of paper with “(842) 997-1043” written on it. What stories are you writing in your head now? Did I give it to you printed on a little piece of cardboard at the beginning of a business meeting? Scrawled on a napkin in a bar?

    Now go back and reread those last two questions. Notice how much insight you were able to extract from the context embedded in the questions once you recognized that string of numbers as a phone number, as opposed to just a string of ten digits.

    Without the meaning of the number, you had the content, and you were trying to perform your analytics, but you were severely limited. The enabling layer in the middle is the idea of a phone number. Given just a little bit of punctuation, your brain transformed a random string of numbers into a thing in the world that you can do something with. It also enabled you to extract new information value out of other contextual information, like the kind of paper the number was written on or the location in which the paper was exchanged.

    There was a moment in the history of software when computing machines were given this layer in the middle for phone numbers. On one day, a phone number in an email wasn’t clickable (or tappable). The next day, it was. Clicking on it now offers you several predictable options. Would you like to associate this number with a person in your address book? Would you like to dial it now?

    Software developers deal with this “in between” layer all the time. They call it an “information model.” Consider, for example, the address book which stores the phone number that you clicked on. It has an information model that knows that phone numbers can belong to people. It knows that people can also have email addresses, physical addresses, company affiliations, and other things. Emphasis is on things. John Smith isn’t just a string of letters; it’s the name of a person. (842) 992-1043 isn’t just a string of numbers, spaces, and punctuation marks; it’s John Smith’s phone number.

    Information models represent meaning, including relationships between things (such as the relationship between John Smith and his phone number). Software architects work very hard to replicate the relevant ways in which we parse and store meaning in our heads to perform tasks that are relevant to the software being built. Suppose John Smith lives in Europe, where phone numbers have different numbers of digits. Suppose he writes his phone number as 842.992.1043. Suppose he has multiple phone numbers which he uses for different purposes in different locations. An information model has to capture all of these nuances. It’s hard work, but it’s a well understood process. Software developers do it all the time. They are constantly mapping bits of the human world into their software.

    For some reason, humans seem to struggle when thinking about how to map learning and teaching in this way. Even experienced and formally trained humans who think about it all the time struggle with it. We could chalk that up to how incredibly complex the human mind is. And there’s something to that. Learning is comprised of an incredibly complicated and varied array of processes. I’m going to write about that problem and its implications for, say, adaptive learning in a future post. But I don’t think that’s the whole story because even basic building blocks seem hard for us to wrap our heads around.

    Hard, but not impossible. Let’s give it a try.

    The pedagogical meaning in the middle

    Imagine you’re in a literature class studying The Odyssey. Your professor asks you the name of Odysseus’ dog. You may have one of two common reactions to this question. One is to try to come up with a thoughtful answer. The other is to roll your eyes in frustration. The reaction you have is related to the meaning you believe is attached to the name of Odysseus’ dog. If the name has no meaning—if it’s the Greek equivalent of “Spot” or “Rex”—then you might roll your eyes because you don’t think this question is has value relative to its purpose, which is to help you understand The Odyssey. The instructor’s question is a thing that is supposed to have a use. Specifically, it is supposed to help you learn—or show what you have already learned—about the literature. You expect it to fulfill one or both of those functions. You expect that the question, answer, and purpose for asking are all related to each other and to The Odyssey. This is the information model in your head that drives your interactions with your instructor. When your instructor asks you a question that doesn’t lead to an answer that helps you better understand the literature, then you get frustrated. It’s a little like being asked to memorize a phone number without knowing whose phone number it is or why you need to know it.

    When we are designing a lesson, either in software or in an old-fashioned, human-to-human class, we are (consciously or unconsciously) drawing on this information model.

    A basic information model for teaching and learning

    Let’s see how this works with some more interesting questions about Odysseus’ dog. Suppose your instructor asked you, “Why does the author write about the dog at all? What is the function of the dog in the story?” That feels like meaningful question. If you’ve read The Odyssey, you may remember that the dog only appears in one scene and dies almost immediately. It’s not like the dog (whose name is “Argos,” by the way) is a major character in the story. It shows up exactly once, at a moment in the story that is very dramatic for reasons that seem completely unrelated to the dog dying. What’s that about?

    Now let’s try a similar mental exercise to the one we tried with the phone number. Imagine your instructor asking that question in different contexts. What would it mean if she gave you the question to think about before you read the passage? How would the meaning of the question—or maybe more more precisely, how would the purpose of the question change if your instructor asked that question in class discussion after you had already read the passage? How would it function differently if the question came as an essay exam question after the class discussion?

    In context, we can answer these questions fairly easily. The instructor is asking the student a question about a work of literature that is intended to stimulate or test the student’s understanding of some specific aspect of the literature. We can judge the nuances regarding stimulating versus testing understanding based on the context. That is our tacit information model. And yet, both educators and software developers, and even experts who study learning, can struggle to hold onto the model in the general case.

    This is an important problem to solve if we want to improve education in general and EdTech in particular.

    Making tacit educational knowledge explicit

    As humans, we have an enormous amount of experience with learning through questions or challenges. Why is there a dog in that scene? How do I calculate how steep a hill is? Why are my scrambled eggs so runny? What is the guitar chord in that song I love? What will happen if I lick a frozen flag pole? We try to answer these questions. The emphasis is on try. We learn by doing (even if the thing we are doing is in our heads, like imagining what might happen if we lick that frozen flag pole). If we don’t get the answer we’re looking for the first time, we may try again or try something else. These are goal-directed learning feedback loops. As animals, we understand these loops instinctively. It’s an essential part of what makes us human. One translation of the sapiens part of homo sapiens is “one who knows.” This knack for knowing is how humans survive as a species. For example, I want to believe that most people who lick frozen flagpoles once don’t do it again. On the other hand, people who are serious about making good scrambled eggs may experiment repeatedly with pan heat, scrambling technique, and other elements. Each of these, in turn, produces a question that leads us to construct a goal-directed learning feedback loop. “What if the eggs are runny because I’m not getting them scrambled evenly enough?” And so we try again.

    Teaching is the craft of creating, optimizing, and sequencing goal-directed feedback loops that help learners learn. But because these loops are so intuitively available to us, we often don’t think explicitly about what we’re doing. Many of us don’t have a clear mental model for a fundamental task that we perform all the time. Even those of us who do can struggle to keep that model in our conscious minds and continue to refine it. It’s hard to refine your model if you can’t keep it front-of-mind as you’re testing it. And it’s really hard to translate it into a software information model.

    In my personal view, one of the most exciting goals of the OLI/ETX collaboration is the opportunity to work on this challenge of making a broad swathe of our tacit knowledge about educational processes explicit. On the software architecture side, the exercise of translating the very different learning experiences supported by OLI and Smart Sparrow into a common information model describing goal-directed learning feedback loops helps us generalize. I believe we are searching for the language of educational DNA.

    Let’s take that analogy seriously. In actual DNA, all of life in its infinite variety and complexity is encoded using just four letters in strands of DNA. I believe we can describe the basic, universal building blocks that encode learning in all its infinite variety and complexity. Of course, knowing those building blocks does not unlock all the secrets of life or learning by itself. But it’s a pretty big step in the right direction. An ambitious but achievable intermediate goal would be to be able to create and test any style of effective courseware design, including adaptive learning designs, using a single information design and architecture.

    The more important aspect of this aspiration is on the human side. If we can create a system that helps both educators and students translate the teaching and learning knowledge they apply instinctively into an explicit model, see the benefits that doing so gives them in terms of extracting more meaning out of educational contexts, and practice translating their tacit knowledge about teaching learning into explicit knowledge, that would be transformative. We would become better teachers and learners because we would be translating teaching and learning from arts to crafts and, when possible, to science.

    To sum up, I believe the software that is emerging from the collaboration between OLI and ETX can help educators and students become more effective at learning about teaching and learning while simultaneously creating an architecture that can implement almost any style of courseware on the market today and some that aren’t on the market yet. It can accomplish both of these goals by baking a model for the basic building block of teaching and learning—the goal-directed feedback loop—into every aspect of the platform, from the user experience to the deep architecture. This, in turn, will enable other capabilities that I will write about in future posts.

    If you want to talk about this (or any other aspect of the project), come to the Blursday Social webinar with OLI’s Norman Bier and ETX’s Ariel Anbar on this Thursday, 2/4 at 4 PM ET.

  • This Week’s Blursday Social: The CMU/ASU Courseware Platform Collaboration

    Happy not-2020 anymore! In celebration of the slightly-less-awfulness, we are restarting Blursday Socials with a bang. You may have been following my blog post series on the collaboration between Carnegie Mellon University’s OLI group and ASU’s ETX group on a next-generation courseware platform. (It’s actually a lot more than that, but that’s a good place to start in terms of unpacking what it really is.)

    Whenever a whizzy project or collaboration gets announced, the first question that everybody asks is, “Is this a thing?” As in, how real is this, really? And then the next question is, “To the degree that this is a thing, what kind of thing is it?” We will definitely answer the first question and get a start on the second one.

    Leading up this conversation with me will be two good friends. From the OLI side is Simon Initiative Executive Director, OLI Director, and connoisseur of fine bourbon Norman Bier.

    Norman at the 2019 EEP Summit

    From the ETX side, we have ETX Director, ASU President’s Professor, and explorer of habitable worlds Ariel Anbar.

    Ariel exploring a world we hope will remain habitable

    You’ve got questions? We’ve got (some) answers.

    This session is scheduled for this Blursday, 2/4 at 4 PM ET. Dress code is COVID-casual. This will be an informative social event, not a broadcast.

    BYOB.

  • A Courseware Platform for Expressing Pedagogical Intent

    A Courseware Platform for Expressing Pedagogical Intent

    In a recent post, I announced a collaboration between Carnegie Mellon University’s (CMU) Open Learning Initiative (OLI) group and Arizona State University’s Center for Education Through eXploration (ETX) to build a next-generation, open-source, open-standards project, which has been facilitated by the Empirical Educator Project (EEP). As I noted in that post, because each group has intra- and extra-university diplomacy and policy responsibilities, I have pre-approved the previous post and this one with ETX and OLI, respectively. With these two posts in place, I will explore some implications in subsequent posts without worrying about coloring outside the lines.

    The first post in this series described the nature of the collaboration. This one will describe the nature of the project itself.

    A recap of how we got here

    Here’s the sequence of events that led to the current project:

    First, about a year and a half ago, CMU announced at the EEP summit that they were releasing an open-source collection of over $100 million’s worth of learning engineering software they had developed in a collection they call the OpenSimon Initiative. OpenSimon initially included parts of the OLI’s platform—also called OLI—which is both the original template for most current-generation courseware platforms and a platform designed from the start to support research into applied learning science. (The reasons why they could not initially release all of OLI are unimportant for this story. The resolution of the problem, which I’ll get to momentarily, is more salient.) Norman Bier, who is both the Executive Director of the Simon Initiative and the Director of OLI, spearheaded OpenSimon’s release and has led OLI’s participation in the collaborations to come.

    Separately, in the spring of 2019, ETX Director Professor Ariel Anbar decided to fund a courseware interoperability standards effort as part of an Open Educational Resources (OER) development grant the US Department of Education had awarded to the Center (a project which has since been branded OpenSkill). Since ETX planned to develop their OER courses on the proprietary Smart Sparrow platform, Ariel believed that those courses would need to be portable to other platforms to satisfy the spirit of OER. So he hired EEP to help convene a group and facilitate the work. One of the participants that I recommended was OLI. We had a kick-off convening in September of 2019.

    Shortly after that convening, Pearson announced that it had acquired the SmartSparrow platform and would be discontinuing support of the product in its current form. (As I noted in my previous post, I have nothing to say about Pearson’s actions other than that ETX stakeholders have told me that the company has worked with the university to make the transition as smooth as possible.) Unfortunately, since we do not yet have courseware interoperability standards, Pearson’s ability to make the transition smooth is limited. There is no easy way to get the functioning courseware out of Smart Sparrow and import it intact into some other one.

    Rather than letting this substantial transition derail the interoperability work—ETX has over $15 million’s worth of grant–funded courseware that they need to migrate off of Smart Sparrow by a fixed deadline—Ariel decided his predicament proved that interoperability is too important to ignore. So he began looking for partners who would be willing to invest in this goal.

    During roughly this same time period, OLI began developing the next generation of their platform, code-named “Torus,” which could be released as open-source. (The code repository for the project is available on Github today.) After some discussion and exploration, ETX and OLI agreed to be primary partners on this project, developing a specification that would enable ETX’s Smart Sparrow-based experiences to run intact on Torus. They also agreed to design the architecture to use existing interoperability standards whenever possible, propose enhancements to those standards when it is helpful, and to invent non-standard approaches as needed. Torus will be a reference implementation for proposed courseware interoperability standards. It will demonstrate the viability of the proposed standards by showing that it can import and run ETX’s existing Smart Sparrow intact and by enabling ETX to continue authoring Smart Sparrow-style courseware in the future. ETX has engaged EEP to facilitate this collaboration.

    A rare opportunity

    In this project, we have two different applied academic centers that both build courseware for use at scale. They focus on substantially different (though complementary) research-based course design approaches. Each has chosen a platform because of its suitability for their respective approaches. Both perform efficacy research on the courseware they develop. By bringing them together in this way, we have a rare opportunity to intentionally drive the evolution of the entire product category, starting with a consideration of research-supported pedagogy and driving it deep into the fundamental structure of how courseware platforms are built by influencing their interoperability standards. Technology is the means, not the end. The real goal is to develop a generalized approach for courseware platform architecture that can implement a range of evidence-based design approaches and drive continuous improvement of digital curricular materials that demonstrably support student success and improve equity.

    For this reason, I will not be writing about the technical details in this post. Instead, I will focus on foundational pedagogical and functional concepts that drive the design decisions. Readers who are specifically interested in technical interoperability or architectural questions will have to read between the lines. I promise to cover these topics more explicitly in later posts.

    Because we have been building toward this public collaboration for quite some time, I have been laying a breadcrumb trail regarding this project since long before I could talk about it explicitly. If you want to really study this project, I recommend reading the other posts that I reference. The links to previous e-Literate posts embedded in this one are more important than usual.

    First principles

    Let’s start by laying out some basic pedagogical principles that both organizations endorse and that infuse all aspects of plans for this project, from the interoperability standards to the software architecture to the user experience.

    First, “to learn” is an active verb. Students cannot “get learned” or “be learned.” They must do the learning themselves.

    Learning results from what the student does and thinks, and only from what the student does and thinks. The teacher can advance learning only by influencing the student to learn.

    Herb Simon

    “Influencing the student to learn” can be interpreted as provoking students to think and do things that you hope will stimulate them to learn. People learn when they try something or think about something and come to a conclusion or insight based on that process of exploration. Part of the craft of teaching is knowing what to give students to think about or try.

    The actual learning happens inside a student’s head, which means that it is not directly observable. So another part of teaching is giving students tasks to perform that provide observable clues as to what they may or may not have learned. We call this teaching task “assessment.” Importantly, self-taught learners assess themselves. For example, if I am learning how to make scrambled eggs, I will try to scramble and egg and observe how it comes out. I won’t know if I learned how to make scrambled eggs until I try to scramble one. Likewise, I won’t know if I’ve failed to learn how to scramble an egg. If I assess myself, then I can keep trying things and observing the outcomes until I learn whatever it was that I missed the first time (like, for example, how to tell if the pan is the correct temperature for scrambling an egg). Self-taught learners are, in part, self-assessing learners. They construct tests for themselves to figure out what they haven’t gotten right yet. Educators assist learners with that process.

    Regardless of whether I have a cooking coach or if I am teaching myself, I may very well decide what to try next based on the results of my assessment. If my eggs come out poorly, then I will think about what I might have done wrong and try something else. If they came out well, then I might try incorporating them into a breakfast burrito. Or I might move on to learning how to make bacon.

    In other words, learning is a feedback loop, where the input is the learning experience and the output is the assessment result. Learning may happen in between the input and the output, inside the black box that is the learner’s head. It may also happen when the learner sees the output, which might become the input for another learning feedback loop. A failed attempt to scramble an egg may cause me to do or think something that enables me to learn how to successfully scramble an egg.

    It works something like this:

    1. Input: I read a recipe for scrambled eggs.
    2. [Weird things happen in the dark place between my ears.]
    3. Output: I try to scramble an egg.
    4. Inference: I judge whether I am ready to move onto making a breakfast burrito or if I have not yet achieved egg-scrambling competency.

    (It’s not exactly this clear cut. For example, I learn things about scrambling eggs in the process of scrambling an egg.)

    As part of that last step, I’m evaluating whether I’ve learned what I set out to learn. Hence, the feedback loop. Based on that inference I make from the assessment results, I decide whether to repeat the learning feedback loop, adjust it, or move on to a new one. But again, a critical point here is that Step 2—the actual learning—is invisible. The brain is a black box. We only know what goes in and what comes out.

    A widely practiced discipline in learning design, known as “backward design,” is built on these fundamental insights. At its simplest, backward design has three steps:

    1. Specify the desired learning goal or competency (e.g., scrambling an egg)
    2. Design the assessment that will test progress toward the competency (e.g., trying to scramble an egg)
    3. Design the experience that will stimulate the learner to do or think something that will hopefully enable them to learn (e.g., reading a recipe)

    Steps two and three in backward design correspond to the output and input of the learning process. While step one was implicit in my previous example—I started off wanting to learn how to scramble an egg—it must be explicit when designing a curriculum using this method.

    The backward design learning feedback loop. (Credit: Carnegie Mellon University)

    In fact, the curriculum can be thought of as the scope and sequence of these specified learning goals. The scope refers to the full range of competencies that I need to master to reach a larger goal. Is my ultimate aim to learn how to make a bacon-and-egg breakfast or a breakfast burrito? The sequence is the order. I can’t learn how to make a breakfast burrito until I first learn how to scramble an egg. In other words, we can think of the process of learning design—and teaching, and learning—as constructing a series of nested learning feedback loops. Many educators (and learners) construct these loops instinctively and unconsciously, which means that they sometimes miss opportunities to make their designs more effective. But whether consciously or not, all learners, all educators, and all designers of instruction construct sequences of feedback loops. Because without learning feedback loops, there is no learning.

    Computers are (mostly) dumber than humans

    So humans can’t learn, and therefore can’t teach, without feedback loops. They can’t tell if learning is happening, never mind provoking thought that leads to learning. Guess what? Computers can’t either. And no amount of data or fancy AI algorithms will change that, because both learning and teaching happen via feedback loops. I’ve seen (and written about) a lot of magical thinking in EdTech that talks about data as if it’s fairy dust: one speck is pretty much the same as any other, but the more you have of it, the more magic you get.

    I wrote a blog post last December called Pedagogical Intent and Designing for Inquiry which was all about this misconception and how it harms our efforts to build effective EdTech. “Pedagogical intent” should be interpreted as the reasoning behind the design of the learning feedback loop. For example, the software might detect patterns in students’ answers that suggest a particular skill is not being taught well (or at all). Or it might identify a prerequisite skill or bit of knowledge a student missed earlier that is causing her to struggle now.

    To find these insights, the computer needs more information about the activities than just the clicks. It needs information about pedagogical intent. Why is the educator assigning the student a particular activity to perform? What was a particular assessment answer choice intended to suggest about the student’s internal mental processes? Without knowing the answers to these questions, the computers are as blind as we are. Here’s an example from that post:

     ACT recently released a paper on detecting non-cognitive education-relevant factors like grit and curiosity through LMS activity data logs. This is a really interesting study that I hope to write more about in a separate post in the near future, but for now, I want to focus on how labor-intensive it was to conduct. First author John Whitmer, formerly of Blackboard, is one of the people in the learning analytics community who I turn to first when I need an expert to help me understand the nuances. He’s top-drawer, and he’s particularly good at squeezing blood from a stone in terms of drawing credible and useful insights from LMS data.

    Here’s what he and his colleagues had to do in order to draw blood from this particular stone…

    First, they had to look at the syllabi. With human eyeballs. Then they had to interview the instructors. You know, humans having conversations with other humans. Then the humans—who had interviewed the other humans in order to annotate the syllabi that they looked at with their human eyeballs—labeled the items being accessed in the LMS with metadata that encoded the pedagogical intent of the instructors. Only after they did all that human work of understanding and encoding pedagogical intent could they usefully apply machine learning algorithms to identify patterns of intent-relevant behavior by the students.

    LMSs are often promoted as being “pedagogically neutral.” (And no, I don’t believe that Moodle is any different.) Another way of putting this is that they do not encode pedagogical intent. This means it is devilishly hard to get pedagogically meaningful learning analytics data out of them without additional encoding work of one kind or another.

    Pedagogical Intent and Designing for Inquiry

    So far, machine learning has not proven to be generally effective at deducing pedagogical intent based on low-level data like student clicks or time they spend watching a video. It’s getting pretty good at deducing pedagogical intent from data when it is given inputs that are already well-structured to supply hints of that intent. LMS activity logs are not well-structured in this way. Ironically, analog textbook chapters are because they are designed to give humans clues about these things using only text organization, pictures, font styles, and the like. But that doesn’t help us to reliably deduce pedagogical intent in the realm of the digital. There ought to be a better way.

    Luckily, there is one.

    Learning feedback loops in courseware design

    It turns out that virtually every commercial courseware product on the market today, and virtually every course created with substantial input from a professionally trained learning designer, has been built using backward design. Furthermore, most interesting and useful learning analytics in commercial courseware products today get virtually all of their utility from this backward design and very little, if any, from some super-genius artificial intelligence algorithm. Machine learning is most often useful when it can “read” the backward learning design to work its magic. Today’s learning analytics and adaptive learning systems are best thought of as algorithms that attempt to squeeze as much insight and value as possible from the pedagogical intent that has been encoded into the courseware platform in a machine-readable form. They may use other data too. But as the ACT study above illustrates, those other data have limited value when they are disconnected from the context of the learning feedback loop designs and the pedagogical intent behind them.

    Last September, I wrote a three-part post series on exactly this topic called “Content as Infrastructure” (the title of which was borrowed from David Wiley). You can probably skip the first post since it covers a lot of the same ground you’ve just read. The second post shows examples of different products that take advantage of backward design to create better educational products, while the third one digs into the various types of insights that can be gained from computers when they have access to semantic information about pedagogical intent. If you want to dig a little deeper, you can go back to my post from 2012 about how CMU’s learning engineering techniques use learning feedback loops to identify hidden prerequisites that the educators may not have realized they need to address (like the recipe writer who doesn’t realize that not all aspiring egg scramblers know how to tell if the pan is the right temperature). Or you can read my post from 2013 about how ETX’s Habitable Worlds course creates experiences that provoke student learning.

    Generalizing feedback loops

    As promised, I will keep this post non-technical. But given the pedagogical principles above, we can begin to think through the basic requirements for a universal technology design.

    First, we need standard ways to describe the smallest and simplest learning feedback loops, like a single multiple-choice question. The student reads the question, chooses an answer, and receives feedback. Simple. And yet, not. A well-written multiple-choice question will have wrong answers—or “distractors,” as they are often (problematically) called—that are intended to diagnose specific common misunderstandings. Students might be given multiple tries on the question, thus traversing the feedback loop multiple times. They might get feedback based on their answer which is intended to help them learn or reinforce their learning. They might be given the option to ask for a hint. Or multiple hints. And each successive hint may be designed to get at a different possible blocker to the student’s learning the correct answer. A single well-design multiple-choice question implemented in a system that gives options for multiple tries, feedback, and hints can be absolutely loaded with pedagogical intent. We want a system that captures as much of that as possible and as automatically as possible. The more that the author’s pedagogical intent can be captured as part of the natural authoring process—rather than requiring the author to add that information afterward—the better.

    Second, we need a system that enables learning feedback loops to be connected to each other in different ways, and to create loops within loops. On a fundamental level, if I want to make a bacon-and-egg breakfast, I have to learn how to scramble an egg. If I want to learn how to scramble an egg, I have to learn how to tell if the pan is the right temperature. If I want to learn how to become a short-order cook in a diner, then I need to learn all these skills plus many more.

    We also need the feedback loops to be connectable in many ways. A common approach in OLI courses is to give students a string of activities that address the same “learning objective”—a label we attach to a learning goal of a certain granularity that is meaningful in the context of the larger course—and watch to see if the students’ error rates on those problems go down. This requires the ability to tie those problems together in a sequence. A common approach in ETX courses is to create “guided experiential learning” exercises, in which students learn by exploring. This requires the ability to branch students to different learning feedback loops based on their answer to the previous one (even if the branch is just a short detour that leads back to the main path). Both approaches are valid and not mutually exclusive. They can be used in the same course. Sometimes they can even be incorporated into the same exercise.

    When we talk about interoperability—which was where this project started—we can mean at least two very different and equally important goals. The first is to be able to export a learning feedback loop design of any grain size—a multiple-choice question, a quiz, a loop that addresses a skill, or a learning objective, or a competency, or a lesson, a course, a certificate program, etc. (You might be surprised at the amount of thought that goes into designing a single multiple-choice question to create a truly diagnostic learning feedback loop.)

    The generic word we use for a compound learning loop of any size is an “ensemble.” I’ve seen a lot of debate and hand-wringing over the years about what a “learning object” is, how big or small one can be, and so on. The problem with “object” is that it’s a fancy name for “thing.” Of course very different kinds of learning things should be very different sizes. In this project, we talk about learning feedback loops, which also can be very different sizes. But they are all the same basic type of thing, in the sense that they have learning inputs and produce learner outputs which are diagnostic of particular learning goals. Any learning thing—whether it is a picture or a course unit or something else—which does not have these characteristics is not a learning feedback loop.

    We should be able to move the content, the execution logic of the learning loops, and the encoding of the pedagogical intent, intact, from one system to another. We also should be able to analyze student activities in the context of the pedagogical intent of the learning loop the same way for any courseware in any system. We should be able to know what the learning goal is, what the student’s assessment answer suggests about progress toward that goal, and so on. The interoperability project aspires to achieve both of these goals, although they will take some time to refine and to take them through an official standards-making process.

    And we want to create a system that encourages educators to author learning feedback loops that are well-crafted and continuously improve that content based on feedback from the system on how well it is helping the students. Most educators—particularly ones who have no formal training in education, like the vast majority of college professors—do not naturally think about what they are doing as crafting series of learning feedback loops. Even when they understand the basic idea, they may not have all the skills that could help them (just as understanding the basic concept of scrambling an egg does not automatically convey the skill of making yummy scrambled eggs). For example, a learning designer recently told me that some of the professors she works with don’t know that they should try to write multiple-choice distractors designed to diagnose specific misconceptions. In an ideal world, a courseware platform would have an authoring interface, analytics, and nudges that all encourage courseware designers—whether they are first-time amateurs or skilled professionals—to constantly be working to test and refine their learning feedback loops.

    This thing is real

    These are obviously big ambitions that will take a long time to realize. Some of them will take at least few years to get right or to get through formal standards-making processes. That said, this is more than just another grand idea that will probably never happen.

    It is happening right now.

    Here are some milestones that the collaboration has achieved so far:

    • Torus exists. The source code is available. Some simple courses are in production on a stable version of it.
    • OLI had several teams build sample lessons on Torus during a virtual summer school program.
    • ETX, with the help of Unicon’s architects and developers, has exported the lessons from Smart Sparrow, including the logic of the learning loop design.
    • Unicon has built several iterations of a harness that can run a wide range of ETX Smart Sparrow with increasing fidelity. To be clear, they are not rebuilding these lessons. They are running the exported lessons, with the full learning experience, intact.
    • The collaboration team has demonstrated the ability to launch the ETX lessons from OLI.
    • IMS Global, one of the major technical interoperability standards bodies for education, has been kept apprised of the work and has been actively supportive of it.
    • Unicon’s latest iteration of the Smart Sparrow test harness runs the full representative sample of ETX lessons with production-quality fidelity. (A few minor tweaks remain to be completed.)

    Here are some planned milestones:

    • The collaboration team is in the process of generalizing the runtime commands for the ETX learning loops into a specification that can be implemented in Torus and submitted as a formal interoperability specification candidate.
    • In parallel, ETX and Unicon are working on a developer’s aid that will enable ETX experts to begin creating new ETX-style lessons in the first quarter of 2021. While it will not be suitable for general use, it will help us continue to refine the specification and think about broader authoring requirements.
    • By the fall of 2021, ETX expects to migrate all of its Sparrow courses onto Torus and OLI expects to put a substantial subset of its courses into production on Torus.
    • Work on enriching the authoring interface will proceed in parallel, with the goal of releasing an authoring solution that fully supports multiple styles of course design by January of 2022.
    • OLI will complete its migration of all its courses to Torus by the fall of 2022. Given the current size and growth of the two projects, I expect the combined annual enrollments on the platform to be well north of 150,000 – 200,000 at a minimum at that time.

    The progress and productivity of this collaboration has been amazing. I’ve never seen anything like it in my entire career. And the architectural innovations are quite clever. I can’t wait to start writing about them.

    But the main take-away for now should be that this project is real, it is already producing tangible results, and it will be out in the world in a meaningful way within a year.

    Some of you have already asked me if you can observe or even participate in this project. We are thinking about ways to do that and will have some answers for you in the new year. There will likely be several options, depending on the nature and seriousness of your interest. Opportunities for observation will likely come before we can open up for participation, particularly given the aggressive deadlines that both ETX and OLI have to meet. But this is a public good project with a core commitment to openness. If you are interested in getting involved, please contact me. I’m gathering information about interest so that we can begin planning a community strategy.

    Stay tuned for more

    This post barely scratches the surface of the ambition. In future posts, I will be digging deeper into the details. In the meantime, I suggest again that you (re)read the second and third installments of my Content as Infrastructure post series to get a little more grounding in how much work the basic structures I’ve described here can accomplish and my post Pedagogical Intent and Designing for Inquiry regarding why capturing pedagogical intent is so important for gaining insights on teaching and learning out of these systems.

  • Next Blursday Social (12/10): Filling EdTech Potholes with ReThink Education’s Matt Greenfield

    Welp, it’s been a year, hasn’t it? Everybody, including the EdTech-averse, has had to go all-in on EdTech all at once. And it’s exposed some potholes. Some we knew were there, others we learned to live with and forgot about, and still others we didn’t see coming. Now is a good time to think about lessons learned as we look toward future changes, both foreseen and unforeseen, both fast and slow.

    So we’re going to chat with venture capitalist and friend of e-Literate Matt Greenfield, Managing Partner at the ReThink Education venture capital firm. In a prior life, Matt was an English professor at CUNY, Columbia, and Bowdin. So he’s seen the world from both sides of the EdTech company/academic line. We’re all going to chat with Matt about the potholes that need filling.

    Matt Greenfield, Managing Partner at ReThink Education

    This will be the last Blursday Social of 2020, so don’t miss the thrilling season finale!

    Remember, Blursday Socials are much more conversation and participation than broadcast. Think of it as being like trivia night at your favorite bar. There will be some organized activity, but a lot of it will be just connecting with friends. Some folks will be regulars, others will be semi-regulars, and still others will drop in occasionally. They start every Thursday at 4 PM Eastern Time. The bar closes at 5:30.

    The dress code is casual; BYOB.

    I recommend that you also subscribe to the iCal feed, which you can do by going to this page and clicking on the “iCal Feed” button on the bottom. I have events tentatively lined up for the handful of weeks, which I will populate to the calendar as they are finalized.

    RSVP here: https://www.runtheworld.today/app/invitation/13547

  • A Next-Generation Open Source Courseware Platform Collaboration

    I started the Empirical Educator Project (EEP) two and half years ago based on the idea that higher education could greatly accelerate progress in technology-enabled education if only they could get better at sharing their projects and collaborating on addressing their common or complementary needs. Since then, I’ve been able to write about increased sharing facilitated through the EEP network but not much about collaborations.

    Today I have some massive news to share on that front. Carnegie Mellon University’s Open Learning Initiative (OLI) and Arizona State University’s Center for Education Through eXploration (ETX) are collaborating on a next-generation, open-source, open-standards platform for designing and delivering courseware.

    That’s a lot of words, yet it doesn’t really convey the heart of the work. There is much to unpack here.

    In this first post of a multi-post series, I will describe the drivers and origins of the collaboration itself. The project is an evolution of contributions from these two groups that I have previously written about here in e-Literate. In the next post, I will describe some of the key architectural concepts of the project. And in subsequent posts, I will explore some of the implications.

    On the nature of academic collaborations

    Before I get into the details of the project, it’s worth taking a moment to talk through some lessons learned about fostering EEP-style collaborations. First, they take time and energy to bring to fruition. While EEP generated a lot of excitement from the beginning simply by helping academic groups discover colleagues in different organizations who were doing complementary work, helping these initial connections to evolve from casual cross-fertilization to deliberate and sustained collaboration takes a lot of time and hard work on everyone’s part. This is not a surprise. But I am learning a lot about the specific kinds of facilitation necessary to bring collaborative projects to fruition. (This topic probably deserves its own blog post at some point.)

    Second, one of the complexities that need to be considered is the internal diplomacy that each stakeholder has to manage. Note that the partners I have listed here are ETX and OLI, not ASU and Carnegie Mellon. Likewise, the title of this blog post is not the much sexier “ASU and Carnegie Mellon collaborate on…”. Both institutions are rightfully protective of their respective brands and how their involvement is characterized in various projects. And that’s just the tip of the iceberg. Each participant has to manage a whole host of intra-institutional concerns that affect when and how they talk about such collaborations publicly.

    The collaboration I will be writing about in this series has been underway for some time and is quite advanced at this point. Yet I could not write about it before now without complicating important conversations that were (and still are) proceeding inside these institutions. Even now, this blog post is not an official press release and should not be considered an official statement from either university. I have asked the major project leaders at each institution to review my first two blog posts in the series to help me characterize the project appropriately. That said, if and when you see a press release jointly issued from the two institutions, you should consider that more official statement to be another milestone in the evolving relationship.

    With those caveats out of the way, I’m going to walk through the process by which contributions led to a ground-breaking collaboration. In the process, I’ll provide a first sketch of exactly what that collaboration is.

    Carnegie Mellon’s OpenSimon contribution

    In the spring of 2019, Carnegie Mellon University used the EEP summit that they hosted to announce OpenSimon, a collection of open-source projects built over the years by their faculty to advance their research and application of applied learning science—or, as they call it, “learning engineering.” The investment these projects collectively represented was well north of $100 million in grant money, staff time, etc. This was an official, university-sanctioned contribution. They issued a press release. University Provost Jim Garrett kicked off the announcement at the summit.

    People at the EEP summit and the in wider EdTech world were simultaneously excited and a little bewildered. The toolkit provides a sprawling array of functionality in a variety of areas. Some of its components tilt more toward research, while others are immediately useful in teaching applications. It was hard to know where to look for the low-hanging fruit. Even for me.

    One of the crown jewels in OpenSimon is the OLI software, which is developed and maintained by the eponymous OLI team. Originally conceived almost 20 years ago, the OLI platform set the template for modern mainstream courseware while the research on the courses built on that platform established some of the earliest and most robust evidence we have for the effectiveness of this kind of teaching tool. Without OLI, there probably would be no Pearson Revel, no Cengage MindTap, no McGraw-Hill Education Connect or SmartBooks, Wiley+ or Knewton, and so on. Today, OLI hosts somewhere in the range of 50 Carnegie Mellon-designed courses for over 50,000 formal course enrollments annually, not including courses and lessons created by faculty at other institutions or the many students who use OLI courses for self-study outside of a formal course adoption.

    My coverage of OpenSimon to-date has left out one critical update. OLI is in the process of rebuilding the platform, incorporating lessons learned from the past two decades in how to design such a system and reimplementing the platform on a modern architecture that takes advantage of the cloud services and technology stacks available in 2020. The new platform, code-named “Torus,” has already been used during the university’s 2020 summer school program in which students practiced building simple course modules. OLI plans to migrate the first tranche of their courses in production onto the new platform by the fall of 2021 and the remainder by the fall of 2022. In other words, within 18 months, this next-generation platform will be hosting all of the 100,000+ student enrollments in production.

    As with the other OpenSimon projects, Torus is being released as an open-source project under the MIT license.

    ETX’s courseware interoperability initiative

    About a month after the OpenSimon announcement, I was approached by Ariel Anbar, the ASU Presidential Professor and Director of ETX. I have been a long-time admirer of Ariel and his ETX colleagues, having written about their Habitable Worlds course as far back as 2013. From early on, ETX used the Smart Sparrow courseware platform to create their immersive digital lessons and courses in a pedagogical style that they call “guided experiential learning.” When I ran into Ariel at the 2019 ASU+GSV conference, he had recently received a $2.5 million grant from the Department of Education to build OER courseware that follows this pedagogical style on Smart Sparrow. But he foresaw a problem. While his content would be released under a Creative Commons license, the platform required to run it was proprietary. To his mind, this violated the spirit of OER.

    Furthermore, having worked very closely with Smart Sparrow to build over $15 million in grant-funded content already (from funders including, NASA, NSF, and the Bill and Melinda Gates Foundation), Ariel recognized that all such platforms were hurt more than they were helped by that lock-in. Without a wealth of content that could be “played” on a proprietary platform, the companies that make the platforms will always be financially at-risk and therefore risky to use. If the stand-alone courseware platform product category is ever to become sustainable, it needs an ecosystem of interoperable content.

    Ariel hired me, under the auspices of EEP, to convene a coalition to draft interoperability standards for courseware content that could be submitted to a standards body as a first draft of an official specification. In doing so, ETX was offering to make an EEP-style contribution. They were extending their project in ways that advance the public good by enabling other educators and educator-supporting companies to benefit and contribute.

    As I wrote about in a blog post late last year, our first meeting included OLI, Lumen Learning, Smart Sparrow, and CogBooks as potential platform implementors, with ETX acting as the convener and some allies at Maricopa Community Colleges who were involved with the grant providing an access-oriented academic perspective. ((At the time, Lumen Learning and Smart Sparrow were both EEP sponsors, and Carnegie Mellon was our 2019 academic sponsor.)) The group met in September 2019 to workshop the scope and basic structure of several kinds of interoperability standards. IMS Global CEO Rob Abel offered his blessing to the project and some free consultation hours from one of his engineers. The group had the outline of an action plan shortly after our first meeting and was off to a promising start.

    Fate intervenes

    Shortly after the interoperability kick-off meeting, I learned that Pearson had acquired the assets of Smart Sparrow. ((Disclosure: Pearson is a current EEP sponsor.)) They wanted Sparrow’s next-generation version of the platform for internal projects and did not intend to continue supporting the product for existing customers. It’s not my place to characterize the relationship between Pearson and Sparrow’s customers regarding this transition, other than to make the observation that it seems to be proceeding in a supportive manner that has satisfied the Sparrow customers I have spoken with. But the most important point for the ETX-led interoperability project is that the abstract concern it was intended to address suddenly became concrete. ETX had over $15 million of content development efforts invested in Smart Sparrow courseware and plans for over $10 million more. They were already supporting somewhere in the neighborhood of 50,000 student enrollments a year (many of which came through their Inspark network). They needed a way to move their content to a new platform that would preserve the pedagogical intent and learning design. It wouldn’t be enough to move the assets—the text, videos, quiz questions, and so on. Those assets were assembled in particular ways to create specific and differentiated learning experiences for students. ETX needed a way to preserve their past and future investments in guided experiential learning.

    We needed to accelerate our timeline by not only designing but also implementing interoperability with at least one of the partners in the collaboration. Of these, OLI turned out to be the best fit in terms of timeline and development ambitions. Both groups needed to implement new platforms in roughly the same time frame. (ETX needs to migrate its content off of Sparrow by June of 2022, although the goal to finish the migration sooner.) Both groups were highly collaboration-minded and open to cross-pollination. At first blush, the fit seemed good.

    (CogBooks has also been involved in the interoperability discussions, although I’m not at liberty to characterize their past or prospective involvement at the moment other than to give them credit for coming to the table and investing effort into early interoperability prototypes.)

    But there were—and still are—challenges to be faced too. First, the alignment of the two groups’ timelines is a double-edged sword. On the one hand, it is good to have collaboration partners with well-defined requirements and similar delivery schedules. On the other, neither group could afford to allow the time cost of collaboration to throw off their respective delivery schedules. Both ETX and OLI have responsibilities to internal and external stakeholders that they must meet. Further, Sparrow deviates from the course design archetype that OLI created more substantially than many other platforms in the market. The architecture of the new platform would have to accommodate divergent design approaches. And the two teams did not know each other or their respective work very well.

    The fit looked promising. But the project would face substantial challenges.

    The collaboration comes together

    ETX engaged Unicon—another EEP sponsor—to help with the software development on their side. The first step was to prototype a harness for running ETX lessons outside of Sparrow in a way that would preserve the integrity of the learning design and could be launched from courseware platforms using a variant of the IMS LTI interoperability standard. Unicon has completed several increasingly sophisticated versions of the harness. ETX also continued to engage me—again, under the auspices of EEP—to help facilitate the growing collaboration. ETX and OLI respectively invested significant amounts of internal staff time to articulate requirements, think through architectural designs, align road maps, and get to know each other. Needless to say, this took time and good will on all sides.

    The collaborators have reached a level of mutual confidence that the project has been greenlit. They plan to expand OLI Torus to accommodate the authoring and delivery of Smart Sparrow-style—and, more importantly, ETX-style—courses. Both organizations expect to support substantial numbers of students and faculty in courses migrated from their respective legacy platforms by the fall of 2021, with ongoing work continuing into 2022. Both organizations are making substantial engineering investments in the project; OLI through its internal development team and ETX through its contract with Unicon (as well as ETX-internal project management and learning design subject matter expertise). They are collaborating on project management, and both have their (respectively formidable) learning design/learning engineering experts involved with the design. ETX also continues to contract EEP to facilitate the collaboration and help keep the two universities’ product plans and goals aligned and as complementary as possible. Torus will be released under a commercial-friendly open source license. Further, it will act as a reference implementation for draft courseware interoperability specifications. (Again, these last two sentences may come across as bags of words to you, but they are vital to the project’s ambitions. I will unpack them in subsequent blog posts in this series.)

    This is an incredibly ambitious project with many miles to go (and many critical decisions to make) before it is complete. But to be clear, it is happening. Resources have been committed. Development is well underway. Several interoperability prototypes have been built according to increasingly detailed specifications. And the innovations that are resulting from the collaboration are considerable. The current generation of courseware platforms is a collection of highly idiosyncratic applications that were developed in relative isolation, are generally fairly old, and are often maintained by organizations with challenging or problematic business models. This project is gaining insights by comparing two very different and very innovative platforms in a collaboration between two very different but very accomplished academic course design teams with respective histories of supporting faculty and students at a reasonably large scale across a diverse range of learning contexts.

    It is thrilling.

    In my next post in this series, I will describe several of the architectural and design innovations that have emerged from the collaboration so far.

  • Next Blursday Social (12/3): Rethinking Virtual Conferences with EDUCAUSE ELI

    Next Blursday Social (12/3): Rethinking Virtual Conferences with EDUCAUSE ELI

    After a brief hiatus for Turkey and such, Blursday Social with e-Literate LIVE! will back next Blursday, 12/3 from 4 PM to 5:30 PM ET. This week we’ll be talking with some good friends at EDUCAUSE about our experiences, joys, pains, hopes, and dreams regarding virtual conferences to help them think about the upcoming virtual EDUCAUSE ELI conference. Come chat with Kathe Pelletier, Director of the EDUCAUSE Teaching and Learning Program, and Beth Kroll, EDUCAUSE’s Director of Conferences and Events.

    Kathe Pelletier and Beth Kroll

    Remember, Blursday Socials are much more conversation and participation than broadcast. Think of it as being like trivia night at your favorite bar. There will be some organized activity, but a lot of it will be just connecting with friends. Some folks will be regulars, others will be semi-regulars, and still others will drop in occasionally. They start every Thursday at 4 PM Eastern Time. The bar closes at 5:30.

    The dress code is casual; BYOB.

    I recommend that you also subscribe to the iCal feed, which you can do by going to this page and clicking on the “iCal Feed” button on the bottom. I have events tentatively lined up for the handful of weeks, which I will populate to the calendar as they are finalized.

    RSVP here: https://www.runtheworld.today/app/invitation/12965

  • Blursday Social is Off This Week and Next

    The bar is closed for renovations this week and off for Thanksgiving next week. We’ll be re-opening on Blursday, December 3rd.

    Stay tuned for more information about upcoming guests.