e-Literate

Present is Prologue

Tag: Empirical Educator Project

  • A Courseware Platform for Expressing Pedagogical Intent

    A Courseware Platform for Expressing Pedagogical Intent

    In a recent post, I announced a collaboration between Carnegie Mellon University’s (CMU) Open Learning Initiative (OLI) group and Arizona State University’s Center for Education Through eXploration (ETX) to build a next-generation, open-source, open-standards project, which has been facilitated by the Empirical Educator Project (EEP). As I noted in that post, because each group has intra- and extra-university diplomacy and policy responsibilities, I have pre-approved the previous post and this one with ETX and OLI, respectively. With these two posts in place, I will explore some implications in subsequent posts without worrying about coloring outside the lines.

    The first post in this series described the nature of the collaboration. This one will describe the nature of the project itself.

    A recap of how we got here

    Here’s the sequence of events that led to the current project:

    First, about a year and a half ago, CMU announced at the EEP summit that they were releasing an open-source collection of over $100 million’s worth of learning engineering software they had developed in a collection they call the OpenSimon Initiative. OpenSimon initially included parts of the OLI’s platform—also called OLI—which is both the original template for most current-generation courseware platforms and a platform designed from the start to support research into applied learning science. (The reasons why they could not initially release all of OLI are unimportant for this story. The resolution of the problem, which I’ll get to momentarily, is more salient.) Norman Bier, who is both the Executive Director of the Simon Initiative and the Director of OLI, spearheaded OpenSimon’s release and has led OLI’s participation in the collaborations to come.

    Separately, in the spring of 2019, ETX Director Professor Ariel Anbar decided to fund a courseware interoperability standards effort as part of an Open Educational Resources (OER) development grant the US Department of Education had awarded to the Center (a project which has since been branded OpenSkill). Since ETX planned to develop their OER courses on the proprietary Smart Sparrow platform, Ariel believed that those courses would need to be portable to other platforms to satisfy the spirit of OER. So he hired EEP to help convene a group and facilitate the work. One of the participants that I recommended was OLI. We had a kick-off convening in September of 2019.

    Shortly after that convening, Pearson announced that it had acquired the SmartSparrow platform and would be discontinuing support of the product in its current form. (As I noted in my previous post, I have nothing to say about Pearson’s actions other than that ETX stakeholders have told me that the company has worked with the university to make the transition as smooth as possible.) Unfortunately, since we do not yet have courseware interoperability standards, Pearson’s ability to make the transition smooth is limited. There is no easy way to get the functioning courseware out of Smart Sparrow and import it intact into some other one.

    Rather than letting this substantial transition derail the interoperability work—ETX has over $15 million’s worth of grant–funded courseware that they need to migrate off of Smart Sparrow by a fixed deadline—Ariel decided his predicament proved that interoperability is too important to ignore. So he began looking for partners who would be willing to invest in this goal.

    During roughly this same time period, OLI began developing the next generation of their platform, code-named “Torus,” which could be released as open-source. (The code repository for the project is available on Github today.) After some discussion and exploration, ETX and OLI agreed to be primary partners on this project, developing a specification that would enable ETX’s Smart Sparrow-based experiences to run intact on Torus. They also agreed to design the architecture to use existing interoperability standards whenever possible, propose enhancements to those standards when it is helpful, and to invent non-standard approaches as needed. Torus will be a reference implementation for proposed courseware interoperability standards. It will demonstrate the viability of the proposed standards by showing that it can import and run ETX’s existing Smart Sparrow intact and by enabling ETX to continue authoring Smart Sparrow-style courseware in the future. ETX has engaged EEP to facilitate this collaboration.

    A rare opportunity

    In this project, we have two different applied academic centers that both build courseware for use at scale. They focus on substantially different (though complementary) research-based course design approaches. Each has chosen a platform because of its suitability for their respective approaches. Both perform efficacy research on the courseware they develop. By bringing them together in this way, we have a rare opportunity to intentionally drive the evolution of the entire product category, starting with a consideration of research-supported pedagogy and driving it deep into the fundamental structure of how courseware platforms are built by influencing their interoperability standards. Technology is the means, not the end. The real goal is to develop a generalized approach for courseware platform architecture that can implement a range of evidence-based design approaches and drive continuous improvement of digital curricular materials that demonstrably support student success and improve equity.

    For this reason, I will not be writing about the technical details in this post. Instead, I will focus on foundational pedagogical and functional concepts that drive the design decisions. Readers who are specifically interested in technical interoperability or architectural questions will have to read between the lines. I promise to cover these topics more explicitly in later posts.

    Because we have been building toward this public collaboration for quite some time, I have been laying a breadcrumb trail regarding this project since long before I could talk about it explicitly. If you want to really study this project, I recommend reading the other posts that I reference. The links to previous e-Literate posts embedded in this one are more important than usual.

    First principles

    Let’s start by laying out some basic pedagogical principles that both organizations endorse and that infuse all aspects of plans for this project, from the interoperability standards to the software architecture to the user experience.

    First, “to learn” is an active verb. Students cannot “get learned” or “be learned.” They must do the learning themselves.

    Learning results from what the student does and thinks, and only from what the student does and thinks. The teacher can advance learning only by influencing the student to learn.

    Herb Simon

    “Influencing the student to learn” can be interpreted as provoking students to think and do things that you hope will stimulate them to learn. People learn when they try something or think about something and come to a conclusion or insight based on that process of exploration. Part of the craft of teaching is knowing what to give students to think about or try.

    The actual learning happens inside a student’s head, which means that it is not directly observable. So another part of teaching is giving students tasks to perform that provide observable clues as to what they may or may not have learned. We call this teaching task “assessment.” Importantly, self-taught learners assess themselves. For example, if I am learning how to make scrambled eggs, I will try to scramble and egg and observe how it comes out. I won’t know if I learned how to make scrambled eggs until I try to scramble one. Likewise, I won’t know if I’ve failed to learn how to scramble an egg. If I assess myself, then I can keep trying things and observing the outcomes until I learn whatever it was that I missed the first time (like, for example, how to tell if the pan is the correct temperature for scrambling an egg). Self-taught learners are, in part, self-assessing learners. They construct tests for themselves to figure out what they haven’t gotten right yet. Educators assist learners with that process.

    Regardless of whether I have a cooking coach or if I am teaching myself, I may very well decide what to try next based on the results of my assessment. If my eggs come out poorly, then I will think about what I might have done wrong and try something else. If they came out well, then I might try incorporating them into a breakfast burrito. Or I might move on to learning how to make bacon.

    In other words, learning is a feedback loop, where the input is the learning experience and the output is the assessment result. Learning may happen in between the input and the output, inside the black box that is the learner’s head. It may also happen when the learner sees the output, which might become the input for another learning feedback loop. A failed attempt to scramble an egg may cause me to do or think something that enables me to learn how to successfully scramble an egg.

    It works something like this:

    1. Input: I read a recipe for scrambled eggs.
    2. [Weird things happen in the dark place between my ears.]
    3. Output: I try to scramble an egg.
    4. Inference: I judge whether I am ready to move onto making a breakfast burrito or if I have not yet achieved egg-scrambling competency.

    (It’s not exactly this clear cut. For example, I learn things about scrambling eggs in the process of scrambling an egg.)

    As part of that last step, I’m evaluating whether I’ve learned what I set out to learn. Hence, the feedback loop. Based on that inference I make from the assessment results, I decide whether to repeat the learning feedback loop, adjust it, or move on to a new one. But again, a critical point here is that Step 2—the actual learning—is invisible. The brain is a black box. We only know what goes in and what comes out.

    A widely practiced discipline in learning design, known as “backward design,” is built on these fundamental insights. At its simplest, backward design has three steps:

    1. Specify the desired learning goal or competency (e.g., scrambling an egg)
    2. Design the assessment that will test progress toward the competency (e.g., trying to scramble an egg)
    3. Design the experience that will stimulate the learner to do or think something that will hopefully enable them to learn (e.g., reading a recipe)

    Steps two and three in backward design correspond to the output and input of the learning process. While step one was implicit in my previous example—I started off wanting to learn how to scramble an egg—it must be explicit when designing a curriculum using this method.

    The backward design learning feedback loop. (Credit: Carnegie Mellon University)

    In fact, the curriculum can be thought of as the scope and sequence of these specified learning goals. The scope refers to the full range of competencies that I need to master to reach a larger goal. Is my ultimate aim to learn how to make a bacon-and-egg breakfast or a breakfast burrito? The sequence is the order. I can’t learn how to make a breakfast burrito until I first learn how to scramble an egg. In other words, we can think of the process of learning design—and teaching, and learning—as constructing a series of nested learning feedback loops. Many educators (and learners) construct these loops instinctively and unconsciously, which means that they sometimes miss opportunities to make their designs more effective. But whether consciously or not, all learners, all educators, and all designers of instruction construct sequences of feedback loops. Because without learning feedback loops, there is no learning.

    Computers are (mostly) dumber than humans

    So humans can’t learn, and therefore can’t teach, without feedback loops. They can’t tell if learning is happening, never mind provoking thought that leads to learning. Guess what? Computers can’t either. And no amount of data or fancy AI algorithms will change that, because both learning and teaching happen via feedback loops. I’ve seen (and written about) a lot of magical thinking in EdTech that talks about data as if it’s fairy dust: one speck is pretty much the same as any other, but the more you have of it, the more magic you get.

    I wrote a blog post last December called Pedagogical Intent and Designing for Inquiry which was all about this misconception and how it harms our efforts to build effective EdTech. “Pedagogical intent” should be interpreted as the reasoning behind the design of the learning feedback loop. For example, the software might detect patterns in students’ answers that suggest a particular skill is not being taught well (or at all). Or it might identify a prerequisite skill or bit of knowledge a student missed earlier that is causing her to struggle now.

    To find these insights, the computer needs more information about the activities than just the clicks. It needs information about pedagogical intent. Why is the educator assigning the student a particular activity to perform? What was a particular assessment answer choice intended to suggest about the student’s internal mental processes? Without knowing the answers to these questions, the computers are as blind as we are. Here’s an example from that post:

     ACT recently released a paper on detecting non-cognitive education-relevant factors like grit and curiosity through LMS activity data logs. This is a really interesting study that I hope to write more about in a separate post in the near future, but for now, I want to focus on how labor-intensive it was to conduct. First author John Whitmer, formerly of Blackboard, is one of the people in the learning analytics community who I turn to first when I need an expert to help me understand the nuances. He’s top-drawer, and he’s particularly good at squeezing blood from a stone in terms of drawing credible and useful insights from LMS data.

    Here’s what he and his colleagues had to do in order to draw blood from this particular stone…

    First, they had to look at the syllabi. With human eyeballs. Then they had to interview the instructors. You know, humans having conversations with other humans. Then the humans—who had interviewed the other humans in order to annotate the syllabi that they looked at with their human eyeballs—labeled the items being accessed in the LMS with metadata that encoded the pedagogical intent of the instructors. Only after they did all that human work of understanding and encoding pedagogical intent could they usefully apply machine learning algorithms to identify patterns of intent-relevant behavior by the students.

    LMSs are often promoted as being “pedagogically neutral.” (And no, I don’t believe that Moodle is any different.) Another way of putting this is that they do not encode pedagogical intent. This means it is devilishly hard to get pedagogically meaningful learning analytics data out of them without additional encoding work of one kind or another.

    Pedagogical Intent and Designing for Inquiry

    So far, machine learning has not proven to be generally effective at deducing pedagogical intent based on low-level data like student clicks or time they spend watching a video. It’s getting pretty good at deducing pedagogical intent from data when it is given inputs that are already well-structured to supply hints of that intent. LMS activity logs are not well-structured in this way. Ironically, analog textbook chapters are because they are designed to give humans clues about these things using only text organization, pictures, font styles, and the like. But that doesn’t help us to reliably deduce pedagogical intent in the realm of the digital. There ought to be a better way.

    Luckily, there is one.

    Learning feedback loops in courseware design

    It turns out that virtually every commercial courseware product on the market today, and virtually every course created with substantial input from a professionally trained learning designer, has been built using backward design. Furthermore, most interesting and useful learning analytics in commercial courseware products today get virtually all of their utility from this backward design and very little, if any, from some super-genius artificial intelligence algorithm. Machine learning is most often useful when it can “read” the backward learning design to work its magic. Today’s learning analytics and adaptive learning systems are best thought of as algorithms that attempt to squeeze as much insight and value as possible from the pedagogical intent that has been encoded into the courseware platform in a machine-readable form. They may use other data too. But as the ACT study above illustrates, those other data have limited value when they are disconnected from the context of the learning feedback loop designs and the pedagogical intent behind them.

    Last September, I wrote a three-part post series on exactly this topic called “Content as Infrastructure” (the title of which was borrowed from David Wiley). You can probably skip the first post since it covers a lot of the same ground you’ve just read. The second post shows examples of different products that take advantage of backward design to create better educational products, while the third one digs into the various types of insights that can be gained from computers when they have access to semantic information about pedagogical intent. If you want to dig a little deeper, you can go back to my post from 2012 about how CMU’s learning engineering techniques use learning feedback loops to identify hidden prerequisites that the educators may not have realized they need to address (like the recipe writer who doesn’t realize that not all aspiring egg scramblers know how to tell if the pan is the right temperature). Or you can read my post from 2013 about how ETX’s Habitable Worlds course creates experiences that provoke student learning.

    Generalizing feedback loops

    As promised, I will keep this post non-technical. But given the pedagogical principles above, we can begin to think through the basic requirements for a universal technology design.

    First, we need standard ways to describe the smallest and simplest learning feedback loops, like a single multiple-choice question. The student reads the question, chooses an answer, and receives feedback. Simple. And yet, not. A well-written multiple-choice question will have wrong answers—or “distractors,” as they are often (problematically) called—that are intended to diagnose specific common misunderstandings. Students might be given multiple tries on the question, thus traversing the feedback loop multiple times. They might get feedback based on their answer which is intended to help them learn or reinforce their learning. They might be given the option to ask for a hint. Or multiple hints. And each successive hint may be designed to get at a different possible blocker to the student’s learning the correct answer. A single well-design multiple-choice question implemented in a system that gives options for multiple tries, feedback, and hints can be absolutely loaded with pedagogical intent. We want a system that captures as much of that as possible and as automatically as possible. The more that the author’s pedagogical intent can be captured as part of the natural authoring process—rather than requiring the author to add that information afterward—the better.

    Second, we need a system that enables learning feedback loops to be connected to each other in different ways, and to create loops within loops. On a fundamental level, if I want to make a bacon-and-egg breakfast, I have to learn how to scramble an egg. If I want to learn how to scramble an egg, I have to learn how to tell if the pan is the right temperature. If I want to learn how to become a short-order cook in a diner, then I need to learn all these skills plus many more.

    We also need the feedback loops to be connectable in many ways. A common approach in OLI courses is to give students a string of activities that address the same “learning objective”—a label we attach to a learning goal of a certain granularity that is meaningful in the context of the larger course—and watch to see if the students’ error rates on those problems go down. This requires the ability to tie those problems together in a sequence. A common approach in ETX courses is to create “guided experiential learning” exercises, in which students learn by exploring. This requires the ability to branch students to different learning feedback loops based on their answer to the previous one (even if the branch is just a short detour that leads back to the main path). Both approaches are valid and not mutually exclusive. They can be used in the same course. Sometimes they can even be incorporated into the same exercise.

    When we talk about interoperability—which was where this project started—we can mean at least two very different and equally important goals. The first is to be able to export a learning feedback loop design of any grain size—a multiple-choice question, a quiz, a loop that addresses a skill, or a learning objective, or a competency, or a lesson, a course, a certificate program, etc. (You might be surprised at the amount of thought that goes into designing a single multiple-choice question to create a truly diagnostic learning feedback loop.)

    The generic word we use for a compound learning loop of any size is an “ensemble.” I’ve seen a lot of debate and hand-wringing over the years about what a “learning object” is, how big or small one can be, and so on. The problem with “object” is that it’s a fancy name for “thing.” Of course very different kinds of learning things should be very different sizes. In this project, we talk about learning feedback loops, which also can be very different sizes. But they are all the same basic type of thing, in the sense that they have learning inputs and produce learner outputs which are diagnostic of particular learning goals. Any learning thing—whether it is a picture or a course unit or something else—which does not have these characteristics is not a learning feedback loop.

    We should be able to move the content, the execution logic of the learning loops, and the encoding of the pedagogical intent, intact, from one system to another. We also should be able to analyze student activities in the context of the pedagogical intent of the learning loop the same way for any courseware in any system. We should be able to know what the learning goal is, what the student’s assessment answer suggests about progress toward that goal, and so on. The interoperability project aspires to achieve both of these goals, although they will take some time to refine and to take them through an official standards-making process.

    And we want to create a system that encourages educators to author learning feedback loops that are well-crafted and continuously improve that content based on feedback from the system on how well it is helping the students. Most educators—particularly ones who have no formal training in education, like the vast majority of college professors—do not naturally think about what they are doing as crafting series of learning feedback loops. Even when they understand the basic idea, they may not have all the skills that could help them (just as understanding the basic concept of scrambling an egg does not automatically convey the skill of making yummy scrambled eggs). For example, a learning designer recently told me that some of the professors she works with don’t know that they should try to write multiple-choice distractors designed to diagnose specific misconceptions. In an ideal world, a courseware platform would have an authoring interface, analytics, and nudges that all encourage courseware designers—whether they are first-time amateurs or skilled professionals—to constantly be working to test and refine their learning feedback loops.

    This thing is real

    These are obviously big ambitions that will take a long time to realize. Some of them will take at least few years to get right or to get through formal standards-making processes. That said, this is more than just another grand idea that will probably never happen.

    It is happening right now.

    Here are some milestones that the collaboration has achieved so far:

    • Torus exists. The source code is available. Some simple courses are in production on a stable version of it.
    • OLI had several teams build sample lessons on Torus during a virtual summer school program.
    • ETX, with the help of Unicon’s architects and developers, has exported the lessons from Smart Sparrow, including the logic of the learning loop design.
    • Unicon has built several iterations of a harness that can run a wide range of ETX Smart Sparrow with increasing fidelity. To be clear, they are not rebuilding these lessons. They are running the exported lessons, with the full learning experience, intact.
    • The collaboration team has demonstrated the ability to launch the ETX lessons from OLI.
    • IMS Global, one of the major technical interoperability standards bodies for education, has been kept apprised of the work and has been actively supportive of it.
    • Unicon’s latest iteration of the Smart Sparrow test harness runs the full representative sample of ETX lessons with production-quality fidelity. (A few minor tweaks remain to be completed.)

    Here are some planned milestones:

    • The collaboration team is in the process of generalizing the runtime commands for the ETX learning loops into a specification that can be implemented in Torus and submitted as a formal interoperability specification candidate.
    • In parallel, ETX and Unicon are working on a developer’s aid that will enable ETX experts to begin creating new ETX-style lessons in the first quarter of 2021. While it will not be suitable for general use, it will help us continue to refine the specification and think about broader authoring requirements.
    • By the fall of 2021, ETX expects to migrate all of its Sparrow courses onto Torus and OLI expects to put a substantial subset of its courses into production on Torus.
    • Work on enriching the authoring interface will proceed in parallel, with the goal of releasing an authoring solution that fully supports multiple styles of course design by January of 2022.
    • OLI will complete its migration of all its courses to Torus by the fall of 2022. Given the current size and growth of the two projects, I expect the combined annual enrollments on the platform to be well north of 150,000 – 200,000 at a minimum at that time.

    The progress and productivity of this collaboration has been amazing. I’ve never seen anything like it in my entire career. And the architectural innovations are quite clever. I can’t wait to start writing about them.

    But the main take-away for now should be that this project is real, it is already producing tangible results, and it will be out in the world in a meaningful way within a year.

    Some of you have already asked me if you can observe or even participate in this project. We are thinking about ways to do that and will have some answers for you in the new year. There will likely be several options, depending on the nature and seriousness of your interest. Opportunities for observation will likely come before we can open up for participation, particularly given the aggressive deadlines that both ETX and OLI have to meet. But this is a public good project with a core commitment to openness. If you are interested in getting involved, please contact me. I’m gathering information about interest so that we can begin planning a community strategy.

    Stay tuned for more

    This post barely scratches the surface of the ambition. In future posts, I will be digging deeper into the details. In the meantime, I suggest again that you (re)read the second and third installments of my Content as Infrastructure post series to get a little more grounding in how much work the basic structures I’ve described here can accomplish and my post Pedagogical Intent and Designing for Inquiry regarding why capturing pedagogical intent is so important for gaining insights on teaching and learning out of these systems.

  • Resilience Network: Resources

    I have organized my last two posts into the beginning of a series, which allows me to create a running table of contents on the relevant blog posts. This post is the third in a series focused on the creation of an education resilience network that can respond more effectively in times of crisis, whether the crisis is a pandemic, a hurricane, a war, or some other disaster that disrupts the normal functioning of educational institutions. I will have at least one announcement to make regarding the formation of such a network very soon. In the meantime, I will be using this series to lay out a vision for what such a network could look like and how it could function. In the context of the current COVID-19 crisis, my goal is to organize a better response for next term while providing some tips to the people and organizations that are working on an immediate response regarding how to increase their impact (e.g., putting an explicit and permissive license for re-use on any resources that are intended as donations or contributions to the community).

    As a community, we’re learning a lot about what a good (or bad) disaster response looks like by watching our governments and health systems. The logistical challenges are daunting. We have to produce good information resources for everyone, like what hygiene measures work and how to practice social distancing (or how to use a web conference system or develop a rhythm for working in an online class). We have to make sure the information about such effective practices is accurate and gets to everyone. We also need to get more specialized information to caregivers, like how to protect themselves and prevent disease transmission in hospitals (or how to create an effective online lesson). Then there’s information that needs to go to people who run care-giving institutions, like how to get test kits (or access to increased online teaching infrastructure). All of this needs to be accomplished consistently, accurately, and efficiently to reach everyone who needs accurate and useful knowledge. Some of that knowledge, like proper hand-washing technique (or how to create an effective instructional video) has a long shelf-life, while other knowledge, like how healthcare workers (or educators) should respond to local needs based on changes in available resources and supplies, is continually evolving.

    Beyond creating and distributing information resources that are helpful—and combatting the spread of misinformation that is harmful—we also have to identify and distribute other resources. Help, either in the form of consultative expertise or supplemental front-line workers, is a crucial one. We need to match available resources with current needs quickly and effectively, all while providing some means for vetting the help to make sure that the people, however well-intentioned they may be, are qualified to offer help. And then there are non-informational resources, whether they are N95 masks or web conference accounts. Some of these will be new resources that people need for the first time, while others will be providing financial relief to prevent the loss of ongoing access to current resources.

    How well we respond to the current crisis can make an enormous difference that differentially impacts more vulnerable populations. In an epidemic, the homeless, prisoners, and people without health insurance are among those who are least likely to be protected by our existing response mechanisms. Likewise, we know that first-generation students are more likely to struggle in online education programs even when those programs are carefully designed and not thrown together in haste. Our response to the current crisis can also put us in either a better or a worse position after the immediate crisis has passed. We can pass emergency economic legislation that protects workers and preserves jobs, or we can pass legislation that enables companies to use bailout money for stock buybacks and executive bonuses. Likewise, we can engage with educators on how to serve their students more effectively online, or we can just start spending money on a whole lot of Zoom accounts and robot tutors when the free offers run out.

    I don’t intend to denigrate the efforts or good intentions of anyone’s current response to the COVID-19 situation when I write this, but if our crisis response capability does not evolve past Google Sheets, webinars, and limited-time free product offers, then we are going to fail to support the people with the greatest need during this extended crisis while also failing to learn how to respond better to the next one.

    In this blog post, I’m going to write about the different kinds of resources for which we need to develop a supply chain in an education resilience network. In the next post, I will write about the challenges of creating, vetting, and distributing those resources.

    Content

    By content, I mean information that can be broadcast one-to-many rather than person-to-person expertise sharing. This encompasses a vast array of information resources targetted at distinct but overlapping audiences, including (but not limited to) the following:

    • Curriculum-specific information, like how to teach your dance class online
    • Technical information, like how to post an announcement in your LMS
    • Learning design information, like how to make better use of formative assessments in an online context
    • Policy information, like guidelines for what instructors can and cannot expect their students to be able to do during the crisis
    • Logistical information, like where to seek help for particular kinds of problems

    These resources can be either durable or not and either general or context-specific. Different characteristics call for different responses:

    [wptb id=11021]

    We should be mindful during this time of massive change that now is the time when new practices—either good or bad—take hold. For example, we know that instructors only tend to redesign their courses substantially when there is an extrinsic driver forcing them to rethink how they teach. We are currently experiencing the mother of all extrinsic drivers. We should be taking the opportunity to make sure we are teaching them about evidence-based effective practices. This is, in part, an equity issue. Disadvantages students are disproportionately vulnerable in the move to online. We lose them more easily. As a sector, we have an ethical responsibility to ensure that we are applying everything we have learned about educating well in order to help these students. That is even more important at a time when these very same students are particularly vulnerable in the rest of their lives, such as during a global pandemic that is shutting down the economy.

    Resources

    In an educational resilience response to a crisis, where the emphasis is on moving students to online, there will be a need for technology as well as information. People should have defined resource centers where they can go to find what they need and get advice about how to use it. I’ll offer one example. I am getting inundated with messages about COVID-19 resources and offers of help that various folks want me to help them promote. While I could easily turn e-Literate into a running announcement board for such offers, I have resisted that temptation. It’s not what e-Literate does best, and I see no point in duplicating work that others are already doing. For example, ISTE and EdSurge include a directory of vendor offers on their Learning Keeps Going web site. They have the resources to maintain such a directory and are a natural home for it. We should just use that rather than creating something else (or 100 something elses).

    I would love to see them include guidance to both schools and vendors about topics like how to design and vet such offers to address student data privacy concerns. In other words, there is a whole set of practice knowledge that attaches to this kind of a list. To the degree that one organization, network, or community becomes a hub for a resource (whatever that resource may be), it should also become a community hub for sharing information about the effective use of that resource.

    In this way, an education resilience network isn’t a hub-and-spoke kind of centralized response but rather a network consisting of multiple nodes of expertise concentration that are coordinating with each other as needed. I’ll have more to say about this in my next post.

    People

    This is an extension of my previous paragraph—including my comment about having more to say about this in my next post—but the most precious resource we need to coordinate is humans who can help. Many organizations are stepping up to play a role in matching helpers with need, but (a) a lot of companies and universities are just tossing information out into the ether without the benefit of coordination, and (b) the associations and other community nodes that are coordinating with their constituents are not coordinating with each other. This is the hardest part of the education resilience network to optimize, but it is also the most important. Ultimately, we will find that effective content and resource production and distribution in an emergency are second-order effects of the ways in which we organize ourselves—or don’t—for a whole-sector response to the crisis.

    Resource needs are also dynamic. At this moment in the midst of a crisis for which the education sector has not been well prepared, I find that most people I talk to fall into one of two categories. Either they are about to drop from exhaustion because they are frantically working the front line or they are climbing the walls trying to find some way to be useful because the work that they normally do to help has ground to a halt. We are doing a very poor job of coordinating those imbalances at the moment. And they will change as we finish the current term and people start preparing for the next one. We need organizational structures that enable us to balance human resource utilization—matching people who need help with people who can offer help—more dynamically.

    That challenge will be the topic of my next post.

  • COVID-19 and e-Literate

    Friends, the first thing I want to say to you is that you are in my thoughts. As I watch so many of you make heroic efforts to support your students and colleagues even as you worry about yourselves and your loved ones, I am inspired. My sister, who is both a teacher and a rabbi, has banned the phrase “social distancing” from her vocabulary, preferring to use the term “physical distancing.” She is right. Now is the time to be closing the social distance among us, not increasing it. That is exactly what so many of you are doing. You have inspired me to think more seriously about what e-Literate should be doing to help.

    That’s what this post is about. Feedback, as always, is welcomed.

    Immediate term: postponing a webinar

    Right at this moment, the most useful thing I can think of to do is to stay out of your way. After discussing the situation with Standard of Proof webinar guests Tyton Partners, we agreed that this week is not a good time for the conversation we had planned to have. It’s unfortunate because their research has a direct bearing on how to better support the students who are most likely to be hurt in a sudden, unplanned transition to online education. But this week, many of you are either preparing for that transition, in the middle of it, or trying to deal with the immediate aftermath of it. You have more time-sensitive priorities at the moment, including self-care.

    We will set a new date for the webinar once the dust settles and it becomes clear to us that more of you are in a position to have a constructive conversation about something other than your immediate responsibilities. I don’t think we’ll have to wait terribly long for this, but it won’t be in March.

    The role of the Empirical Educator Project (EEP)

    I have come to the conclusion that EEP can play a supporting role in this crisis. Therefore, it should. I am embarassed to admit that this wasn’t my idea. I got a text from a friend who is also an academic participant in the EEP network. When I didn’t catch on to his suggestion immediately, he found time to call me, after a full day of work, while he was cooking dinner for his family, to explain his thinking and urge me to consider the possibility.

    Among other things, our response to COVID-19 is a collective action problem. But in education, there is an unusual amount of goodwill and agreement about our high-level collective responsibilities. The real problem is that there are costs to collaboration, and the coin in which those costs most often need to be paid is the one which educators are sorely lacking: time. The sector has considerable expertise on how to deal with this crisis as competently and humanely as possible under the circumstances, but it is not evenly distributed and it is not being distributed effectively at the moment.

    I’ll give you one heartbreaking example. Some of the good folks in the POD network have taken precious time away from their crisis management at their own institutions to organize a list of information resources about remote teaching continuity plans that their respective institutions have put together so that their colleagues at institutions that don’t have such plans can benefit from them. About 200 institutions have contributed so far. Two hundred people who are directly responsible for helping their institutions make the transition right now took a moment out of their long, long days to share with their colleagues at other institutions. They are doing what they can. It is not their fault that the value of this information resource is severely limited. First, a lot of the people who need it aren’t going to know about it. It’s in a Google Sheet. One of a bazillion. Every time I tell people about it, they say, “Oh, is that the one that Bryan Alexander is organizing?”

    (No, that one is here. It has over 250 contributions. I don’t know how much overlap there is between Bryan’s sheet and POD’s.)

    Second, while the list is better than nothing, using it is far from easy. Some of the links are to resources that are behind a login. So they’re useless to outsiders at the moment. Others go to sites with varying amounts of information of varying quality covering varying topics. Somebody trying to make use of the resource in its current form would have to keep clicking through to web sites and sifting through them until they found what they needed. Maybe.

    And yet, this is hardly the first situation that has called for a more organized response to helping fellow educators. Where was this level of effort for the universities in Puerto Rico during the hurricane, for example? Nor will this be the last such crisis. Both natural and human-created disasters disrupt the ability to support students somewhere in the world all the time. The pace of these disruptions is not going to slow down any time soon. The sector needs to become more resilient. We need to find ways of lowering the cost of educators helping each other so that we can achieve more impact with less educator burnout.

    The Empirical Educator Project was designed to address this kind of collective action problem. Its founding hypothesis is that, while higher education is in the early stages of making the transition from a philosophical commitment to student success toward operational excellence at enabling it, this transition is being both slowed and deformed by ineffective sharing of actionable knowledge. This opens the door wider for both dangerous shortcuts and harmful opportunism. The problem we face with COVID-19 is the same, only accelerated. There is both opportunity and danger here. The opportunity is that the stress on our system is making the flaws more obvious and gives us a chance to optimize the system for the challenges of the present and future. The danger is that the imperative to act quickly will lead us to act unwisely. We need to both lower the cost of effective knowledge sharing and put some ethical guardrails in place.

    This is what EEP will try to help with.

    Short-term: Improving the vendor response

    Let’s be honest: While educators are going to try to do their best they can for their students, the likelihood that most students will lose educational quality this term is pretty much 100%. No EdTech magic will change that. There will be no miracle “vaccine” this month or next. Everyone will do what they can. The short-term priorities are preventing deaths from the disease and supporting each other, including but not limited to our students, in getting through the immediate economic and mental health consequences of the crisis. The majority of vendors would have very limited ability to help this term even if they come to the table with the purest of intentions and a near-perfect understanding of what is needed.

    That said, there are good reasons to engage responsibly with vendors right now. For one thing, while some of them can’t help with the problems of this term, others can, in ways that might or might not have anything to do with products. A good friend of mine who is also the CEO of an EdTech company told me that he has smart employees who are sitting on the bench right now due to project delays caused by the crisis and would like to see them volunteering rather than sitting on their hands. Meanwhile, I’ve been having calls with a number of companies that are looking for guidance on how they can be more helpful. Also, even if a vendor can’t help with the problems of this term, there is time to organize a better response for next term, when there is a non-trivial chance that we will still be dealing with COVID-19 challenges. And even if we aren’t, we should be preparing for the next crisis as part of our response to this one.

    Expanding the sponsorship base of EEP was always part of the plan. While I have been very careful in the early years to only accept sponsorship either from companies that have already demonstrated they are aligned with EEP’s goals and values or particular people at companies who I know and trust, my ultimate goal has always been to help a wide range of EdTech companies that want to be better servants to education learn how to do so (and be rewarded proportionately for good behavior). I have decided that the current situation calls for me to broaden the tent a little earlier than planned. If a company comes to me for support in learning how to be more helpful, if I perceive that they are coming to me with a baseline level of earnestness, and if I can see a way to help them do the right thing, then I will help them.

    I’ll provide more detail on what “doing the right thing” means to me in the next section.

    Medium-term: Fostering a resilience network

    As I wrote in the previous section, we are likely to face continuing COVID-19 challenges in the next term and certain to face challenges from other crises into the indefinite future. While I mean no disrespect to the good people who are organizing the crisis response, we need to get further along than a bunch of separate Google Sheets floating in the ether. We need to get better at sharing effective practices and supporting policies and at distributing knowledge of such things more broadly. Where there are resource gaps, we need to identify what they are and advocate for filling those gaps as part of our coherent response plans. This is primarily about people, knowledge, and organization. The technology should be an enabler of practices that educators have reason to believe are effective. It is crucial but second-order. The first-order problem is the creation of a resilience network.

    Right now, most educators I know are focused on the immediate problem and can’t spare much thought for next term, never mind the indefinite future. It’s all many of them can do to put their line in a spreadsheet or make a call to a friend who they think can help. But that will change—somewhat—and we need to be ready to make the most effective use possible of the work that will come with the after-action reviews and the prep for next term. I expect to engage the EEP network—and the wider academic world—in this conversation when they can come up for air.

    In the meantime, as I talk to the vendors, my focus is going to be on supporting that network. I’m not interested in creating a laundry list of limited-time free offers (though I don’t pass blanket judgment on such offers either). Rather, I’m interested in finding ways that the vendors can support the academics in diffusing relevant knowledge more widely and supporting each other more effectively. For this reason, I will be mindful about offers to use platforms. I don’t eschew them by any means. Google Sheets is a platform, and it is better than having nothing at the moment. But I will be focused on using platforms when they support the network purpose and consider the tradeoffs carefully. Laura Czerniewicz has a good guest post on PhilonEdTech (POET) about her experience thinking through such tradeoffs from an academic perspective (among other relevant topics).

    I know there is a wide range of feelings in the academic community and I don’t claim to have the best, one-size-fits-all answer. But I can play the role of an honest broker. My personal experience in the conversations I’m having with folks who work at EdTech companies right now is that they want to help. They feel that if they can, then they should. But it’s hard to figure out how to be helpful. If I can help to reduce the social distance between them and the people they want to support in this time of need, then I will. Some of that support will come in the form of help now, but a lot of it will be focused on preparing for the next round.

    Educators need to lead on this, and they can’t do that effectively while also responding to the immediate crisis. Those of us who are not immediately engaged in addressing the crisis need to prepare to help those folks make the most of their efforts going forward.

    Stay tuned.

  • SoP Webinar on Instructional Designer Professional Development

    Improvement in post secondary education will require converting teaching from a solo sport to a community based research activity.

    Herb Simon

    For all the talk lately of the “future of work,” we don’t talk enough about the future of work for educators. We have a growing shortage of well-trained instructional designers, course architects, accessibility experts, learning engineers, and similar specialists. This gap is only going to grow as improving student outcomes becomes increasingly critical to sustainability for colleges.

    That’s one reason why I’m pleased to announce an e-Literate Standard of Proof webinar on iDesign’s LX Pathways program, a new competency-based digital curriculum for professionals looking to develop these skills. iDesign created them, in part, to help with their own employee recruitment challenges as the company grows. This is a great resource for the sector, which was developed by iDesign, made available free or cheap—depending on which option you take and how far you go with it—to anyone, and tested with Harvard School of Education graduate students.

    Come hear iDesign Chief Learning Officer Whitney Kilgore and Patrice Torcivia Prusko, the Associate Director of Learning Design at the Harvard University Graduate School of Education Teaching and Learning Lab talk with me about LX Pathways.

    The webinar is on Thursday, February 20th, at 2 PM ET.

    Sign up here!

  • D2L Steps Up for EEP

    I’m delighted to announce that D2L has become the Empirical Educator Project’s first official Foundational Sponsor. The major principle behind sponsorship in EEP has always been that we only accept sponsors who have something to contribute in addition to money. The same is true for Foundational Sponsors. D2L is offering more to the academic participants in the Empirical Educator Project than just money and free resources. They have both demonstrated through participation to-date that they are good participants and offered enough value in future participation to earn pride of place. Their behavior is a model for other commercial participants in the educational community to emulate.

    To begin with, D2L VP of Market Research Kenneth Chapman, who I’ve known almost as long as I’ve been in EdTech, has been incredibly supportive of our work to-date. He arranged an integration demo to support Carnegie Mellon University’s OpenSimon announcement at our last summit. He’s sent staff to CMU’s LearnLab summer school as a follow-up. He’s made his people available for our work and has consistently and actively looked for ways to collaborate. Ken’s leadership and enthusiasm convinced me that I wanted to deepen the relationship between D2L and EEP.

    In terms of what D2L is offering as a Foundational Sponsor going forward, as I wrote earlier in this post, there’s the money (and contributions of financial value) and the participation. Let’s address the material part of the contribution first. D2L has committed to sponsoring EEP for the next three years. They have also offered up Brightspace as the EEP online community space for that period of time. This second part is going to be increasingly important as EEP work starts to become year-round and as we prepare to open up at least some of that work—both sharing and participation—to the general public. And D2L has offered up the help of its support team, including superstar Ben Campbell, to help us get the site set up and to teach us how to take maximum advantage of the affordances of the platform.

    By themselves, as generous as those offers are, they wouldn’t be enough to earn D2L a place as an EEP Foundational Sponsor. What sealed the deal was D2L’s interest in increased participation. I can’t share details yet, but at a high level, there are two aspects. First, they are actively interested in offering up Brightspace as a laboratory for experimentation. One natural place where that collaboration may go, as mentioned in D2L’s press release, is integration with Carnegie Mellon University’s OpenSimon software. As I mentioned further up in the post, there has already been a little work done in this regard, and there is plenty more to explore.

    At least as important is D2L’s offer to help bring the work and contributions of the EEP participants to the Brightspace user community. I’m really, really excited about the direction that this part of the conversation is taking and can’t wait to share more details as we nail them down.

    After EEP’s summit this spring, Bart Epstein, the CEO of the Jefferson Education Exchange and a passionate advocate for the kind of efficacy work that EEP is attempting to promote, offered up a note of reasonable skepticism in EdSurge’s coverage of the event and the contributions announced at it:

    When Tesla says that it’s making its battery patents available for free, you can be sure that all of the other car companies have incentives to invest time in reading and understanding those battery patents to read them and see if they can use them,” he said. “But when CMU opens up this software, it’s unclear who is out there that is saying, ‘Oh, there’s something in there that I want.’ We just don’t know how much impact it will have.

    Bart Epstein

    Nowhere is this concern more valid than in higher education. There are probably billions of dollars’ worth of intellectual property contributions that, practically speaking, are languishing on university servers where nobody knows that they exist, what they are good for, or how to use them. If we want to bridge this divide specifically in terms of new knowledge that could actually help students succeed in the real world, then we have to go beyond publishing papers and releasing open source software and OER (as important as those activities are). We have to develop an ecosystem and a culture for the diffusion and uptake of this knowledge.

    I invited D2L to be EEP’s first official Foundational Sponsor because their participation and other contributions will support our work in achieving this ambition.

    Stay tuned for more information as this collaboration continues to evolve.

  • EEP 2019 Summit Videos Are Up

    It took longer than I had hoped, but you can now see most of the Empirical Educator 2019 summit presentations here. (Unfortunately, the videographers didn’t capture the last couple of presentations on OpenSimon.) I’ll return to these after I finish my series on digital curricular materials design, but in the meantime, the talks are available for your enjoyment.

    Dig in.

  • Learning Engineering: A Caliper Example

    Learning Engineering: A Caliper Example

    In my recent IMS update post, I wrote,

    [T]he nature and challenges of interoperability our sector will be facing in the next decade are fundamentally different from the ones that we faced in the last one. Up until now, we have primarily been concerned with synchronizing administration-related bits across applications. Which people are in this class? Are they students or instructors? What grades did they get on which assignments? And how much does each assignment count toward the final course grade? These challenges are hard in all the ways that are familiar to anyone who works on any sort of generic data interoperability questions. 
    But the next decade is is going to be about data interoperability as it pertains to insight. Data scientists think this is still familiar territory and are excited because it keeps them at the frontier of their own profession. But this will not be generic data science, for several reasons.

    I then asserted the following positions:

    • Because learning processes are not directly observable, blindly running machine learning algorithms against the click streams in our learning platforms will probably not teach us much about learning.
    • On the other hand, if our analytics are theory-driven, i.e., if we start with some empirically grounded hypotheses about learning processes and design our analytics to search for data that either support or disprove those hypotheses, then we might actually get somewhere.
    • Because learning analytics expressions written in the IMS Caliper standard can be readily translated into plain English, Caliper could form a basis for expressing educational hypotheses and translating them into interoperable tools for testing those hypotheses across the boundaries of tech tools and platforms.
    • The kind of Caliper-mediated conversation I imagined among learning scientists, practicing educators, data scientists, learning system designers, and others, is relevant to a term coined and still used heavily at Carnegie Mellon University—”learning engineering.”

    In this post, I’m going to explore the last two points in more detail.

    What the heck is “learning engineering”?

    The term “learning engineering” was first used by Nobel laureate and Carnegie Mellon University polymath Herbert Simon in 1966. It has been around for quite a while. But it is a term whose time as finally has come and, as such, we are seeing the usual academic turf wars over its meaning and value. On the one hand, some folks love it, embrace it, and want to apply it liberally. IEEE has an entire group devoted to defining it. As is always the case, some of this sort of enthusiasm is thoughtful, and some of it is less so. At its worst, there is a tendency for people to get tangled up in the term because it provides a certain je ne sais quoi they’ve been yearning for to describe the aspects of their jobs that they really want to be doing as change agents rather than the mundane tasks that they keep being dragged back into doing, much like the way some folks are wrapping “innovation” and “design” around themselves like a warm blanket. It’s perfectly understandable, and I think it attaches to something real in many cases, but it’s hard to say exactly what that is. And, of course, where there are enthusiasts in academia, there are critics. Again, some thoughtful, while others…less so. (Note my comment in the thread on that particularly egregious column.)

    If you want to get a clear sense of the range of possible meanings of “learning engineering” as used by people who actually think about it deeply, one good place to start would be Learning Engineering for Online Education: Theoretical Contexts and Design-Based Examples edited by Chris Dede, John Richards, and Bror Saxberg. (I am still working on getting half a day’s worth of Carnegie Mellon University video presentations on their own learning engineering work ready for posting on the web. I promise it is coming.) There are a lot of great take-aways from that anthology, one of which is that even the people who think hard about the term and work together to put together something like a coherent tome on the subject don’t fully agree on what the term means.

    And that’s really OK. Let’s just set a few boundary conditions. On the one hand, learning engineering isn’t an all-encompassing discipline and methodology that is going to make all previous roles, disciplines, and methodologies obsolete. If you are an instructional designer, or a learning designer, or a user experience designer; if you practice design thinking, or ADDIE; be not afraid. On the other hand, learning engineering is not creeping Stalinism either. Think about learning engineering, writ large, as applying data and cognitive sciences to help bring about desired learning outcomes, usually within the context of a team of colleagues with different skills all working together. That’s still pretty vague, but it’s specific enough for the current cultural moment.

    Forget about your stereotypes of engineers and their practices. Do you believe there is a place for applied science in our efforts to improve the ways in which we design and deliver our courses, or try to understand and serve our students needs and goals? If so, what would such an applied science look like? What would a person applying the science need to know? What would their role be? How would they work with other educators who have complementary expertise?

    That is the possibility space that learning engineering inhabits.

    Applied science as a design exercise

    One of the reasons that people have trouble wrapping their heads around the notion of learning engineering is that it was conceived of by very unusual mind. Some of the critiques I’ve seen online of the term position “learning engineering” in opposition to “learning design.” But as Phil Long points out in his essay in the aforementioned anthology, Herb Simon both coined the term “learning engineering” and is essentially the grandfather of design thinking:

    Design science was introduced by Buckminster Fuller in 1963, but it was Herbert Simon who is most closely associated with it and has established how we think of it today. “The Sciences of the Artificial” (Simon, 1967) distinguished the artificial, or practical sciences, from the natural sciences. Simon described design as an ill-structured problem, much like the learning environment, which involves man-made responses to the world. Design science is influenced by the limitations of human cognition unlike mathematical models. Human decision-making is further constrained by practical attributes of limited time and available information. This bounded rationality makes us prone to seek adequate as opposed to optimal solutions to problems. That is, we engage in satisficing not optimizing. Design is central to the artificial sciences: ‘Everyone designs who devises courses of action aimed at changing existing situations into desired ones.’ Natural sciences are concerned with understanding what is; design science instead asks about what should be. this distinction separates the study of the science of learning from the design of learning. Learning scientists are interested in how humans learn. Learning engineers are part of team focused on how students ought to learn.”

    Phil Long, “The Role of the Learning Engineer”

    Phil points out two important dichotomies in Simon’s thinking. The first one: is vs. ought. Natural science is about what is, while design science is about what you would like to exist. What you want to bring into being. The second dichotomy is about well structured vs. poorly structured. For Simon, “design” is a set of activities one undertakes to solve a poorly structured problem. To need or want is human, and to be human is to be messy. Understanding a human need is about understanding a messy problem. Understanding how different humans with different backgrounds and different cognitive and non-cognitive abilities learn, given a wide range of contextual variables like the teaching strategies being employed, the personal relationships between students and teacher, what else is going on in the students’ lives at the time, whether different students are coming to class well fed and well slept, and so on, is pretty much the definition of a poorly structured problem. So as far as Herb Simon is concerned, education is a design problem by definition, whether or not you choose to use the word “engineer.”

    In the next section of his article, Phil then makes a fascinating connection between the evolution of design thinking, which emerged out design science, and learning engineering. The key is in identifying the central social activity that defines design thinking:

    Design thinking represents those processes that designers use to create new designs, possible approaches to problem solutions spaces where none existed before. A problem-solving method has been derived from this and applied to human social interactions iteratively taking the designer and/or co-design participants from inspiration to ideation and then to implementation. The designer and design team may have a mental model of the solution to a proposed problem, but it is essential to externalize this representation in terms of a sketch a description of a learning design sequence, or by actual prototyping of the activities which the learner is asked to engage. [Emphasis added.] All involved can see the attributes of the proposed design solution that were not apparent in the conceptualization of it. this process of externalizing and prototyping design solutions allows it to be situated in larger and different contexts, what Donald Schon called reframing the design, situating it in contexts other than originally considered.

    Phil Long, “The Role of the Learning Engineer”

    So the essential feature that Phil is calling out in design thinking is putting the idea out into the world so that everybody can see it, respond to it, and talk about it together. Now watch where he takes this:

    As learning environments are intentionally designed in digital contexts, the opportunity to instrument the learning environment emerges. Learners benefit in terms of feedback or suggested possible actions. Evaluators can assess how the course performed on a number of dimensions. The faculty and others in the learning-design team can get data through the instrumented learning behaviors, which may provide insight into how the design is working, for whom it is working, and in what context.

    Phil Long, “The Role of the Learning Engineer”

    Rather than a sketch, a wireframe, or a prototype, a learning engineer makes the graph, the dashboard, or the visualization into the externalization. For Herb Simon, as for Phil Long, these design artifacts serve the same purpose. They’re the same thing, basically.

    If you’re not a data person, this might be hard to grasp. (I’m not a data person. This is hard for me to grasp sometimes.) How can you take numbers in a table and turn them into a meaningful artifact that a group of people can look at together, discuss, make sense of, debate, and learn from? What might that even look like?

    Well, it might look something like this, for example:

    Higher ed LMS market share for US and Canada, January 2019
    Phil Hill’s famous squid diagram

    Phil Hill has a graduate degree in engineering. Not learning engineering. Electrical. (Also, he’s not a Stalinist.)

    By the way, when we externalize and share data with a student about her learning processes in a form that is designed to provoke thought and discussion, we have a particular term of art for that in education. It’s called “formative assessment.” If we do it in a way such that the student always has access to such externalizations, which are continually updating based on the student’s actions, we call that “continuous formative assessment.” When executed well, there is evidence that it can be an effective educational practice.

    Caliper statements as learning engineering artifacts

    So here’s where we’ve arrived at this point in the post:

    • Design is a process by which we tackle ill-defined problems of meeting human needs and wants, such as needing or wanting to learn something.
    • Engineering is a word that we’re not going to worry about defining precisely for now, but it relates to applying science to a design problem, and therefore often involves the measurement and numbers.
    • One important innovation in design methodology is the creation of external artifacts early in the design process so that various stakeholders with different sorts of experience and expertise can provide feedback in a social context. In other words, create something that makes the idea more “real” and therefore easier to discuss.
    • Learning engineering includes the skills of creation and manipulation of design artifacts that require more technical expertise, including expertise in data and software engineering.

    The twist with Caliper is that, rather than using visualizations and dashboards as the externalization, we can use human language. This was the original idea of behind the Semantic Web, which is still brilliant in concept, even if the original implementation was flawed. Let’s review that basic idea as implemented in Caliper:

    • You can express statements about the world (or the world-wide web) in three-word sentences of the form [subject] [verb] [direct object] e.g., [student A] [correctly answers] .
    • Because English grammar works the way it does, you can string these sentences together to form inferences, e.g., [tests knowledge of] [multiplying fractions]; therefore, [student A] [correctly answers] [a question about multiplying fractions].
    • We can define mandatory and optional details of every noun and verb e.g., it might be mandatory to know that question 13 was a multiple choice question, but it might be optional to include the actual text of the question, the correct answer, and the distractors.

    That’s it. Three-word sentences, which work the way they do in English grammar, and definitions of the “words.”

    A learning engineer could use Caliper paragraphs as a design artifact to facilitate conversations about refining the standard, the products involved, and the experimental design. I’ll share a modified version of an example I recently shared with an IMS engineer to illustrate this same point.

    Suppose you are interested in helping students become better at reflective writing. You want to do this by providing them with continuous formative assessment, i.e., in addition to the feedback that you give them as an instructor, you want to provide them an externalization of the language in their reflective writing assignments. You want to use textual analysis to help the students look at their own writing through a new lens, find the spots where they are really doing serious thought work, and also the spots where maybe they could think a little harder.

    But you have to solve a few problems in order to do give this affordance to your students. First, you have to develop the natural language analysis tool that can detect cues in the students’ writing that indicate self-reflection (or not). That’s hard enough, but the research is being conducted and progress is being made. The second problem is that you are designing a new experiment to test your latest iteration and need some sort of summative measure to test against. So maybe you design a randomized controlled trial where half the students in the class use the new feedback tool, half don’t, and all get the same human-graded final reflective writing assignment. You compare the results.

    This is an example of theory-driven learning analytics. Your theory is that student reflection improves when students become more aware of certain types of reflective language in their journaling. You think you can train a textual analysis algorithm to reliably distinguish—externalize—the kind of language that you want students to be more aware of in their writing and point it out to them. You want to test that by giving students such a tool and see if their reflective writing does, in fact, improve. Either students’ reflective writing will improve under the test condition, which will provide supporting evidence for the theory, or it won’t, which at the very least will not support the theory and might provide evidence that tends to disprove the theory, depending on the specifics. There are data science and machine learning being employed here, but they are being employed more selectively than just shotgunning an algorithm at a data set and expecting it to come up with novel insights about the mysteries of human cognition.

    Constructing theory-driven learning analytics of the sort described here is challenging enough to do in a unified system that is designed for the experiment. But now we get to the problem for which we will need the help of IMS over the next decade, which is that the various activities we need to monitor for this work often happen in different applications. Each writing assignment is in response to a reading. So the first thing you might want to do, at least for the experiment if not in the production application, is to control for students who do the reading. If they aren’t doing the reading, then their reflective writing on that reading isn’t going to tell you much. Let’s say the reading happens to take place in an ebook app. But their writing takes place in a separate notebook app. Maybe it’s whatever notebook app they normally use—Evernote, One Note, etc. Ideally, you would want them to journal in whatever they normally use for that sort of activity. And if it’s reflective writing for their own growth, it should be an app that they own and that will travel with them after they leave the class and the institution. On the other hand, the final writing assignment needs to be submittable, gradable, and maybe markable. So maybe it gets submitted through an LMS, or maybe through a specialized tool like Turnitin.

    This is an interoperability problem. But it’s a special one, because the semantics have to be preserved through all of these connections in order for (a) the researchers to conduct the study, and then (b) the formative assessment tool to have real value to the students. The people who normally write Caliper metric profiles—the technical definitions of the nouns in Caliper—would have no idea about any of this on their own. Nor would the application developers. Both groups would need to have a conversation with the researchers in order to get the clarity they need in order to define the profiles for this purpose.

    The language of Caliper could help with this if a person with the right role and expertise were facilitating the conversation. That person would start by eliciting a set of three-word sentences from the researchers. What do you need to know? The answers might include statements like the following:

    • Student A reads text 1
    • Student A writes text alpha
    • Text alpha is a learning reflection of text 1
    • Student A reads text 2
    • Text 2 is a learning reflection of texts 1 and 2
    • Etc.

    The person asking the questions of the researcher and the feature designer—let’s call that person the learning engineer—would then ask questions about the meanings and details of the words, such as the following:

    • In what system or systems is the reading activity happening?
    • Do you need to know if the student started the reading? Finished it? Anything finer grained than that?
    • What do you need to know about the student’s writing in order to perform your textual analysis? What data and metadata do you need? And how long a writing sample do you need to elicit in order to perform the kind of textual analysis you intend and get worthwhile results back?
    • What do you mean when you say that text 2 is a reflection of both text 1 and 2, and how would you make that determination?

    At some point, the data scientist and software systems engineers would join in the conversation and different concerns would start to come up, such as the following:

    • Right now, I have no way of associating Student A in the note-taking system with Student A in the reading system.
    • To do the analysis you want, you need the full text of the reflection. That’s not currently in the spec, and it has performance implications. We should discuss this.
    • The student data privacy implications are very different for an IRB-approved research study, an individual student dashboard, and an instructor- or administrator-facing dashboard. Who owns these privacy concerns and how do we expect them to be handled?

    Notice that the Caliper language has become the externalization that we manipulate socially in the design exercise. There are two aspects of Caliper that make this work: (1) the three-word sentences are linguistically generative, i.e., they can express new ideas that have never been expressed before, and (2) every human-readable expression directly maps to a machine-readable expression. These two properties together enable rich conversations among very different kinds of stakeholders to map out theory-driven analytics and the interoperability requirements that they entail.

    This is the kind of conversation by which Caliper can evolve into a standard that leads to useful insights and tools for improving learning impact. And in the early days, it will likely happen one use case at a time. Over time, the working group would learn from having enough of these conversations that design patterns would emerge, both for writing new portions of the specification itself and for the process by which the specification is modified and extended.

    Copyright Carnegie Mellon University, CC-BY