e-Literate

Present is Prologue

Category: Interoperability

  • Barriers to Coherent Digital Learning Experiences: A Full-Stack View

    Barriers to Coherent Digital Learning Experiences: A Full-Stack View

    Lately I’ve been thinking a lot about the problem of fragmented digital learning experiences. The pandemic really highlighted how digital learning journeys are fragmented across multiple tools—an LMS, maybe some tools that plug into the LMS and take the students somewhere else, maybe some digital courseware, maybe a web conferencing tool, and so on.

    This fragmentation creates a few serious problems. First, every hop a student (or educator) must make from one platform to another represents a chance to get lost. Where am I supposed to go? Do I need a separate login? What am supposed to do here? How does this thing work again? And so on. While we live in a world of phone apps where people are used to switching from one to another, the mobile app switching experiences differ in two respects. First, they are relatively well orchestrated. For example, when I launch a web conference from my calendar app, not only does it take me smoothly into the new app with one click, it also maintains a back-link to the app that got me there. Second, the workflows that seem natural are simple and atomic. When I’m hopping once from one app to another and back, I’m fine. But if I’m performing a complex work task that requires me to switch between Zoom, Slack, Google Docs, and maybe a browser, I lose my way easily and often. I spend a non-trivial amount of my day hunting for the correct browser tab in the correct browser window, to the point where, by the time I’ve found what I’m looking for, I sometimes forget why I was looking for it. If this feeling of getting lost among open windows and tabs is familiar to you, too, then you have a visceral sense of the problem in our digital learning experiences. This is the cognitive load we are placing with students who may also be struggling with unfamiliar material, learning new study skills, working in an environment with distractions (like kids and pets), and so on. Again, the pandemic has brought this problem home to all of us.

    The impact on individual learners is bad enough. But the fragmentation also prevents us from having a view into what the learners are doing. In the virtual workplace, it’s easy to lose a sense of what your coworkers are doing in a project—and what they need from you—because you keep losing track of where the Google Doc is that they’re working in. Or maybe you’ve even forgotten that doc exists because it’s completely out of sight for you. In an educator’s context, this means that you often struggle to keep track of how students are doing, and they often struggle to keep track of what they should be doing.

    When we try to understand why such a problem exists—not generically, but for a particular common workflow that feels like it should have a better solution by now—we often find ourselves going down a rabbit hole. There might be a missing technical specification. And maybe that technical specification is missing because there aren’t business drivers for software companies to coalesce around that specification. And maybe the business drivers aren’t there because the customers make purchasing decisions in a certain way. And maybe they make their purchasing decisions the way they do because of their internal organization and culture. Or laws and regulations. Or both. And so we go deeper and deeper into the rabbit hole. How many conference panels have attended that end with you shrugging your shoulders at the seemingly intractable complexity that the panelists have brought to light?

    One of the jobs I take on at e-Literate (and more generally in my professional life) is to selectively go down the rabbit holes that may actually lead somewhere other than further down. I don’t care much about technical specifications, business models, university procurement processes, faculty professional indoctrination, and so on for their own sake. I care about the educational problems that those factors influence. It’s a two-step process: (1) find the subset of hard problems that are not impossible to solve at the moment, and (2) look for an answer that is as simple as possible but no simpler. I often don’t get past step one, even with multiple lengthy posts. These are the sorts of problems that require multiple stakeholders to solve in concert. Sometimes the first step toward solving them is getting everyone to see them in roughly the same way.

    We are at a moment now where the rabbit hole of digital learning experience fragmentation may more tractable than it was before. Not easy, but perhaps possible. While there are a variety of reasons why this is true, the most obvious one is that our shared experience of pandemic life has created the widespread and direct experience of digital frustration that has engendered increased empathy and urgency. More of us get the problem because more of us have lived something like it. So deep, in fact, that I’m going to draw on blog posts that are 15 and 16 years old. I confess this is uncomfortable to me. It’s a little like taking out your journal from middle school and reading it at a party.

    EE litter it

    But 2005 – 2008 was the period when the LMS began a very important step toward solving the fragmentation problem. It’s worth taking a look back to see what’s changed, what hasn’t, and why some problems turned out to be harder than others.

    Adding windows to the walled garden

    Going back to one of the earliest posts on e-Literate, I wrote this in 2004:

    The analogy I often make with Blackboard is to a classroom where all the seats are bolted to the floor. How the room is arranged matters. If students are going to be having a class discussion, maybe you put the chairs in a circle. If they will be doing groupwork, maybe you put them in groups. If they are doing lab work, you put them around lab tables. A good room set-up can’t make a class succeed by itself, but a bad room set-up can make it fail. If there’s a loud fan drowning out conversation or if the room is so hot that it’s hard to concentrate, you will lose students.

    A good [LMS] allows for flexibility in classroom set-up. For example, when I used [a now-defunct open-source LMS]…I was able to meet with stakeholders and try different arrangements of the virtual chairs until we found one that they were comfortable with. The default set-up, which was optimised for on-campus students who took four or five courses at once and needed an aggregation portal, was completely unsuited for my audience, which was a group of full-time bond traders and other financial services people who were taking time out of their twelve-hour work days for one course. It wasn’t worth their time to learn the complex (cluttered?) interface that made sense for a full-time college student. So we streamlined. We turned off functionality that we didn’t need. We renamed pages to fit the students’ expectations. We re-arranged portlet windows on pages and page order on the interface. We did all of this in 15 minutes without any programming.

    In Blackboard, you can’t do that. Sure, you can change the way the buttons look. And you can hide a button. But that’s it. It’s essentially only trivially configurable by the instructor. I strongly believe that this has a significant impact on distance learning drop-out rates.

    Fingernails on Blackboard

    That was the state of the LMS in 2004. You could turn menu items on and off. That was the extent of customization. Over time, these products became more flexible, to the point where at least some of them—I’ll call out Brightspace here, because I’ve seen this flexibility in action fairly recently in that product—are incredibly configurable. Of course, there’s always a trade-off between configurability and ease-of-use, particularly when you’re adding configurability to an existing product with an installed base that expects it to work a certain way. But we’ve seen progress on this first problem over the past 17 years.

    This was also the period when both LMS usage was growing and we began seeing the growth of web 2.0 tools. (“Web 2.0 was a term first used in public at the O’Reilly conference in late 2004, according to Wikipedia.) So we went through a moment when LMS companies started adding half-baked blogs and terrible wikis to their products. It became instantly clear that almost nobody wanted to use these LMS-internal tools.

    But it quickly became apparent that educators wanted an increasing number of specialized tools into their pedagogical workflows. Here’s a story from a 2012 post but which actually happened in the 2005-2006 time frame:

    I often tell the story of when Beth Harris and Steven Zucker (formerly of FIT but now of Khan Academy) took me to see an image annotation tool developed by Columbia University that they were excited about. They were looking for a tool for teaching art history online. Columbia’s tool was really cool, but it was developed for a histology professor. It turns out that the way histology professors want to use and annotate images in the classroom is completely different than the way art history professors do. Some of these may not be sustainable as commercial applications and may work better as non-commercial open source. But, for example, teaching good writing is a pretty large niche application spanning multiple disciplines and should support significant commercial efforts.

    What are EdTech Entrepreneurs Good For

    This passage hints at the rabbit hole problem I referenced at the top of the post, but the main point is that a long tail of learning applications was beginning to develop to accommodate subject-specific learning journeys.

    The trouble was that instructors had no way to integrate these into the LMS, which was the main—and in many cases, only—platform available to them for creating digital learning experiences.

    I was getting increasingly involved with the Sakai LMS community around this time, in part because it seemed like a place to make progress on this problem. Here’s something I wrote in 2007, the year before IMS Global released LTI 1.0:

    Sakai could foster inter-institutional resource-sharing at the level of hosting an LMS, it could also host resource sharing at the level of individual tools. As Chuck Severence pointed out in a relatively recent blog post, there are now at least three methods for integrating tools into Sakai using web services. In addition to lowering the barrier of specialized Java skills, and indeed removing the requirement that your tool be written in Java at all, these methods open up the possibility of remotely hosted tools. Add WSRP and SAML support into the mix, and developers truly have a wide array of options for developing remotely hosted tools.

    I would like to see these techniques refined to make Sakai the absolute best [LMS] for developing remotely hosted tools. I would like to see new support models that enable system administrators to say “yes” more often when faculty members ask for specialized tools. I would like to see new economic models that make it feasible for resource-rich universities developing specialized teaching applications to share them with even the poorest of their peers in the Sakai community, rather than keeping the use of those tools confined to the few professors within the developing institution that had the money to do the work (as happens much of the time today). I would like to see the Sakai Board seek funding for developing this newer, richer kind of open educational resource. The particular strengths of the Sakai community uniquely position it to accomplish these goals.

    Four Ideas for the Future of Sakai

    If you’re technical enough to know what WSRP, SAML, and Java portlets are, you’ll know that these were pretty labor-intensive and clunky solutions to the long tail integration problem. LTI was much easier to implement and was a direct driver of the explosion of learning tools that are available today for integration into LMSs. This list shows 463 apps that can be integrated with an LMS via LTI today.

    But integration via LTI itself mostly means single sign-on and grade return. Depending on how the integration is implemented and how many integrations are used, we still can easily have the lost-in-too-many-windows-and-tabs navigation problem and the I-can’t-see-what-you’re-doing problem. We have integration but not true interoperability.

    Here’s a bit I wrote in 2005 from a speculative piece about an idea that some colleagues and I were (mis)labeling a “Learning Management Operating System (LMOS) Service Broker”:

    Now, it turns out that RSS feeds carry quite a bit of information that could be useful in a learning context. Here’s a list of just a small subset of the information available from my RSS feed, for example:

    • The URL of my blog’s home page
    • The ID of the blog’s author (in my case, it gives my email address as my ID)
    • The software application that generated the posts
    • The URL, title, description, contents, time stamp and category labels for each post

    Notice two things about this data. First, it’s very generic and could be useful information about just about any content online. Second, the post-specific information (in the last bullet point) is pretty much exactly what you need to know about any assignment that a student submits for a class.

    In order to start making use of this data in the context of an LMOS, the blog developer need only make a few relatively minor technical enhancements in order to plug into the service broker:

    1. Write an adapter that enables the RSS feed to talk to the broker. (Since RSS is a very common format, chances are good that such an adapter would already exist.)
    2. Tie the blog into the single sign-on mechanism, so that the LMOS knows that the person that owns a particular blog is also, say, a student in the Psychology 101 class.
    3. Extend the blog to be able to subscribe to category labels that are related to the groups to which the student belongs (e.g., the Psych 101 student should see a “Psych 101” category tag show up in her post category list).

    Now we’re ready for some service broker automagic. Let’s say the student decides to write a blog post on a topic related to her Psych 101 class. As she writes her post, she looks over to the category list. Because the system knows that she is a registered student in Psych 101, it automatically adds “Psych 101” to her category list. She selects the appropriate category heading(s), writes her post, and publishes. The service broker, seeing that the content is labeled “Psych 101”, announces to all the applications within the Psych 101 course environment that it has some student-created content. “Can any of you applications do anything with this student-created content?” it asks.

    It gets the following responses:

    • The class RSS aggregator responds, “Yeah, I can do something with it.” It takes the student’s post and publishes it along with those of the other students in the class.
    • The course activity tracker says, “Me too. Gimme some of that.” It notes the student ID, the time stamp, and the title and URL of the post. Using this information it adds an entry for the student’s class activity on that particular date.
    • The grade gook says, “I can also use that.” Noting that the content is generated by the weblog application, it pulls the post text and URL into the student’s row in the gradebook under the “weblog entries” heading. The instructor can now assign a grade and comment to it.

    Notice that the weblog developer didn’t have to write separate integration code for course activity tracker and grade book. The service broker was able to integrate the new application on-the-fly because the blog publishes the basic required knowledge in a standard format. All the blog developer had to do was write a connector that picks up the categories from the system and works with its single sign-on mechanism. The broker does the rest. It would be the same for any other application, too. You could, for example, use more or less the same mechanism to integrate your discussion board with the grade book and the course activity tracker.

    But wait. There’s more.

    Suppose we make one more minor enhancement. Suppose that individual applications within the course environment could publish categories to share with each other. Suppose, for example, that the grade book could publish a category corresponding to a particular assignment. Our student could select that particular assignment category for her post and the instructor would automagically have it show up in the appropriate grade book column, with the appropriate point scale and weighting, and so on. Let’s imagine, too, that you could set your discussion board to generate a forum topic for particular category (such as the assignment heading in the grade book) and generate a new thread for each post that comes in labeled with that category. Students could continue to post to their personal blogs that travel with them beyond the class, but the instructor could also create class-internal discussions based on those posts. This is all done using fairly generic mechanisms, so developers creating new applications won’t need to do anything special to integrate their new wiki, or simulation, or flux capacitor, or whatever with individual applications already in the course environment.

    But wait. There’s still more.

    Suppose that, in addition to having students publish information into the course, the service broker also let the course publish information out to the student’s personal data store (read “portfolio”). Imagine that for every content item that the student creates and owns in her personal area–blog posts, assignment drafts in her online file storage, etc.–there is also a data store to which courses could publish metadata. For example, the grade book, having recorded a grade and a comment about the student’s blog post, could push that information (along with the post’s URL as an identifier) back out to the student’s data store. Now the student has her professor’s grade and comment (in read-only format, of course), traveling with her long after the system administrator closed an archived the Psych 101 course. She can publish that information to her public e-portfolio, or not, as she pleases.

    – LMOS Services and Service Brokers, Part II

    [Editor’s Note: Sorry for the formatting glitch; the newish WordPress block system still doesn’t handle nested layout formats very easily or intuitively.]

    It’s the meaning in the middle

    The unsung hero in the example above is RSS, which stands for “Really Simple Syndication.” It carries a lot of important information about a document, including but not limited to who wrote it, when they wrote it, what topic(s) it was related to, and what the title is. That information is valuable regardless of whether the document being shared is a blog post, a journal entry, an essay, or a collaboratively authored Google Doc. (There’s nothing in the format that limits us to one author.) Atom, which is a slightly newer cousin to RSS, also supports threading. With that additional information, we could share discussion posts and pull in contextual information about the discussion threads that the posts are in.

    If we think a little bit about how we want to use the metadata, we can think about how we want to write and read to the app creating the document. We might want to supply an assignment name, a learning objective, or a group of collaborators from a class. The options become quite rich without us having to change anything about the metadata structure. It turns out that we share a lot of different types of narrative documents which share common traits. By making those traits incomprehensible to our learning platform, we can build a fundamentally more legible, navigable, and sensible learner journey with accompanying data that enables the students’ educators to act as guides in ways that are difficult in today’s digital learning environments.

    I honestly think that if we took the RSS/Atom structure and added in a representation of pedagogical intent in our curricular materials design, as I wrote about in January, we would cover quite a bit of ground in terms of creating a more seamless and supported learning journey.

    Furthermore, this kind of interoperability creates more entry points for views into that learner journey. If the data across the platforms can be tied to a coherent pedagogical pathway, and if we have structures for getting permission to see those data, then we can view the relevant parts of that journey from anywhere that make sense to view it from. If, for example, you’re in class using a clicker to check students’ understanding of a concept, you might want to be able to pull up relevant data from how they performed on related activities in the LMS, courseware, and maybe even other platforms. In this particular scenario, you’d likely be looking at class aggregate performance rather than the individual learner journey. In yet another scenario, one could imagine a department sitting down and discussing how to fine-tune the course design for a 101 class that is used by a number of different instructors. So they might want to aggregate information across the whole department to see, for example, that students struggle with one particular unit or activity across the various course sections and instructors. The group might then drill down to investigate whether the assessment questions need to be adjusted, the content needs to be enriched, whether those in-class clicker activities help improve later performance, and so on.

    I’m describing a semantic mesh of pedagogical information about a course’s design and the learner’s journey through it across application boundaries. There are no technological barriers to building such a mesh, although it requires hard work and hard thinking about preserving appropriate privacy in this new world.

    I contend that we do not have such a mesh and are not making much progress toward it because the people who think about the learning journey coherence problem don’t tend to be the same people who think about data interoperability. Two types of situations have historically driven the development of EdTech interoperability specifications:

    1. You sell a thing. I sell another thing. Our customers want our things to talk to each other, typically to make their own lives easier. For example, they want the students from your thing to get into my thing without a lot of manual effort that tends to lose them along the way. Or they want the grade from my thing to be reported back to your thing. It’s typically simple administrative stuff.
    2. Speaking of grades, we need some. Our customers use our things to give students grades. Whatever else they are doing, they must also give students grades. Or progress. Or other fundamental stuff related to grading. Not learning. Grading.

    Most technical interoperability conversations about tracking learning tend to involve a mix of people who know too little about pedagogy and people who know too much about it within too narrow a context. Both types tend to get lost in the weeds. We either talk about magical reusable learning objects, which could be anything from a picture to an hour-long module with many parts, or we debate taxonomies of assessment types or pedagogical philosophies round and round in circles.

    We can try a number of different strategies to solve this problem once we put our mind to it. Some are social, some are technological, and the best are a mindful mix of the two. But rather than making this post even longer, I’m going to stop on this point: Before we can settle on an effective method developing a semantic mesh for pedagogy, we need to collectively decide that it is an urgent problem to solve. I think people feel it viscerally from their pandemic experiences. But I don’t think we’re collectively articulating the problem yet.

    This post is my stake in the ground.

  • Announcing a Lesson-level Interoperability Standards Effort

    I’m delighted to announce a project aimed at making it easier to share interactive digital content at the lesson level as well as to establish baseline educational analytics for digital curricular materials. I’m tempted to call this a “courseware” interoperability effort, but its potential application is broader than that term would imply to some folks. For example, the work could support well-structured content in LMSs.

    This effort is consistent in philosophy with my recent “Content as Infrastructure” post series as well as the post of a version of my IMS talk on interoperability, learning analytics, and pedagogical intent. One of the main outputs of the project will be a white paper, released as an Empirical Educator Project (EEP) contribution, describing the standards proposal that is ultimately developed and its value to education. The project also dovetails very nicely with both previously announced and as-yet-unannounced EEP projects, and I’m very excited about the work.

    The idea for the work both grew out of and is funded through a grant from the U.S. Department of Education (ED) to develop “active OER.” In the course of the grant planning process, ASU professor and grant PI Ariel Anbar came to the conclusion that the grant would have a much broader impact if the content being developed were interoperable. He consulted with ED, and they agreed that interoperability would potentially increase the impact of the resources related to the grant. So a small fragment of the grant budget was carved off to test the viability of building a coalition that can make useful progress on proposed standards definitions that are both practically useful and likely to be adopted. At the moment, my work as a facilitator is the main budget expense for the project.

    Business and mission goals

    We had a kick-off meeting of a small group in late October. (More on who was in it, why they were chosen, and how we hope to expand later in this post.) Here are the notes I captured on the goals and ambitions for impact:

    • Reduce platform lock-in for any interactive courseware content, particularly interactive OER content, which will support the following:
      • Increase the quality of existing OER content by enabling the preservation of learning design
      • Increase the supply of interactive OER content by creating a clear and achievable interoperability standard for content developers
      • Increase the availability and value of OER content for value-added platform and service providers by lowering the cost of goods involved in converting the currently available “flat” OER resources into interactive lessons with effective learning design 
      • Enable educators to more easily mix and match interactive content at the lesson level
      • Enable the development of an ecosystem of non-OER content that could be licensed at the lesson level
    • Enable lesson-level, cross-platform, cross-content learning analytics which will support the following:
      • Data-based continuous improvement of learning content, regardless of its source
      • Baseline student learning analytics capabilities that will enable institutions to monitor student progress in an apples-to-apples way across lessons, products, and courses
      • Student- and instructor-facing analytics that will help them analyze how well their respective learning and teaching strategies impact outcomes
    • A vision for implementation and ecosystem development that incentivizes participation for a wide range of commercial and non-commercial value-added participants in order to:
      • Lower the barrier to adoption for courseware platforms by assuring customers that any content they develop or use will not be locked into the licensed platform
      • Lower the cost-of-goods and availability of high-quality pre-existing content for value-added OER curricular materials product and service providers
      • Enable micro-licensing models for commercial content vendors that develop high-value lesson-level content
      • Lower the barrier for non-profit organizations and consortia to create interactive content that is competitive in functionality and measurable quality with baseline student and teacher expectations for commercial courseware

    I’ll provide more of my personal take on these goals in a subsequent post, but there is consensus in the group that we should be working toward a set of goals that are good for everyone—students, instructors, institutions, and value-added content and platform providers.

    Functional and technical goals

    Consistent with the posts I linked to at the top of this one, we’re going to start by identifying questions that educators and students would want to answer about student progress, effectiveness of content design, and effectiveness of learning intervention. Our default atomic unit for this work is the “lesson.” Our starting point for identifying this set of questions will be the ones that the participating implementors have identified as ones that their users/adopters/customers want to answer, but I expect that we’ll expand from that base over the two-year life of the initial project.

    Once we know what questions we want to answer, then we will identify the metadata for the content that is needed to answer those questions. For example, which learning objective(s) does this assessment question assess? Is the assessment formative or summative? That sort of thing. No firm decisions have been made about how this would work on the technical level, but the basic idea is that the pedagogical intent of the learning design would be captured in some machine-readable form.

    Technically, we’d like to build on as much existing standards infrastructure as possible and propose developing as little new work as possible. While the project, as a piece of a larger ED grant, does not have a formal affiliation with an interoperability standards body, I am pleased to say that IMS Global and its CEO, Rob Abel, have been highly encouraging and offered technical support to the group as we think through the effort. IMS has a lot of the infrastructure that would be needed for the effort already baked into its existing specifications. It makes all the sense in the world to try to re-use or extend standards that are already developed and adopted.

    Ultimately, the group will produce a set of recommendations for interoperability standards along with a rationale for those recommendations. The hope and intention are that these recommendations will be taken up and carried forward by the appropriate bodies at the end of the project and that the participants will continue to work together on implementation.

    More on process and risk management

    Standards development is a tricky business. You want to get to an “everybody in the pool” moment, but at the same time, you can’t win everybody over by promising to boil the ocean. So we thought a lot about how to get this process rolling and balance different risks over time.

    At my suggestion, we started by inviting in just a few of the many implementors who ultimately should be at the table. Two—Carnegie Mellon University’s Open Learning Initiative and Lumen Learning—are long-time and active participants in the OER world. While this effort will be helpful to more than just OER, the primary purpose of the grant is for the development of (inter)active OER, so we wanted representatives who could speak to the needs and nuances of the OER ecosystem. The two other implementors we invited—Smart Sparrow and CogBooks—are courseware platform implementers that both work extensively with ASU already. Smart Sparrow is also playing a major role in this OER grant since Ariel has chosen its Inspark Education network to help manage the grant and is building the content on the Smart Sparrow platform. Also, CMU, Lumen, and Smart Sparrow have all been participants in EEP. In addition to the implementers and ASU, we had representatives from Scottsdale Community College and ED at the kick-off meeting.

    This is a small enough group with enough interconnections that we have a good chance of making progress on scoping goals without excessive amounts up-front diplomacy required but diverse enough that we would get different opinions and perspectives. It’s a good group for getting started and for testing the basic idea that what we want to accomplish is doable within a reasonable period of time. Ultimately, however, the project will need more and different folks to be involved if it going to result in broadly implemented interoperability standards. The starting group of four implementer participants is going to work toward a letter they all can sign onto that says they are committing in principle to implement any standards that ultimately flow out of this effort. The value in their commitment at this early stage is to decrease the risk of other implementors who may want to join but are skeptical that the effort will produce results. In parallel, the project is seeking additional funding that would enable us to support the participation of more stakeholders—educators, platform implementers, content developers, and standards committees (and possibly students as well).

    It is early days for this work. So far, the group has only met once. There is still a lot to do and a lot to be figured out. But I am hopeful that we can both develop useful recommendations for advancing interoperability standards and pioneer some new ways of working together on productive EdTech collaboration in the process.

  • Pedagogical Intent and Designing for Inquiry

    I’ve been asked by several folks to write up some version of the talk I gave at the recent IMS learning analytics summit. The focus was on how, going forward, interoperability standards will need to capture pedagogical intent if we are going to develop meaningful learning analytics.

    This isn’t a word-for-word transcription of that talk, but it does capture the gist. The subtitles and literary quotes are mostly from the original presentation. Many thanks to Rob Abel and Cary Brown for inviting me and giving me the opportunity to speak to the IMS community about this important inflection point.

    Interoperability is communication

    We are not here to curse the darkness, but to light the candle that can guide us through that darkness to a safe and sane future.

    John F. Kennedy

    Ten or fifteen years ago, when we talked about what we wanted from EdTech software interoperability, most of the time the things we wanted seemed like they ought to be simple. Mostly, we just wanted to transfer student roster and grade information from one system to another. When we got a little more ambitious, we asked for single sign-on and the ability to put a little window of one application into another one. This was foundational interoperability. It wasn’t “disruptive” or “revolutionary” or otherwise life-altering, but it was important.

    My first close encounter with an IMS interoperability effort was when I worked at Oracle in on Peoplesoft Campus Solutions team. We just wanted to be able to send a course roster to the LMS and have the LMS send a final grade for each student back. Going in, I couldn’t understand why that was hard or why it hadn’t already been done. All I knew was that (a) it was a problem that IMS had tried and failed to solve at least once before, since we were working on the revision of an existing standard, and (b) a consequence of that failure was that IT professionals on campuses everywhere had to cook up their own, often time-intensive, duct-tape-and-chewing-gum solutions to getting roster data into the LMS. As for grades, it made no sense to me that faculty had to copy grades from their electronic LMS grade book and manually re-enter them into their electronic SIS grade book. It seemed weird that this was a thing.

    It turned out that there was a translation problem. Registrars think about classes very differently than instructors and students do. For a registrar, if a student is taking a “statistics for psychology majors” course, and that course can be taken for credit in either the psychology or the math department, then “statistics for psychology” is actually two completely separate courses. And the decision of which of the two courses a student registers for may make the difference between that student meeting the requirements for graduation or not. On the other hand, the students and instructor experience “statistics for psychology” as one course meeting in one place on one schedule with one syllabus and one group of people.

    Good software reflects and supports the needs and expectations of its users. Accordingly, SIS software represented “statistics for psychology” as two courses, while LMS software wanted to create one course space for it. In order to have roster information show up as instructors and students expect it in the LMS and then return final grades as the registrar expects them in the SIS, the standards committee had to recognize that there was a translation problem and design a specification that could function as a two-way translator.

    This was an important lesson for me. Even seemingly simple interoperability challenges can be complicated because they often aren’t about the software so much as they are about the people who use the software. Interoperability design is at least partly a liberal art.

    Now, today, we would have a different possible way of solving that particular interoperability problem than the one we came up with over a decade ago. We could take a large data set of roster information exported from the SIS, both before and after the IT professionals massaged it for import into the LMS, and aim a machine learning algorithm at it. We then could use that algorithm as a translator. Could we solve such an interoperability problem this way? I think that we probably could. I would have been a weaker product manager had we done it that way, because I wouldn’t have gone through the learning experience that resulted from the conversations we had to develop the specification. As a general principle, I think we need to be wary of machine learning applications in which the machines are the only ones doing the learning. That said, we could have probably solved such a problem this way and might have been able to do it in a lot less time than it took for the humans to work it out.

    I will argue that today’s EdTech interoperability challenges are different. That if we want to design interoperability for the purposes of insight into the teaching and learning process, then we cannot simply use clever algorithms to magically draw insights from the data, like a dehumidifier extracting water from thin air. Because the water isn’t there to be extracted. The insights we seek will not be anywhere in the data unless we make a conscious effort to put them there through design of our applications. In order to get real teaching and learning insights, we need to understand the intent of the students. And in order to understand that, we need insight into the learning design. We need to understand pedagogical intent.

    That new need, in turn, will require new approaches in interoperability standards-making. As hard as the challenges of the last decade have been, the challenges of the next one are much harder. They will require different people at the table having different conversations.

    Data and communication are not the same

    O wonder!
    How many goodly creatures are there here!
    How beauteous mankind is! O brave new world
    That has such people in’t!

    William Shakespeare

    The quote above is from The Tempest. Here’s the scene: Miranda, the speaker, is a young woman who has lived her entire life on an island with nobody but her father and a strange creature who she may think of as a brother, a friend, or a pet. One day, a ship becomes grounded on the shore of the island. And out of it comes, literally, a handsome prince, followed by a collection of strange (and presumably virile) sailors. It is this sight that prompts Miranda’s exclamation.

    As with much of Shakespeare, there are multiple possible interpretations of her words, at least one of which is off-color. Miranda could be commenting on the hunka hunka manhood walking toward her.

    “How beauteous mankind is!”

    Or. She could be commenting on how her entire world has just shifted on its axis. Until that moment, she knew of only two other people in all of existence, each of who she had known her entire life and with each of whom she had a relationship that she understood so well that she took it for granted. Suddenly, there was literally a whole world of possible people and possible relationships that she had never considered before that moment.

    “O brave new world / That has such people in’t”

    So what is on Miranda’s mind when she speaks these lines? Is it lust? Wonder? Some combination of the two? Something else?

    The text alone cannot tell us. The meaning is underdetermined by the data. Only with the metadata supplied by the actor (or the reader) can we arrive at a useful interpretation. That generative ambiguity is one of the aspects of Shakespeare’s work that makes it art.

    But Miranda is a fictional character. There is no fact of the matter about what she is thinking. When we are trying to understand the mental state of a real-life human learner, then making up our own answer because the data are not dispositive is not OK. As educators, we have a moral responsibility to understand a real-life Miranda having a real-life learning experience so that we can support her on her journey.

    Intention matters in education

    With regard to moral rules, the child submits more or less completely in intention to the rules laid down for him, but these, remaining, as it were, external to the subject’s conscience, do not really transform his conduct.

    Jean Piaget

    The challenge that we face as educators is that learning, which happens completely inside the heads of the learners, is invisible. We can not observe it directly. Accordingly, there are no direct constructs that represent it in the data. This isn’t a data science problem. It’s an education problem. The learning that is or isn’t happening in the students’ heads is invisible even in a face-to-face classroom. And the indirect traces we see of it are often highly ambiguous. Did the student correctly solve the physics problem because she understands the forces involved? Because she memorized a formula and recognized a situation in which it should be applied? Because she guessed right? The instructor can’t know the answer to this question unless she has designed a series of assessments that can disambiguate the student’s internal mental state.

    In turn, if we want to find traces of the student’s learning (or lack thereof) in the data, we must understand the instructor’s pedagogical intent that motivates her learning design. What competency is the assessment question that the student answered incorrectly intended to assess? Is the question intended to be a formative assessment? Or summative? If it’s formative, is it a pre-test, where the instructor is trying to discover what the student knows before the lesson begins? Is it a check for understanding? A learn-by-doing exercise? Or maybe something that’s a little more complex to define because it’s embedded in a simulation? The answers to these questions can radically change the meaning we assign to a student’s incorrect answer to the assessment question. We can’t fully and confidently interpret what her answer means in terms of her learning progress without understanding the pedagogical intent of the assessment design.

    But it’s very easy to pretend that we understand what the students’ answers mean. I could have chosen any one of many Shakespeare quotes to open this section, but the one I picked happens to be the very one from which Aldous Huxley derived the title of his dystopian novel Brave New World. In that story, intent was flattened through drugs, peer pressure, and conditioning. It was reduced to a small set of possible reactions that were useful in running the machine of society. Miranda’s words appear in the book in a bitterly ironic fashion from the mouth of the character John, a “savage” who has grown up outside of societal conditioning.

    We can easily develop “analytics” that tell us whether students consistently answer assessment questions correctly. And we can pretend that “correct answer analytics” are equivalent to “learning analytics.” But they are not. If our educational technology is going to enable rich and authentic vision of learning rather than a dystopian reductivist parody of it, then our learning analytics must capture the nuances of pedagogical intent rather than flattening it.

    This is hard.

    Some more examples

    The lesson assessment example is easy enough to understand. (My post series on content as infrastructure explores it in more detail.) But the more one looks around at the full range of analytics that are truly aimed at supporting student success, the clearer the lesson about capturing intent becomes.

    Take, for example, summer melt. I recently just hosted an entire hour-long Standard of Proof webinar on this topic. (You should watch it. It’s good.) Here’s a slide from that webinar which illustrates the obstacles that first-generation students face in getting from their college acceptance letter to the first day of class:

    The Summer Melt Maze

    Credit: Lindsay Page

    Each of the text labels represents an obstacle that first-generation students in particular may struggle to overcome, because their circumstances are complex (e.g., no parent to provide a parental signature), because they don’t have parents or guardians who have the training and knowledge to help them, and/or because they’re seventeen-year-old kids. I don’t know about you, but I don’t think I had to navigate a single one of these obstacles without parental help, and I don’t know if I could or would have done so without it.

    Now think about developing a software solution to identify the specific barrier a student is struggling with and provide appropriate help. Could you solve the problem simply by sucking in enough data and running a machine learning algorithm? I very much doubt it. And even if you could, think about how invasive you would have to be to do so. Think about the privacy implications. The cure would be worse than the disease.

    Georgia State University took a different approach. They have invited students to share their intent. They use a chatbot. A chatbot is a conversational interface. Students can ask directly about the problems they were encountering. Once students specify their intent, then machine learning can be used to further disambiguate it. For example, the software can figure out that a student who writes “I have no money” may be asking for help obtaining financial aid. But only because there is a conversational interface, and because the software behind that interface has been programmed to anticipate and respond to a certain range of questions that a student might come to the chatbot for answers. Through a combination of the interface layer, the data layer, and the usage context, the educational intent was encoded into the system.

    Here’s another example that postdates my talk. ACT recently released a paper on detecting non-cognitive education-relevant factors like grit and curiosity through LMS activity data logs. This is a really interesting study that I hope to write more about in a separate post in the near future, but for now, I want to focus on how labor-intensive it was to conduct. First author John Whitmer, formerly of Blackboard, is one of the people in the learning analytics community who I turn to first when I need an expert to help me understand the nuances. He’s top-drawer, and he’s particularly good at squeezing blood from a stone in terms of drawing credible and useful insights from LMS data.

    Here’s what he and his colleagues had to do in order to draw blood from this particular stone:

    The online interaction features were generated from the LMS clickstream data. After manual inspection, we determined that the action field alone (e.g., “opened”) was insufficient to address our research questions and needed to be joined with the label of the item that the action was taken in reference to, which was a complex pairing. For example, course item values in the LMS data include “Week 12: Electrochemistry” or “CHEM 102 Practice Exam 4B,” which were easily interpretable from the course syllabus, while others (e.g., “2/27 CL” or “18.7 RQ”) required confirmation from the instructor. Hence, we created broader activity categories for these activity events in the LMS data using the course syllabus with confirmation from the instructor which resulted in 110 unique activity events in the LMS data that were recoded to a total of 21 activity categories as described in Table 4.

    First, they had to look at the syllabi. With human eyeballs. Then they had to interview the instructors. You know, humans having conversations with other humans. Then the humans—who had interviewed the other humans in order to annotate the syllabi that they looked at with their human eyeballs—labeled the items being accessed in the LMS with metadata that encoded the pedagogical intent of the instructors. Only after they did all that human work of understanding and encoding pedagogical intent could they usefully apply machine learning algorithms to identify patterns of intent-relevant behavior by the students.

    LMSs are often promoted as being “pedagogically neutral.” (And no, I don’t believe that Moodle is any different.) Another way of putting this is that they do not encode pedagogical intent. This means it is devilishly hard to get pedagogically meaningful learning analytics data out of them without additional encoding work of one kind or another.

    Interoperability without intent creates chaos

    If Jorge Luis Borges’ Library of Babel could have existed in reality, it would have been something like the Long Room of Trinity College.

    Christopher de Hamel

    I want to underscore the point that simply collecting more data and writing more clever algorithms will not help us find a way out of the problem of that last example. It is a problem of epistemic closure. Data and knowledge are not the same, and more data do not necessarily unlock more knowledge.

    “The Library of Babel” is a short story by the great Jorge Luis Borges. (It’s only nine pages long. You should read it.) The story describes a world that perfectly captures the nature of the problem we face:

    The universe (which others call the Library) is composed of an indefinite and perhaps infinite number of hexagonal galleries, with vast air shafts between, surrounded by very low railings. From any of the hexagons one can see, interminably, the upper and lower floors. The distribution of the galleries is invariable. Twenty shelves, five long shelves per side, cover all the sides except two; their height, which is the distance from floor to ceiling, scarcely exceeds that of a normal bookcase. One of the free sides leads to a narrow hallway which opens onto another gallery, identical to the first and to all the rest. To the left and right of the hallway there are two very small closets. In the first, one may sleep standing up; in the other, satisfy one’s fecal necessities. Also through here passes a spiral stairway, which sinks abysmally and soars upwards to remote distances….

    There are five shelves for each of the hexagon’s walls; each shelf contains thirty-five books of uniform format; each book is of four hundred and ten pages; each page, of forty lines, each line, of some eighty letters which are black in color. There are also letters on the spine of each book; these letters do not indicate or prefigure what the pages will say….

    The orthographical symbols are twenty-five in number. This finding made it possible, three hundred years ago, to formulate a general theory of the Library and solve satisfactorily the problem which no conjecture had deciphered: the formless and chaotic nature of almost all the books. One which my father saw in a hexagon on circuit fifteen ninety-four was made up of the letters MCV, perversely repeated from the first line to the last. Another (very much consulted in this area) is a mere labyrinth of letters, but the next-to-last page says Oh time thy pyramids. This much is already known: for every sensible line of straightforward statement, there are leagues of senseless cacophonies, verbal jumbles and incoherences….

    Five hundred years ago, the chief of an upper hexagon came upon a book as confusing as the others, but which had nearly two pages of homogeneous ines. He showed his find to a wandering decoder who told him the lines were written in Portuguese; others said they were Yiddish. Within a century, the language was established: a Samoyedic Lithuanian dialect of Guarani, with classical Arabian inflections. The content was also deciphered: some notions of combinative analysis, illustrated with examples of variations with unlimited repetition. These examples made it possible for a librarian of genius to discover the fundamental law of the Library. This thinker observed that all the books, no matter how diverse they might be, are made up of the same elements: the space, the period, the comma, the twenty-two letters of the alphabet. He also alleged a fact which travelers have confirmed: In the vast Library there are no two identical books. From these two incontrovertible premises he deduced that the Library is total and that its shelves register all the possible combinations of the twenty-odd orthographical symbols (a number which, though extremely vast, is not infinite): Everything: the minutely detailed history of the future, the archangels’ autobiographies, the faithful catalogues of the Library, thousands and thousands of false catalogues, the demonstration of the fallacy of those catalogues, the demonstration of the fallacy of the true catalogue, the Gnostic gospel of Basilides, the commentary on that gospel, the commentary on the commentary on that gospel, the true story of your death, the translation of every book in all languages, the interpolations of every book in all books.

    Jorge Luis Borges

    The rest of the story is a rumination on how humans might make sense of this infinite series of rooms—this “data lake,” in modern parlance—in absence of any information about the intent of its creator.

    Since the Library contains every possible book with that number of pages, lines and characters, somewhere in all these rooms must exist a book that explains exactly what it all means. So it’s a data search problem, right?

    Wrong. Because there also exist books that are extremely similar but differ in minor but critical details. And books that argue why the book with the Truth is actually false. For that matter, there are books that contain many of the exact same words in the exact same order but are written in languages in which the words mean different things. How can one tell which account is the Truth?

    One can’t.

    If we are going to make progress toward educationally useful analytics, then we must ruthlessly expunge all traces of magical thinking about data. There are fundamental limits to what the data can tell us. Even systems that are designed for pedagogical intent do not necessarily encode it in a way that is useful for interoperable analytics. In some cases, it may be encoded at the user interface layer. A courseware authoring platform may never label an assessment as “formative” or “summative” in the data because the intended distinction is obvious to the users. In other cases, the data may be encoded in an idiosyncratic manner that does not map well to other systems (either of software or of thought). In still other cases, it may be designed badly and incorrectly or misleadingly reflect the pedagogical intent. It would be relatively easy to create a data lake of Babel which we could explore infinitely in a fruitless search for meaning.

    That way lies madness.

    If we want useful educational analytics, then we cannot simply worship the data and the algorithms. The humans must do some of the learning.

    The “semantic web” is all about intent

    I love you. You are the object of my affection and the object of my sentence.

    Mignon Fogarty

    One of the triggers for my being invited to speak at IMS about learning analytics in the first place was a previous post I wrote which was (partly) on how the structure of Caliper, which is borrowed from the structure of the semantic web, supports new sorts of interoperability conversations. Since you can read that blog post, I won’t repeat the argument in detail here. But the gist is that we both have to and can boil down chains of inference that combine pedagogical intent into simple human language that educators can understand and articulate for themselves. The triple structure of the semantic web—a simple three-word sentence with a subject, a verb, and a direct object—is designed to enable non-technical humans to string thoughts and inferences together in ways that enable more technical humans to translate those inference patterns into data structures and interoperability requirements.

    Right now, IMS Caliper adopters are largely using this vernacular as just one more IT tool for largely old-school data centralization. So now they use a “lake” instead of a “warehouse.” It’s still a centralized and IT-specialized mindset which is not suited for thinking about making meaning from multiple applications, never mind talking with other humans about interpreting the intent of users working across multiple applications.

    This is the problem that must be solved over the next decade to make real progress on educational analytics. It is at least partly a liberal arts problem. And it will be at least a decade’s worth of work, though we don’t have to wait that long to see early results.

    You get to choose the world we live in

    O wonder!
    How many goodly creatures are there here!
    How beauteous mankind is! O brave new world
    That has such people in’t!

    Shakespeare or Huxley?

    So which will it be? A brave new world in which we experience continuous wonder at how beauteous mankind is, or one in which “learning” is defined by the ability to correctly answer a series of questions and earn some digital badges? The difference between these possible futures is not one in which we embrace or reject technology, or data. It’s one in which we either embrace or ignore the complexity of human learning and the reality that we must make a conscious effort to ensure that some of this complexity is encoded into the data if we are going to design analytics systems that are educationally useful. We need to elicit specific reactions from our students as expert educators and encode the pedagogical intent for eliciting those reactions along with the reactions themselves.

    .The play’s the thing!

    William Shakespeare
  • Learning Engineering: A Caliper Example

    Learning Engineering: A Caliper Example

    In my recent IMS update post, I wrote,

    [T]he nature and challenges of interoperability our sector will be facing in the next decade are fundamentally different from the ones that we faced in the last one. Up until now, we have primarily been concerned with synchronizing administration-related bits across applications. Which people are in this class? Are they students or instructors? What grades did they get on which assignments? And how much does each assignment count toward the final course grade? These challenges are hard in all the ways that are familiar to anyone who works on any sort of generic data interoperability questions. 
    But the next decade is is going to be about data interoperability as it pertains to insight. Data scientists think this is still familiar territory and are excited because it keeps them at the frontier of their own profession. But this will not be generic data science, for several reasons.

    I then asserted the following positions:

    • Because learning processes are not directly observable, blindly running machine learning algorithms against the click streams in our learning platforms will probably not teach us much about learning.
    • On the other hand, if our analytics are theory-driven, i.e., if we start with some empirically grounded hypotheses about learning processes and design our analytics to search for data that either support or disprove those hypotheses, then we might actually get somewhere.
    • Because learning analytics expressions written in the IMS Caliper standard can be readily translated into plain English, Caliper could form a basis for expressing educational hypotheses and translating them into interoperable tools for testing those hypotheses across the boundaries of tech tools and platforms.
    • The kind of Caliper-mediated conversation I imagined among learning scientists, practicing educators, data scientists, learning system designers, and others, is relevant to a term coined and still used heavily at Carnegie Mellon University—”learning engineering.”

    In this post, I’m going to explore the last two points in more detail.

    What the heck is “learning engineering”?

    The term “learning engineering” was first used by Nobel laureate and Carnegie Mellon University polymath Herbert Simon in 1966. It has been around for quite a while. But it is a term whose time as finally has come and, as such, we are seeing the usual academic turf wars over its meaning and value. On the one hand, some folks love it, embrace it, and want to apply it liberally. IEEE has an entire group devoted to defining it. As is always the case, some of this sort of enthusiasm is thoughtful, and some of it is less so. At its worst, there is a tendency for people to get tangled up in the term because it provides a certain je ne sais quoi they’ve been yearning for to describe the aspects of their jobs that they really want to be doing as change agents rather than the mundane tasks that they keep being dragged back into doing, much like the way some folks are wrapping “innovation” and “design” around themselves like a warm blanket. It’s perfectly understandable, and I think it attaches to something real in many cases, but it’s hard to say exactly what that is. And, of course, where there are enthusiasts in academia, there are critics. Again, some thoughtful, while others…less so. (Note my comment in the thread on that particularly egregious column.)

    If you want to get a clear sense of the range of possible meanings of “learning engineering” as used by people who actually think about it deeply, one good place to start would be Learning Engineering for Online Education: Theoretical Contexts and Design-Based Examples edited by Chris Dede, John Richards, and Bror Saxberg. (I am still working on getting half a day’s worth of Carnegie Mellon University video presentations on their own learning engineering work ready for posting on the web. I promise it is coming.) There are a lot of great take-aways from that anthology, one of which is that even the people who think hard about the term and work together to put together something like a coherent tome on the subject don’t fully agree on what the term means.

    And that’s really OK. Let’s just set a few boundary conditions. On the one hand, learning engineering isn’t an all-encompassing discipline and methodology that is going to make all previous roles, disciplines, and methodologies obsolete. If you are an instructional designer, or a learning designer, or a user experience designer; if you practice design thinking, or ADDIE; be not afraid. On the other hand, learning engineering is not creeping Stalinism either. Think about learning engineering, writ large, as applying data and cognitive sciences to help bring about desired learning outcomes, usually within the context of a team of colleagues with different skills all working together. That’s still pretty vague, but it’s specific enough for the current cultural moment.

    Forget about your stereotypes of engineers and their practices. Do you believe there is a place for applied science in our efforts to improve the ways in which we design and deliver our courses, or try to understand and serve our students needs and goals? If so, what would such an applied science look like? What would a person applying the science need to know? What would their role be? How would they work with other educators who have complementary expertise?

    That is the possibility space that learning engineering inhabits.

    Applied science as a design exercise

    One of the reasons that people have trouble wrapping their heads around the notion of learning engineering is that it was conceived of by very unusual mind. Some of the critiques I’ve seen online of the term position “learning engineering” in opposition to “learning design.” But as Phil Long points out in his essay in the aforementioned anthology, Herb Simon both coined the term “learning engineering” and is essentially the grandfather of design thinking:

    Design science was introduced by Buckminster Fuller in 1963, but it was Herbert Simon who is most closely associated with it and has established how we think of it today. “The Sciences of the Artificial” (Simon, 1967) distinguished the artificial, or practical sciences, from the natural sciences. Simon described design as an ill-structured problem, much like the learning environment, which involves man-made responses to the world. Design science is influenced by the limitations of human cognition unlike mathematical models. Human decision-making is further constrained by practical attributes of limited time and available information. This bounded rationality makes us prone to seek adequate as opposed to optimal solutions to problems. That is, we engage in satisficing not optimizing. Design is central to the artificial sciences: ‘Everyone designs who devises courses of action aimed at changing existing situations into desired ones.’ Natural sciences are concerned with understanding what is; design science instead asks about what should be. this distinction separates the study of the science of learning from the design of learning. Learning scientists are interested in how humans learn. Learning engineers are part of team focused on how students ought to learn.”

    Phil Long, “The Role of the Learning Engineer”

    Phil points out two important dichotomies in Simon’s thinking. The first one: is vs. ought. Natural science is about what is, while design science is about what you would like to exist. What you want to bring into being. The second dichotomy is about well structured vs. poorly structured. For Simon, “design” is a set of activities one undertakes to solve a poorly structured problem. To need or want is human, and to be human is to be messy. Understanding a human need is about understanding a messy problem. Understanding how different humans with different backgrounds and different cognitive and non-cognitive abilities learn, given a wide range of contextual variables like the teaching strategies being employed, the personal relationships between students and teacher, what else is going on in the students’ lives at the time, whether different students are coming to class well fed and well slept, and so on, is pretty much the definition of a poorly structured problem. So as far as Herb Simon is concerned, education is a design problem by definition, whether or not you choose to use the word “engineer.”

    In the next section of his article, Phil then makes a fascinating connection between the evolution of design thinking, which emerged out design science, and learning engineering. The key is in identifying the central social activity that defines design thinking:

    Design thinking represents those processes that designers use to create new designs, possible approaches to problem solutions spaces where none existed before. A problem-solving method has been derived from this and applied to human social interactions iteratively taking the designer and/or co-design participants from inspiration to ideation and then to implementation. The designer and design team may have a mental model of the solution to a proposed problem, but it is essential to externalize this representation in terms of a sketch a description of a learning design sequence, or by actual prototyping of the activities which the learner is asked to engage. [Emphasis added.] All involved can see the attributes of the proposed design solution that were not apparent in the conceptualization of it. this process of externalizing and prototyping design solutions allows it to be situated in larger and different contexts, what Donald Schon called reframing the design, situating it in contexts other than originally considered.

    Phil Long, “The Role of the Learning Engineer”

    So the essential feature that Phil is calling out in design thinking is putting the idea out into the world so that everybody can see it, respond to it, and talk about it together. Now watch where he takes this:

    As learning environments are intentionally designed in digital contexts, the opportunity to instrument the learning environment emerges. Learners benefit in terms of feedback or suggested possible actions. Evaluators can assess how the course performed on a number of dimensions. The faculty and others in the learning-design team can get data through the instrumented learning behaviors, which may provide insight into how the design is working, for whom it is working, and in what context.

    Phil Long, “The Role of the Learning Engineer”

    Rather than a sketch, a wireframe, or a prototype, a learning engineer makes the graph, the dashboard, or the visualization into the externalization. For Herb Simon, as for Phil Long, these design artifacts serve the same purpose. They’re the same thing, basically.

    If you’re not a data person, this might be hard to grasp. (I’m not a data person. This is hard for me to grasp sometimes.) How can you take numbers in a table and turn them into a meaningful artifact that a group of people can look at together, discuss, make sense of, debate, and learn from? What might that even look like?

    Well, it might look something like this, for example:

    Higher ed LMS market share for US and Canada, January 2019
    Phil Hill’s famous squid diagram

    Phil Hill has a graduate degree in engineering. Not learning engineering. Electrical. (Also, he’s not a Stalinist.)

    By the way, when we externalize and share data with a student about her learning processes in a form that is designed to provoke thought and discussion, we have a particular term of art for that in education. It’s called “formative assessment.” If we do it in a way such that the student always has access to such externalizations, which are continually updating based on the student’s actions, we call that “continuous formative assessment.” When executed well, there is evidence that it can be an effective educational practice.

    Caliper statements as learning engineering artifacts

    So here’s where we’ve arrived at this point in the post:

    • Design is a process by which we tackle ill-defined problems of meeting human needs and wants, such as needing or wanting to learn something.
    • Engineering is a word that we’re not going to worry about defining precisely for now, but it relates to applying science to a design problem, and therefore often involves the measurement and numbers.
    • One important innovation in design methodology is the creation of external artifacts early in the design process so that various stakeholders with different sorts of experience and expertise can provide feedback in a social context. In other words, create something that makes the idea more “real” and therefore easier to discuss.
    • Learning engineering includes the skills of creation and manipulation of design artifacts that require more technical expertise, including expertise in data and software engineering.

    The twist with Caliper is that, rather than using visualizations and dashboards as the externalization, we can use human language. This was the original idea of behind the Semantic Web, which is still brilliant in concept, even if the original implementation was flawed. Let’s review that basic idea as implemented in Caliper:

    • You can express statements about the world (or the world-wide web) in three-word sentences of the form [subject] [verb] [direct object] e.g., [student A] [correctly answers] .
    • Because English grammar works the way it does, you can string these sentences together to form inferences, e.g., [tests knowledge of] [multiplying fractions]; therefore, [student A] [correctly answers] [a question about multiplying fractions].
    • We can define mandatory and optional details of every noun and verb e.g., it might be mandatory to know that question 13 was a multiple choice question, but it might be optional to include the actual text of the question, the correct answer, and the distractors.

    That’s it. Three-word sentences, which work the way they do in English grammar, and definitions of the “words.”

    A learning engineer could use Caliper paragraphs as a design artifact to facilitate conversations about refining the standard, the products involved, and the experimental design. I’ll share a modified version of an example I recently shared with an IMS engineer to illustrate this same point.

    Suppose you are interested in helping students become better at reflective writing. You want to do this by providing them with continuous formative assessment, i.e., in addition to the feedback that you give them as an instructor, you want to provide them an externalization of the language in their reflective writing assignments. You want to use textual analysis to help the students look at their own writing through a new lens, find the spots where they are really doing serious thought work, and also the spots where maybe they could think a little harder.

    But you have to solve a few problems in order to do give this affordance to your students. First, you have to develop the natural language analysis tool that can detect cues in the students’ writing that indicate self-reflection (or not). That’s hard enough, but the research is being conducted and progress is being made. The second problem is that you are designing a new experiment to test your latest iteration and need some sort of summative measure to test against. So maybe you design a randomized controlled trial where half the students in the class use the new feedback tool, half don’t, and all get the same human-graded final reflective writing assignment. You compare the results.

    This is an example of theory-driven learning analytics. Your theory is that student reflection improves when students become more aware of certain types of reflective language in their journaling. You think you can train a textual analysis algorithm to reliably distinguish—externalize—the kind of language that you want students to be more aware of in their writing and point it out to them. You want to test that by giving students such a tool and see if their reflective writing does, in fact, improve. Either students’ reflective writing will improve under the test condition, which will provide supporting evidence for the theory, or it won’t, which at the very least will not support the theory and might provide evidence that tends to disprove the theory, depending on the specifics. There are data science and machine learning being employed here, but they are being employed more selectively than just shotgunning an algorithm at a data set and expecting it to come up with novel insights about the mysteries of human cognition.

    Constructing theory-driven learning analytics of the sort described here is challenging enough to do in a unified system that is designed for the experiment. But now we get to the problem for which we will need the help of IMS over the next decade, which is that the various activities we need to monitor for this work often happen in different applications. Each writing assignment is in response to a reading. So the first thing you might want to do, at least for the experiment if not in the production application, is to control for students who do the reading. If they aren’t doing the reading, then their reflective writing on that reading isn’t going to tell you much. Let’s say the reading happens to take place in an ebook app. But their writing takes place in a separate notebook app. Maybe it’s whatever notebook app they normally use—Evernote, One Note, etc. Ideally, you would want them to journal in whatever they normally use for that sort of activity. And if it’s reflective writing for their own growth, it should be an app that they own and that will travel with them after they leave the class and the institution. On the other hand, the final writing assignment needs to be submittable, gradable, and maybe markable. So maybe it gets submitted through an LMS, or maybe through a specialized tool like Turnitin.

    This is an interoperability problem. But it’s a special one, because the semantics have to be preserved through all of these connections in order for (a) the researchers to conduct the study, and then (b) the formative assessment tool to have real value to the students. The people who normally write Caliper metric profiles—the technical definitions of the nouns in Caliper—would have no idea about any of this on their own. Nor would the application developers. Both groups would need to have a conversation with the researchers in order to get the clarity they need in order to define the profiles for this purpose.

    The language of Caliper could help with this if a person with the right role and expertise were facilitating the conversation. That person would start by eliciting a set of three-word sentences from the researchers. What do you need to know? The answers might include statements like the following:

    • Student A reads text 1
    • Student A writes text alpha
    • Text alpha is a learning reflection of text 1
    • Student A reads text 2
    • Text 2 is a learning reflection of texts 1 and 2
    • Etc.

    The person asking the questions of the researcher and the feature designer—let’s call that person the learning engineer—would then ask questions about the meanings and details of the words, such as the following:

    • In what system or systems is the reading activity happening?
    • Do you need to know if the student started the reading? Finished it? Anything finer grained than that?
    • What do you need to know about the student’s writing in order to perform your textual analysis? What data and metadata do you need? And how long a writing sample do you need to elicit in order to perform the kind of textual analysis you intend and get worthwhile results back?
    • What do you mean when you say that text 2 is a reflection of both text 1 and 2, and how would you make that determination?

    At some point, the data scientist and software systems engineers would join in the conversation and different concerns would start to come up, such as the following:

    • Right now, I have no way of associating Student A in the note-taking system with Student A in the reading system.
    • To do the analysis you want, you need the full text of the reflection. That’s not currently in the spec, and it has performance implications. We should discuss this.
    • The student data privacy implications are very different for an IRB-approved research study, an individual student dashboard, and an instructor- or administrator-facing dashboard. Who owns these privacy concerns and how do we expect them to be handled?

    Notice that the Caliper language has become the externalization that we manipulate socially in the design exercise. There are two aspects of Caliper that make this work: (1) the three-word sentences are linguistically generative, i.e., they can express new ideas that have never been expressed before, and (2) every human-readable expression directly maps to a machine-readable expression. These two properties together enable rich conversations among very different kinds of stakeholders to map out theory-driven analytics and the interoperability requirements that they entail.

    This is the kind of conversation by which Caliper can evolve into a standard that leads to useful insights and tools for improving learning impact. And in the early days, it will likely happen one use case at a time. Over time, the working group would learn from having enough of these conversations that design patterns would emerge, both for writing new portions of the specification itself and for the process by which the specification is modified and extended.

    Copyright Carnegie Mellon University, CC-BY
  • The IMS at an Inflection Point

    The IMS at an Inflection Point

    A few weeks back, I had the pleasure of attending the IMS Learning Impact Leadership Institute (LILI). For those of you who aren’t familiar with it, IMS is the major learning application technical interoperability organization for higher education and K12 (and is making some forays into the corporate training and development world as well). They’re behind specifications like LIS, which lets your registrar software automagically populate your LMS course shell with students, and LTI, which lets you plug in many different learning applications. (I’ll have a lot more to say about LTI later in this post.)

    While you may not pay much attention to them if you aren’t a technical person, they have been and will continue to be vital to creating the kind of infrastructure necessary to support more and better teaching and learning affordances in our educational technology. As I’ll describe in this post, I think the nature of that role is likely to evolve somewhat as the interoperability needs of the sector are beginning to evolve.

    The IMS is very healthy

    I’m happy to report that the IMS appears to be thriving by any obvious measure. The conference was well attended. It attracted a remarkably diverse group of people for an event hosted by an organization that could easily be perceived as techie-only. Furthermore, the attendees seemed very engaged and the discussions were lively.

    On more objective measures, the organization’s annual report bears out this impression of strong engagement. They have strong international representation across a range of organization types.

    From the IMS Global 2018 Annual Report

    Whether your measure is membership, product certifications, or financial health, the IMS is setting records.

    From the IMS Global 2018 Annual Report

    This state of affairs is even more remarkable given that, 13 years ago, there was some question as to whether the IMS was financially sustainable.

    From the IMS Global 2018 Annual Report

    If you look carefully at this graph, you’ll see three distinct periods of improvement: 2005-2008, 2009-2013, and 2013-2018. Based on what I know about the state of the organization at the time, first period can most plausibly be attributed to immediate changes implemented by Rob Abel, who took over the reins of the organization in February of 2006 and likely saved it from extinction. Likewise, the magnitude of growth in the second period is consistent with that of a healthy membership organization that has been put back on track.

    But that third period is different. That’s not normal growth. That’s hockey stick growth.

    I am not a San Franciscan. By and large, I do not believe in heroic entrepreneur geniuses who change the world through sheer force of will. Whenever I see that kind of an upward trend, I look for a systemic change that enabled a leader or organization—through insight, luck, or both—to catch an updraft.

    There is no doubt in my mind that the IMS has capitalized on some major updrafts over the last decade. That is an observation, not a criticism. That said, the winds are changing, in part because the IMS has helped move the sector through an important period of evolution and is now helping to usher in the next one. That will raise some new challenges that the IMS is certainly healthy enough to take on but will likely require them to develop a few new tricks.

    The world of 2005

    In the first year of the chart above, when the IMS was in danger of dying, there was very little in the way ed tech to interoperate. There were LMSs and registrar systems (a.k.a. SISs). Those were the two main systems that had to talk to each other. And they did, after a fashion. There was an IMS standard at the time, but it wasn’t a very good one. The result was that, even with the standard, there was a person in each college or university IT department whose job it was to manage the integration process, keep it running, fix it when it broke, and so on. This was not an occasional tweak, but a continual effort that ran from the first day of class registration through the last day of add/drop. If you picture an old-timey railroad engineer shoveling coal into the engine to keep it running and checking the pressure gauge every ten minutes to make sure it didn’t blow up, you wouldn’t be too far off. As for reporting final grades from the LMS’s electronic grade book automatically to the SIS’s electronic final grade record, well, forget it.

    If you ignore some of the older content-oriented specifications, like QTI for test questions and Common Cartridge for importing static course content, then that was pretty much it in terms of application-to-application interoperability. Once you were inside the LMS, it was basically a bare-bones box with not much you could add. Today, the IMS lists 276 officially certified products that one can plug into any LMS (or other LTI-compliant consumer), from Academic ASAP to Xinics Commons. I am certain that is a substantial undercount of the number of LTI-compatible applications, since not all compatible product makers get officially certified. In 2005, there were zero, because LTI didn’t exist. There were LMS-specific extensions. Blackboard, for example, had Building Blocks. But with a few exceptions, most weren’t very elaborate or interesting.

    My personal experience at the time was working at SUNY Systems Administration and running a search committee for an LMS that could be centrally hosted—preferably on a single instance—and potentially support all 64 campuses. For those who aren’t familiar with it, SUNY is a highly diverse system, with everything from rural (and urban) community colleges to R1s to everything in between, with some specialty schools thrown into the mix like the Fashion Institute of Technology, a medical school or two, an ophthalmology school, and so on. Both the pedagogical needs and the on-campus support capabilities across the system were (and presumably still are) incredibly diverse. There simply was not any existing LMS at the time, with or without proprietary extensions, that could meet such a diverse set of needs across the system. We saw no signs that this state of affairs was changing at pace that was visible to the naked eye, and relatively few signs that it was even widely recognized as a problem.

    To be honest, I came to the realization of the need fairly slowly myself, one conversation at a time. A couple of art history professors dragged me excitedly to Columbia University to see an open source image annotation tool, only to be disappointed when they discovered that the tool was developed to teach clinical histology, which uses image annotation to teach in an entirely different way than is typically employed in art history classes. An astronomy professor at a community college on the far tip of Long Island, where there was relatively little light pollution, wanted to give every astronomy student in SUNY remote access to his telescope if only we could figure out how to get it to talk to the LMS. Anyone who has either taught a been an instructional designer for a few wildly different subjects has a leg up on this insight (and I had done both), but even so, there are levels of understanding. The art history/histology thing definitely took me by surprise.

    A colleague and I, in an effort to raise awareness about the problem, wrote an article about the need for “tinkerable” learning environments in eLearn Magazine. But there were very few models at the time, even in the consumer world. The first iPhone wasn’t released until 2007. The first practically usable iPhone wasn’t released until 2008. (And we now know that even Steve Jobs was secretly skeptical that apps on a phone were a good idea.) It is a sign of just how impoverished our world of examples was in January of 2006 that the best we could think of to show what a world of learning apps could be like was Google Maps:

    There are several different ways that software can be designed for extensibility. One of the most common is for developers to provide a set of application programming interfaces, or APIs, which other developers can use to hook into their own software. For example, Blackboard provides a set of APIs for building extensions that they call “Building Blocks.” The company lists about 70 such blocks that have been developed for Blackboard 6 over the several years that the product version has been in existence. That sounds like a lot, doesn’t it? On the other hand, in the first five months after Google made the APIs available for Google Maps, at least ten times that many extensions have been created for the new tool. Google doesn’t formally track the number of extensions that people create using their APIs, but Mike Pegg, author of the Google Maps Mania weblog, estimates that 800-900 English-language extensions, or “mash-ups,” with a “usable, polished Google Maps implementation” have been developed during that time—with a growth rate continuing at about 1,000 new applications being developed every six months. According to Pegg, “There are about five sites out there that facilitate users to create a map by taking out an account. These sites include wayfaring.com, communitywalk.com, mapbuilder.net—each of these sites probably has hundreds of maps for which just one key has been registered at Google.” (Google requires people who are extending their application to register for free software “keys.” Perhaps for this reason, Chris DiBona, Google’s own Open Source Program Manager, has heard estimates that are much higher. “I’ve seen speculation that there are hundreds or thousands,” says DiBona, noting that estimates can vary widely depending on how you count.

    Nevertheless, even the most conservative estimate of Google Maps mash-ups is higher than the total number of extensions that exist for any mainstream LMS by an order of magnitude.

    There seemed little hope for this kind of growth any time in the foreseeable future. By early 2007, having failed to convince SUNY to use its institutional weight to push interoperability forward, I had a new job working at Oracle and was representing them on a specification development committee at the IMS. It was hard, which I didn’t mind, but it was also depressing. There was little incentive for the small number of LMS and SIS vendors who dominated specification development at that time to do anything ambitious. To the contrary, the market was so anemic that the dominant vendors had every reason to maintain their dominance by resisting interoperability. Every step forward represented an internal battle within those companies between the obvious benefit of a competitive moat and the less obvious enlightened self-interest of doing something good for customers. This is simply not the kind of environment in which interoperability standards grow and thrive.

    And yet, despite the fact that it certainly didn’t feel like it, change was in the air.

    Glaciers are slow, but they reshape the planet

    For starters, there was the LMS, which was both a change agent in of itself and an indicator of deeper changes in the institutions that were adopting them. EDUCAUSE data shows that the US LMS market became saturated some time roughly around 2003. At that time, Blackboard and WebCT had the major leads as #1 and #2, respectively. The dynamic for the next 10 years was a seesaw, with new competitors rising and Blackboard buying and killing them off as fast as it could. Take a look at the period between 2003 and 2013 in Phil’s squid graph: ((By the way, if you haven’t subscribed to Phil’s new blog yet, then you really, really should. Like, right now. I’ll wait.))

    It was absolutely vicious.

    None of this would materially affect the standards making process inside the IMS until, first, Blackboard’s practice of continually buying up market share eventually failed (thus allowing an actual market with actual market pressures to form) and, second, until the management team that came up with this decidedly anti-competitive strategy…er…chose to spend more time with their respective families. (I’ll have more to say about Heckle and Jeckle and their lasting impact on market perceptions in a future post.)

    But the important dynamic during this period is that customers kept trying to leave Blackboard (even if they found themselves being reacquired shortly thereafter) and other companies kept trying to provide better alternatives. So even though we didn’t have a functioning, competitive market that could incentivize interoperability, and even though it certainly didn’t feel like we had one, some of the preconditions for one were being established.

    Meanwhile online education growth was being driven by no fewer than three different vectors. First, for-profit providers were hitting their stride. By 2005, the University of Phoenix alone was at over 400,000 enrollments. Second, public access-oriented institutions, many of which had been seeded a decade earlier with grants from the Sloane Foundation, were starting to show impressive growth as well. A couple were getting particular attention. UMUC, for example, may not have had over 400,000 online enrollments in 2005, but they had well over 40,000, which is enough to get the attention of anyone in charge of an access-oriented public university’s budget. More quietly, many smaller schools were having online success that were proportional to their sizes and missions. For example, when I arrived at SUNY in 2005, they had a handful of community colleges that had self-sustaining online degree programs that supported both the missions and the budget of the campuses. Many more were offering individual courses and partial degrees in order to increase access for students. (Most of New York is rural, after all.)

    The third driver of online education, which is more tightly intertwined with the first two than most people realize, is that Online Program Management companies (OPMs) were taking off. The early pioneers, like Deltak (now Wiley Education Services), Embanet, Compass Education (now both subsumed into Pearson), and Orbis (recently acquired by Grand Canyon University) had proved out the model. The second wave was coming. Academic Partnerships and 2Tor (now 2U) were both founded in 2008. Altius Education came in 2009. In 2010, Learning House (now also owned by Wiley) was founded.

    Counting online enrollments is a notoriously slippery business, but this chart from the Babson survey is highly suggestive and accurate enough for our purpose:

    If you’re a campus leader and thirty percent of your students are taking at least one online class, that becomes hard for you to ignore. Uptime becomes far more important. Quality of user experience becomes far more important. Educational affordances become far more important. Obviously, thirty percent is an average, and one that is highly unevenly distributed across segments. But it’s significant enough to be market-changing.

    And the market did change. In a number of ways, the biggest one being that it became an actual, functioning market (or at least as close to one as we’ve gotten in this space).

    When glaciers recede

    Let’s revisit that second growth period in the IMS graph—2008 to 2013—and talk about what was happening in the world during that period. For starters, online continued its rocket ride. The for-profits peaked in 2010 at roughly 2 million enrollments (before beginning their spectacular downward spiral shortly thereafter). Not-for-profits (and odd mostly-not hybrids) ramped up the competition. ASU launched its first online 4-year degree in 2006. SNHU started a new online unit in 2009. WGU expanded into Indiana in 2010, which was the same year that Embanet merged with Compass Knowledge and was promptly bought by Pearson. (Wiley acquired Deltak two years later.)

    Once again, the more online students you have, the less you are able to tolerate downtime, a poor user interface that drives down productivity, or generic course shells that make it hard to teach students what they need to learn in the ways in which they need to learn. Instructure was founded in 2008. They emphasized a few distinctions from their competitors out of the gate. The first was their native multitentant cloud architecture. Reduced downtime? Check. The second was a strong emphasis on usability. The big feature that they touted which was their early runaway hit was Speed Grader. Increased productivity? Check.

    Instructure had found their updraft to give them their hockey stick growth.

    But they also emphasized that they were going to be a learning platform. They weren’t going to build out every tool imaginable. Instead, they were going build a platform and encourage others to build the specialized the tools that teachers and students need. And they would aggressively encourage the development and usage of standards to do so. On the one hand, this fit from a cultural perspective. Instructure was more like a Silicon Valley company than its competitors, and platforms were hot in the Valley. On the other hand, it was still a little weird for the education space. There still weren’t good interoperability standards for what they wanted to do. There still hadn’t been an explosion of good learning tools. This is one of those situations where it’s hard to tell how much of their success was prescience and how much of it was luck that higher ed caught up with their cultural inclination at that exact moment.

    Co-evolution

    The very same year that Brian Whitmer and Devlin Daley founded Instructure, Chuck Severence and Mark Alier were mentoring Jordi Piguillem on a Google Summer of Code project that would become the initial implementation of LTI. In 2010, the same year that Instructure scored its first major win with the Utah Education Network, IMS Global released the final specification for LTI v1.0. All this time that the market had felt like it had been standing still, it had actually been iterating. We just hadn’t been experiencing the benefits of it. Chuck, who had been thinking about interoperability in part through his work on Sakai, had been tinkering. Students like Brian and Devlin, who had been frustrated with their LMS, had been tinkering. The IMS, which actually had a precursor specification before LTI, had been tinkering. While conditions hadn’t become visible on the surface of the glacier, way down, a mile below, the topology of the land was changing.

    Meanwhile in Arizona, in 2009, the very first ASU+GSV summit was held. I admit that I have had writer’s block regarding this particular conference the last few years. It has gotten so big that it’s hard to know how to think about it, much less how to sum it up. In 2009, it was an idea. What if a university and a company that facilitates start-ups (in multiple ways) got together to encourage ed tech companies to work more effectively with universities? That’s my retrospective interpretation of the original vision. I wasn’t at many of those early conferences and I certainly wasn’t an insider. It was hard for me, with my particular background, to know what to make of it then and even harder now.

    But something clicked for me this year when it turned out that IMS LILI was held at the same hotel that the ASU+GSV summit had been at a couple of months earlier. How does the IMS get to 523 product certifications and $8 million in the bank? A lot of things have to go right for that to happen, but for starters, there have to be 523 products to certify and lots of companies that can afford to pay certification fees. That economy simply did not exist in 2008. Without it, there would be no updraft to ride and consequently no hockey stick growth. ASU+GSV’s phenomenal growth, and the ecosystem that it enabled, was another major factor influenced what I saw at IMS LILI this month.

    There is a lot of chicken-and-egg here. LTI made a lot of this possible, and the success LTI (and IMS Global) have experienced would not have been possible without a lot of this. The harder you stare at the picture, the more complicated it looks. This is what “systems thinking” is all about. There isn’t a linear cause-and-effect story. There are multiple interacting feedback loops. It’s a complex adaptive system, which means that it doesn’t respond in linear or predictable ways.

    Update: I got a note from Rob Abel noting that a lot of the growth in the last leg came from an explosion of participation in the K12 space. That’s good color and consistent with what I’ve seen in my last couple of LILI conference visits. It’s also consistent with the rest of this analysis. K12 benefitted from all of the dynamics above—the maturation of the LMS market, the dynamics in higher education online that pushed toward SaaS and usability, the massive influx of venture funding, and so on. All of those developments, plus the work inside IMS, made the K12 growth possible, while the dynamics inside K12 added another feedback loop to this complex adaptive system.

    But respond it finally did. We have some semblance of a functioning market, and with its rise, blockers preventing the formation of a vibrant interoperability standards ecosystem of the type we have today have largely fallen. Now we have to address the blockers of the formation of the vibrant interoperability ecosystem that we will need tomorrow. Because it will be qualitatively different. Tomorrow’s blockers are not market formation problems but rather collaboration methodology problems. They are about creating meaningful learning learning analytics, which will require solving some wicked problems that can only be tackled through close and well structured interdisciplinary work. That most definitely includes the standards design process itself.

    After the glacier comes the flood

    What I saw at the IMS LILI this year was, I think, a milestone. The end of an era. Market pressures now favor interoperability. The same companies that were the most resistant to developing and implementing useful interoperability standards in 2007 are among the most aggressive champions of interoperability today. This is not to say that foundational interoperability work is “over.” Far from it. Rather, the conditions finally exist where it can move forward as it should, still hard but relatively unimpeded by the distortions of a dysfunctional market.

    That said, the nature and challenges of interoperability our sector will be facing in the next decade are fundamentally different from the ones that we faced in the last one. Up until now, we have primarily been concerned with synchronizing administration-related bits across applications. Which people are in this class? Are they students or instructors? What grades did they get on which assignments? And how much does each assignment count toward the final course grade? These challenges are hard in all the ways that are familiar to anyone who works on any sort of generic data interoperability questions.

    But the next decade is is going to be about data interoperability as it pertains to insight. Data scientists think this is still familiar territory and are excited because it keeps them at the frontier of their own profession. But this will not be generic data science, for several reasons. (I will tell you right now that some of them disagree with me on this. Vehemently.) First, even in the most richly instrumented fully online environments that we have today, they are highly data impoverished relative to what we need to make good inferences about teaching and learning. For heaven’s sake, Amazon still recommends things that I have already bought. If I just bought a toaster oven last month, then how likely is it that I want to buy another one now? And I buy everything on Amazon. If they don’t know enough to make good buying recommendations on consumer products, then there’s no way that our learning environments are going to have enough data to make judgements that are orders of magnitude more sophisticated.

    Well then, some answer, we’ll just collect more data! More more more! We’ll collect everything! If we collect every bit of data, then we can answer any question. (That is a pretty close paraphrase of what one of the IMS presenters said in one of the handful of learning analytics talks I went to.)

    No. You won’t collect “everything”—even if we ignore the obvious, glaring ethical questions—because you don’t know what “everything” is. Computer folks, having finally freed themselves from the shackles of SQL queries and data marts, are understandably excited to apply that newfound freedom to the important problem space of learning. But it is not a good fit, because we don’t have a good understanding of the basic cognitive processes involved in learning. As I wrote about (at length) in a previous post, we have to employ multiple cutting-edge machine learning techniques just to get glimpses of learning processes even when we are directly monitoring students’ brain activity because these are extraordinarily complex processes with multiple hidden variables. Trying to tease out learning processes inside a student’s head based on learning analytics from running machine learning algorithms on LMS data is a little like trying to monitor the digestive processes of a flatworm on the bottom of the Marianas Trench based on studying the wave patterns on the surface of the ocean. There are too many invisible mediating layers to just run a random forest algorithm on your data lake—it all sounds very organic, doesn’t it?—and pop out new insights about how students learn.

    That doesn’t mean we should just throw up our hands, by any means. To the contrary, IMS Global has some extraordinarily good tools close at hand for tackling this problem. But it does mean that they are going to have to take some of the stakeholder engagement strategies they’ve been working at diligently to the next level, to the point where the standards-making process itself may evolve over time.

    Theory-driven interoperability

    There is an excellent data and processing resource that the learning analytics folks have yet to think deeply about how to leverage, as far as I can tell from the conference. The computational power is impressive (and impressively parallel). It is the collective intelligence of educators and learning scientists. Because there are too many confounds to making useful direct inferences from the data, educational inferencing needs to be theory-driven. You need to start with at least some idea of what might be going on inside the learner’s head. One that can be either supported or disproven based on evidence. And you need to know what that evidence might look like. If you can spell all that out, then you can start doing interesting things with learning analytics, including machine learning. There is room for learning science, data science, and on-the-ground teaching expertise at the table. In fact, you need all those kinds of expertise. But the folks with those respective kinds of know-how need to be able to talk to each other and work together in the right ways, which is really hard.

    The IMS has an outstanding foundation for this sort of work, because their Caliper specification turns out to provide the basis for a perfectly lovely lingua franca. To begin with, its fundamental structure is triples, which is the same basic idea as the original concept behind the semantic web. If you’re not a computer person and this is starting to make your eye’s glaze over, don’t worry, because this is plain English. Three-word sentences, in fact. Noun, verb, direct object. Student takes test. Question assesses learning objective. Student highlights sentence. Sentence discusses Impressionism.

    IMS Caliper expresses learning analytics in statements that can easily be translated into three-word plain-English sentences. These sentences can be strung together into coherent paragraphs. Notice, for example, how the last two example sentences are related. Three-word sentences in this format can be chained together to form longer thoughts. New thoughts. With this one, very simple grammatical structure, we have a language that is generative in the linguistic sense. As long as you have words to put into these grammatical placeholders, you can string thoughts together. Or “chain inferences,” to sling the lingo. And it turns out, unsurprisingly, that Caliper has a mechanism for defining these words in ways that both humans and machines can understand them.

    That has to be the bridge. Humans have to understand the utterances well enough to be able express their theories on the front end and understand whatever the machine is telling them it may have learned on the back end. Machines have to understand them specifically enough to be able to parse the sentences in their own, literal, machine-y way. Theoretically, Caliper could be an ideal language to enable educators and computer scientists to discuss theories about how to better support students as well as how to test those theories together.

    The challenge is that the IMS community, at least based on what I saw in the sessions I attended, is not using the specification as an interdisciplinary communication tool in this way yet. What I saw happening instead was a lot of very earnest data scientists pumping as much Caliper data as the can into their data lakes. They come to the conference, give a talk and, to their credit, shrug their shoulders and admit that they really don’t know what to do with those data yet. But then they go home and build bigger pipes, because that’s their job. That’s what they do.

    It’s not their fault. I’ve been friends with some of these folks for a very long time indeed. There are good people here. But if you work in the IT department, and you’re not a learning scientist or a classroom educator, and the faculty are somewhere between dismissive and disdainful of the idea of talking to you about working together to improve teaching and learning, then what can you do? You do what you know how to do and hope that things will change for the better over time.

    It’s not the IMS’s fault either. The conference I attended was called the IMS Learning Impact Leadership Institute. That’s not a new name. Caliper has board that helps guide its direction. That board includes educators who are the kind of advocates that I would like to see on such a body. They are productive irritants in the best possible way. But that’s not enough anymore. This is just a really hard problem. It’s the challenge of the next decade. To meet it, we need to do more than just make sure the right people are in the room together. We need to develop new ways of working together. New roles, methodologies, ways of talking with each other, and ways of seeing the world.

    I’m going to preview a bit of a post that I have in my queue for…I’m not sure when, but some time soon…by mentioning “learning engineering.” This term has gotten a lot of buzz lately, along with some criticism. I’ll be writing up my own take on it, but for now I’ll say that one reason I think the term is gaining some currency is that it represents a set of skills for being a mediator in the kind of collaboration that I’m describing here.

    As it turns out, it was coined by Nobel prize-winning polymath and Carnegie Mellon luminary Herb Simon, after whom Carnegie Mellon University’s Simon Initiative was named. And, as it also turns out, the Simon Initiative hosted this year’s EEP summit and made some news in the process by contributing $100 million worth of open source software that they use in their research and pratice of…wait for it…learning engineering.

    Here’s a slide that they used in their talk explaining what the heck learning engineering is and what they are doing when they are doing it:

    Copyright Carnegie Mellon University, CC-BY

    (By the way, the videos of all talks from the summit will be posted online, as promised. Please be patient a little longer.)

    This post has already run long, so rather than unpacking the slide, I’ll leave you with a question or two. Think about this graphic as representing a data-informed continuous improvement methodology involving multiple people with multiple types of expertise. What would that methodology need to look like? Who would have to be at the table, what kinds of conversations would they have to have, and how would they have to work together?

    I’m not suggesting that “learning engineering” is a magical conjuring phrase. But I am suggesting that we need new approaches, new competencies, and likely a new role or two if we are going to get to the next updraft.

  • How and Why the IMS Failed with LTI 2.0

    How and Why the IMS Failed with LTI 2.0

    While at EDUCAUSE last week, we heard from several sources that the IMS is walking away from LTI 2.0. I’m not sure what term the organization is using to characterize what they’re doing; there’s nothing on the public web site that I can find at the moment. But the reality is that they are no longer encouraging the adoption of LTI 2.0 and are actively investing their energy in developing the 1.x branch, starting with something that they’re calling “LTI Advantage.” (More on that in a bit.)

    To be clear, I believe this is a good decision, and I also believe the fact that they felt the need to do so is not particularly scandalous. Everybody makes mistakes. The important thing is to learn from them. So, with that in mind, and with the IMS quarterly meeting coming up this week, I thought it might be useful to write a post-mortem that could potentially be of use going forward.

    Interoperability Standards as Multilateral Trade Deals

    One of the biggest problems the IMS faces is that a lot of the participants on the specification committees don’t fully understand the big picture of how standards adoption works in the real world. They are mostly technologists with a sprinkling of product managers. They usually have the best of intentions. But they do not have the right roles to see the business ramifications of spec development clearly, and they very often do not get adequate guidance from the stakeholders in their respective organizations that do have the right roles.

    First and foremost, interoperability specifications are a lot like multilateral trade agreements. They are agreements that everybody will operate according to certain rules under certain circumstances. The fact that these agreements are written in engineering language and implemented in software code doesn’t change the fact that they are multi-party treaties. One of the major implications that falls out of this circumstance is that organizations will not adopt the standard—or “sign the treaty”—unless they believe the benefits will outweigh the costs. If a specification is going to be broadly adopted, then it needs to be designed so that all the adopting parties will see direct benefit. Remember, every minute of developer time spent implementing a standard could have been spent developing a feature or fixing a bug instead. If their customers don’t care about the benefits of the standard and developers don’t see internal benefits (like reducing the time they have to spend on one-off integrations), then the rational decision for product or project owners is to not implement the standard.

    Such a decision does not mean that all progress grinds to a halt. In the trade world, there are many more bilateral trade agreements than there are multilateral trade agreements. In the software world, the equivalent to a bilateral agreement would be a proprietary integration. Many companies today that want to provide richer integration with another platform—especially with an LMS—will use the multilateral agreement that is LTI 1.x as a baseline and extend it with the bilateral agreements of proprietary integrations. This is as it should be. Not every integration needs to be standards-based. In fact, a lot of innnovation happens beyond the edge of the standard as software developers come up with new ideas about how their products can work together. They hammer out a bilateral trade agreement to get that new idea implemented and to gain a competitive edge for a while. Eventually, enough products may choose to implement the same pattern that it makes sense for everybody to adopt the same way of integrating so that developers don’t have to build slightly different versions of the same integration over and over.

    Interoperability standards, being multilateral trade agreements, are rarely innovative or sexy if they are done right. Once in a while, a working group may come up with a new spin that is generative. But this is always a risky proposition. The technologists and product designers on these working groups, being creative people, want to come up with cool stuff. But “cool” isn’t what drives their employers to adopt. There always needs to be a cost/benefit analysis for each potential adopter. If that analysis doesn’t look good, then no amount of coolness will matter.

    As far as I can tell from the outside, this is precisely what went wrong with LTI 2.0.

    Please Pass (on) the SOAP

    Back in the early days of service-oriented architecture (SOA) in software engineering, everybody was using a complex XML-based protocol called SOAP. The idea was that everybody had data to pass around, and if there were discrete services that could be called for that data, then we could have a lot more interoperability. SOAP was considered secure (or securable, anyway), so it could be used for sensitive data. SOA advocates envisioned a world in which all software provided lots of services for other software to use, creating rich opportunities for interoperability. In fact, they thought there would be so many services that it would be impractical to connect them all up by hand. So they invented something called a service bus that helped each piece of software automagically discover the services available to it from other software in the system and also to offer up its own services to any other software that needed them.

    As you might imagine, this all turned out to be horribly complex and expensive. I happened to work at Oracle at the time that all this peaked, so I had a ring-side seat. The company was selling hugely expensive service management software that was often implemented by hugely expensive consultants. Some of that stuff took off and is in use in some places—usually big IT shops like moneycenter banks. But it never got the ubiquitous adoption that the enthusiasts expected because it was too much work for not enough payoff. There just weren’t that many services to justify the management costs.

    That was roughly ten years ago. Fast forward seven years from then. What is the essence of LTI 2.0? It was essentially an attempt to implement the same idea using the more modern REST style of web programming. It was going to have a strong security model and service discovery. Predictably, it was really hard to implement. A senior development manager from a major LMS vendor—somebody who has been in the industry and involved with the standards process for a very long time—said that LTI 2.0 was by far the most complex standard he’s ever had to implement. If the major LMS providers struggled to implement it, then what are the chances that lots of small LTI-compatible tool makers will implement it? To answer that question, start from zero and subtract a significant number.

    But I don’t think LTI would have had a high chance of success even had it been less complicated, because it wasn’t designed to be a multilateral trade agreement. What problem would a secure service bus with discoverability solve for each of the treaty signatories? The LMS companies do get a clear benefit. For them, many of the customers have LMS systems administrators who spend their days hooking up tools for individual instructors. The LMS gets most of the blame for this situation, not because it’s primarily their fault but because their customers perceive it as a pattern of weakness when they use the product. Automagic integrations would make some of their key stakeholders very happy.

    On the other hand, if you’re a tool vendor, you usually only get blamed for the challenges of one integration: your tool into the customer’s LMS. And even then, the LMS vendor gets most of the heat. At the same time, hardly anybody is passing data to you, and it can be a struggle to even get the LMS vendors to accept the data that you want to give them. So LTI 2.0 offered most tool makers a lot of pain for no obvious benefit.

    The people at the negotiating table for LTI 2.0 were apparently not the right people to negotiate a trade agreement. They were the right people to think of something really cool. For a standard to work, you need the former and want the latter, but only if the people thinking of cool stuff are in close communication with their colleagues who are making the cost/benefit decisions.

    Now, I believe there is an opportunity to get to the cool world that the LTI 2.0 working group envisioned. But to get there, you need to flip the script.

    Services First

    Let’s try another analogy for a moment. In the early days of the telephone, human operators connected every call manually. There was interoperability in the sense that all telephones operated the same way over the network of wires and switchboards, but each individual connection between callers had to be hand-configured. In the early days, that was fine. It made sense. There were few enough callers that it was worth the cost of the operators. But eventually, the number of phone connections grew to the point where hiring humans to manually wire connections was no longer viable for anybody. Call connection times grew, as did service provider costs. At that point it made economic sense to create an automated switchboard.

    LTI 2.0 was an attempt to invent a very fancy automated switchboard at a time when there are still hardly any callers. Once we reach the point where many tool providers are offering multiple data services and, in some cases, also receiving them, then they will see direct benefits from a secure and automated service discovery mechanism. At that point, the negotiations can begin regarding feature richness versus cost of implementation.

    To get there, the IMS needs to flip the script and start by creating a world that is rich enough in services that people need to care more about managing them all. The IMS has made a good (re)start with LTI Advantage, which just adds roster provisioning, grade return, and deep linking to LTI. These are basic hygiene needs. Basically, this means any tool can find out from the LMS who is in the class and what their roles are, which a lot of tools need and is a step up from the way LTI passes user information along now. LTI Advantage can also enable the tool provider to give the LMS links that support single sign-on to specific places within the tool, and it can enable tools to return multiple grades to the LMS. Lots of apps would find these capabilities handy.

    I’ve heard through the grapevine that the next challenge is improving security. Obviously, securing student data is critical. There are a lot of data sharing services that shouldn’t be offered until that security can be guaranteed. (I won’t comment on the security of rostering and grade information in LTI Advantage because I don’t know anything about it, but obviously that is one very sensitive area where schools should be asking questions about security.)

    Once security is in place, the next step beyond that should be to focus on generating enough call connections to keep the switchboard operator busy. I have been arguing for some time that Caliper should be used as a data interoperability exchange standard between apps that operates through the LTI window. This could work for bilateral agreements. For example, let’s say that you want to integrate a blog with an LMS so that posts with a certain tag could be automagically submitted upon publication for a particular assignment. Let’s further suppose that the IMS has not yet developed a standard, a multilateral trade agreement, for that kind of integration. If LTI and Caliper provide tools that make developing that integration easy enough, then the two integrating parties could do the work in the style of an official integration, saving time by drawing on the LTI and Caliper infrastructure that’s already in place. If enough other tool makers become interested in the integration, then it could be submitted for ratification as an official extension of the standard. This only works if designing and implementing a new Caliper profile is easy. But if it’s done right, then we should see the number of services explode.

    At some point, it will become obvious to all parties that they need autodiscovery to manage all these integrations. Then and only then will it make sense to come up with a hopefully simpler implementation of service discovery for the next run at LTI 2.x.

    Open It Up

    To reiterate, the path to getting to the next era of interoperability is to make it easy for many developers to build bilateral integrations that draw on the building blocks of the standards and are easy to incorporate as extensions to the standards. Some of this involves prioritizing the work on LTI and Caliper to make this easy. But it also likely will require the IMS to consider changing its policies in ways that will make some stakeholders uncomfortable.

    A little history is in order here. A bit more than a decade ago, the IMS nearly died. It wasn’t generating enough revenue to sustain the work. While IMS Global is a non-profit, it still has salaries to pay, rent to cover, and so on. Operating it is not free. One of the steps that the organization made to save it from potential insolvency was to put a lot of the work behind a paywall. I don’t like it, but I get it. And it worked. The IMS appears to be much healthier now and has produced some of its best work in a very long time. Life is about trade-offs.

    It may be time to consider a new trade-off. Just like getting the telephone to take off required a network effect—the value of the network increases exponentially with the number of people on it—the value of a world of standards-based services increases exponentially with the number of services on it. To get there, it’s time to at least consider lowering the pay wall. The IMS needs to create an environment with very low barriers to creating standards-compatible integrations. A membership fee, however reasonable it may seem to the membership organization, is a barrier. The IMS leadership should think creatively about how to lower this barrier while maintaining the financial stability of the larger organization.

    The Nub of It

    To sum up, here’s what I think the IMS should do to recover from the LTI 2.0 misstep and foster a step-function change in interoperability:

    1. Promote LTI Advantage and build from there.
    2. Focus next on creating a simple security model that is trustworthy for common use cases.
    3. Focus Caliper efforts on turning it into a quick and easy to use method for creating data interoperability extensions that take advantage of the existing LTI infrastructure.
    4. Make a giant push to get developers to build their bilateral integrations using the LTI and Caliper infrastructure, whether or not they submit those integrations as extensions to the standard. This should explicitly include re-examining policies about giving non-members early access to the work being done inside the IMS in cases where such access will increase the velocity of new Caliper-compatible service generation.
    5. Wait for demand to build on both provider and consumer side before attempting another run at discoverability.
  • Follow-Up From Future Trends Forum Discussion On Learning Platforms

    Follow-Up From Future Trends Forum Discussion On Learning Platforms

    Last Thursday I participated in a Future Trends Forum, hosted on the Shindig platform, with host Bryan Alexander on the topic of “What’s next with the LMS?”. I have to admit this was one of the best virtual discussions I’ve had, and more than half of the session was driven by audience questions. You can check out the Twitter discussion, Storified , or listen to an audio recording of the whole session, thanks to Roxanne Riskin. Update: See YouTube video of event posted at end.

    As Bryan described in his blog post:

    Yet maybe the conversation won’t stop there, at 3:05 pm EDT on June 29th. Because when we broke we had more than thirty (!) unanswered questions remaining from the Forum community. I’d like to share those now, so that Phil can respond, but also so that anyone can dive in, whether or not you participated yesterday.

    I don’t have enough time to address all the of the questions, but I would like to tackle a few. We’re also talking about having a part 2 and bringing in Michael in late August. In the meantime . . .

    • Competitiveness – Is the lack of competitiveness due to a lack of innovation or because IT decision makers are looking for consistency/ease of support?

    This question refers to the point I made in this blog post that in four global regions we have two companies dominating the installed base (Moodle, Blackboard), one dominating new implementations (Canvas) with one gaining recent momentum (D2L Brightspace). That’s four solutions dominating across the globe for higher education, hence the “lack of competitiveness” in the question.

    While I think the market needs more innovation, I don’t think that’s the primary cause of this emerging four-way oligopoly. One issue more important than pure innovation is that ed tech is a difficult market in terms of scaling a business, and it is difficult to remain profitable. Canvas is growing fastest, and Instructure (parent company) plans to be cash-flow positive in 2018. As in, Instructure is not profitable yet. Just two years ago Moody’s changed their outlook on Blackboard to negative due to very high debt to EBITDA ratios and “stagnating revenues”.  We have little public information on D2L, but I have pointed out company layoffs occurring after the large investment rounds. Moodle is open source and not directly a system with profits. This comes at a time when the expectations for competitive LMS offerings is rising in terms of cloud hosting and interoperability and intuitive user experience. This is not an easy business.

    • What are your thoughts about competency-based education (CBE)? In particular, with Elliucian leaving the space, do you see a market for a CBE-targeted LMS? And, which products do you see as leaders? or potential leaders?
    • CBE – You’ve brought up CBE a few times. Do you feel there is slower than expected growth for colleges/univ for CBE. Hence, the sun setting of Ellucian’s platform. Or, perhaps, are they force fitting the current LMS to be their CBE LMS?

    See this post describing the very slow growth of CBE platforms. If you think the institutional LMS market is difficult, try the CBE platform market. We do not have the same level of market data for CBE as we do for LMS, but anecdotally I believe that Sagence Learning (formerly FlatWorld) has won the greatest number of CBEN platform selections in the past year or two.

    • Have you seen any trends in terms of schools with more than one LMS – are places consolidating or fracturing? (always surprised by number of institutions that have more than one)

    There is a general, low-level trend in higher education to have fewer cases of secondary LMS usage. Mostly consolidating while aiming to increase the number of third-party apps working alongside the primary system.

    • hosted vs. not hosted – For those LMSs that aren’t open source, do you have any thoughts on how institutions are managing systems – are they choosing to host themselves or are they choosing to use vendor hosting (or other options)?

    In all four global regions we have covered (North America, Europe, Latin America, Oceania), there is a move towards managed and cloud hosting and away from self-hosting. In North America, more than 90% of new LMS selections are going straight to managed or cloud hosting. Europe trends the same direction but is roughly 50 / 50 for new implementations. And I should point out that these trends exist for open source solutions – maybe not to the same level, but in the same direction.

    • K-12 – following up on Schoology and google classroom, what force is K-12 going to be in the future of higher ed LMS? can the teaching energy and innovation of K-12 energize higher ed teaching…?

    I have written several posts on Google Classroom, although I need to do an update. The general answer is that neither Google Classroom or Facebook Classroom show significant signs of affecting the higher ed market. There is some interest in both platforms, but mostly from a small number of individual faculty. And neither platform is designed to solve the gradebook or system integration needs of higher ed. Yet.

    We have several posts on Schoology, which has a bigger potential impact on higher ed for a K-12 based system.

    • Finally, we partially discussed a set of question from Fred Beshears, summarized in this new post. The general topic is around CBE platforms and whether LMS systems are moving to support “massive student information profiles” (across courses for large numbers of students) or whether this is being relegated to student record systems. We partially address these questions in the Future Trends Forum, but not entirely. Hopefully we’ll see others jumping into the online discussion. Update: See discussion thread at Bryan’s post for discussion on this topic.
    • (Update: Finally, finally) I answered two questions in the session about the potential of the LMS as an integration hub, bringing in third-party apps rather than being monolithic systems. The answers were admittedly aspirational. George Station made a good point on Twitter that there is another side to the LMS impact on pedagogy:

    I do agree that this has happened, as covered further in replies to that tweet.