e-Literate

Present is Prologue

Category: Curricular-Materials

This category includes digital curricular materials, including adaptive learning, assessments, OER, etc.

  • Disruption Disrupted: The Great MOOC Die-Off

    Coursera has announced, with some fanfare, their Coursera for Campus initiative, which Jeff Young at EdSurge has characterized as an attempted entrance into the courseware market but which Coursera Vice President of Enterprise Leah Belsky described as aimed at the LMS market.

    From the EdSurge piece:

    Coursera for Campus is designed specifically with colleges in mind, says Leah Belsky, Coursera’s vice president of enterprise. That means the service includes new features tailored for use in an academic environment, including plagiarism detection to spot cheaters and integration with existing student gradebooks in the learning management systems (LMS) that colleges use.

    Meanwhile, Coursera is opening up its technology platform to any college to use for free to deliver course materials on their own campuses. That means that colleges could use the Coursera software as an alternative to their learning-management system. Belsky argues that Coursera’s system is better designed for delivering online courses and interactive lessons than most LMSes. 

    “We’re talking about a potential major disruption to the LMS market,” she says. “We don’t have all the features of an LMS but what we do have is all the tools to create cutting-edge interactive learning experiences.”

    This is bad framing from a PR perspective, but more importantly, it just plain misses the real potential value proposition and chases a plainly imaginary one instead.

    First, I hear a constant stream of complaints about both major MOOC providers having platforms that are not even fully adequate for their original purpose (though the situation does seem to be improving, particularly in terms of data analytics). At least one very high-profile customer, who I won’t name here, uses the MOOC platforms basically as a store front while putting their actual courses in an LMS. This comes at a time when some Blackboard customers still balk at switching from the classic to the Ultra experience because of feature gaps on level of granularity of test question feedback. MOOC platforms are interesting and have some innovative features, but they are neither mature for their original purpose nor tuned for the broad range of usage that a campus LMS must serve. Second, disruption talk is particularly tone deaf from anyone in a product category that was very recently known for hyping that they would be disrupting the university itself. And finally, disruptive innovation is an unfalsifiable theory that has thus far shown itself to have no predictive power in higher education. It’s a provocative idea that may have some generative intellectual value, but really, it’s well past time for every company that has aspirations in education to drop the word “disrupt” from their public vocabularies. Coursera for Schools may well have a decent value proposition, but this isn’t it.

    Jeff’s framing is closer to the truth: This is primarily a courseware play. There’s a little bit of gray area because courseware platforms and LMS platforms are slowly drifting toward each other in terms of design, but if they ever do converge, it won’t be in the next couple of years. No, this is about repurposing content. And the real story here is that the content needs to be repurposed because we have an overpopulation of MOOCs that are in the midst of a die-off. I’m not saying that MOOC companies are dying off. As far as I can tell, Coursera seems to be healthy. (I have less visibility into EdX’s financial status.) What I mean is that previous generation of the Stanford/MIT/Harvard-style xMOOCs, having failed to achieve either their mission or their sustainability goals, are now being repurposed into other things. Because we don’t have better names for those things, we still call them “MOOCs.” But they don’t meet the definition of Massively Open Online Courses. Even the Stanford/Harvard/MIT definition.

    Meanwhile, there are zombie MOOCs on these platforms that are in the process of getting killed off. Not too long ago, one campus stakeholder told me that their MOOCs basically serve the same purpose as their YouTube marketing videos, except that the YouTube videos get much better viewership and cost a lot less. I don’t expect the MOOC to die entirely, but two years from now, there will be a lot fewer of them than there are now. We just may not recognize that change if we insist on continuing to call any online enrollable thing with more than 30 students a “MOOC.”

    The known failure

    What’s weird is that everybody has known that the xMOOC was a failed experiment within 12 months of it reaching peak hype, and the widely known evidence has only mounted since then. In a January 2019 article in Science that was tellingly titled “The MOOC Pivot,” authors Justin Reich and José A. Ruipérez-Valiente write in the summary,

    When massive open online courses (MOOCs) first captured global attention in 2012, advocates imagined a disruptive transformation in postsecondary education. Video lectures from the world’s best professors could be broadcast to the farthest reaches of the networked world, and students could demonstrate proficiency using innovative computer-graded assessments, even in places with limited access to traditional education. But after promising a reordering of higher education, we see the field instead coalescing around a different, much older business model: helping universities outsource their online master’s degrees for professionals (1). To better understand the reasons for this shift, we highlight three patterns emerging from data on MOOCs provided by Harvard University and Massachusetts Institute of Technology (MIT) via the edX platform: The vast majority of MOOC learners never return after their first year, the growth in MOOC participation has been concentrated almost entirely in the world’s most affluent countries, and the bane of MOOCs—low completion rates (2)—has not improved over 6 years. [Emphasis added.]

    The entire (paywalled) article is worth reading, but honestly, is any of the above a shock to you? xMOOCs, as originally designed, are not replacements for face-to-face classes. They do not lead to reliable course or credential completion. They do not do a good job of serving underserved populations. And they do not create sustainability models by giving a way expensively produced courses and making up for the cost on volume. They have value for some folks. I’m glad they exist in the world. But as an alternative system of education, they are a failure.

    We see this proven out again and again, in multiple variations. Most recently, ASU has largely shut down their Global Freshman Academy, a large undergraduate MOOC experiment designed to give students credit for their first year of college and attract them to matriculate to ASU for a degree. From the IHE article:

    Of 373,000 people who enrolled, only 8,090 completed a course with a grade of C or better, just over 2 percent of all students enrolled. Around 1,750 students (0.47 percent) paid to receive college credit for completing a course, and fewer than 150 students (0.028 percent) went on to pursue a full degree at ASU.

    Was Global Freshman Academy an experiment worth running? Absolutely. In fact, I fervently hope that ASU will be more forthcoming than they have been so far with the lessons they have learned from the experience. Did students in those courses gain value? While that article doesn’t provide significant data on this question, I strongly suspect that many did. Were MOOCs an effective vehicle for saving freshmen roughly 25% of the cost of a college tuition while still getting them on to sophomore year? Clearly not.

    So what has worked as a MOOC-like alternative to the traditional degree? What’s the closest we’ve come to fulfilling the original vision? The most high-profile success has been Georgia Tech’s affordable graduate degrees at scale. Note: I have not seen them refer to these courses as “MOOCs.” They talk about “affordable degrees at scale.” Yes, these are big classes. And yes, Georgia Tech does use MOOC platforms as part of their delivery ecosystem. But their experiment was never about “massively open.” It was about “affordable at scale.” They wanted to see how inexpensively they could offer a degree while keeping some of the same structures and conventional quality checks that their face-to-face programs have. And for Udacity, Georgia Tech’s first MOOC partner, the degree program represented a pivot. (The first of several.) They essentially acted as a kind of Online Program Enabler, which is a weird category that crosses boundaries of platform, content, and services.

    When I look at Coursera’s latest announcement, I see an offering very roughly akin to the one that Udacity originally made for Georgia Tech, but with a heavier emphasis on prepackaged content—likely because Coursera already has a lot of content on the platform that may be quite good but is apparently not going to be disrupting universities, or conventional degrees, any time soon. If Coursera were to become (in part) a two-sided market for universities to buy and sell interactive curricular materials from each other, that’s not necessarily a horrible future for either the company or its customers. But it is suggestive of the collapse in the hype we’ve seen in both the MOOC and the OPM markets.

    OPMs, courseware providers, and “MOOCs”

    I want to return to the portion of that quote from the Science article about MOOC providers turning into OPM companies, which I highlighted but didn’t address. Again, that’s obviously true. All this talk about “micro masters” and “degree pathways” essentially amounts to a claim that many working professionals would like to pursue their post-graduate education in small, career-oriented (but still accredited and tuition-burdened) chunks. Likewise, MOOC providers like to talk about how MOOCs provide an inexpensive sales funnel to get degree students.

    While I haven’t seen direct hard data to support either claim, I’m more inclined to believe the former than the latter. A “micro-masters” program is essentially a large certificate program that can count toward a larger degree. Certificate programs have been around forever, and they sell. On the other hand, I’ve seen no public evidence that the MOOC, a course genre which has trouble getting students to the end of the first course, is going to be successful at getting students to matriculate to a program in significant numbers. The ASU Global Freshman Academy isn’t a direct comparison, since it is undergraduate, but the numbers are still pretty discouraging.

    Reich and Ruipérez-Valiente have their own opinion about the competitive advantage of MOOC providers in the OPM space, and it is revealing:

    The primary competitive advantage of MOOCs relative to established school-as-a- service providers involves cutting labor costs through automation. Many “traditional” online programs include small class sizes, synchronous sessions with instructors, and human-graded assignments. Many degrees offered by universities with the technology and support of Coursera and edX will be one- half or one-quarter as expensive as typical U.S. professional online credentials, with the bulk of savings coming from a combination of larger class sizes, fewer or no synchronous sessions, reduced contact with instructors, and more autograded assignments (12).

    This is precisely the value proposition that the digital homework solution—courseware’s older sibling—brought to the face-to-face lecture hall in survey-level courses. Digital assessment is what enabled those courses to swell to 500 or more students. The authors continue,

    Because MOOC platforms support programs that look more like “traditional” online higher education, the literature on online learning can provide guidance. By most indications, students typically do worse in online courses than in on-campus courses, and the challenges of online learning are particularly acute for the most vulnerable populations of first generation college students, students from low-income families, and underrepresented minorities (13). If low-cost, MOOC-based degrees end up recruiting the kinds of students who have historically been poorly served by online degree programs, student support programs will be vital. Some recent research has explored online and text-message–based interventions for supporting these students, but most research suggests that human connections through advisers, tutors, and peer groups provide the most important student supports (14). These human supports will push against lower tuition costs. MOOC- based degree providers may find that highly effective online learning for diverse populations costs about the same to provide as highly effective residential learning (12).

    When you start playing with this balancing act in order to arrive at…ahem…an affordable degree at scale and quality, you likely end up with something that looks very much like the Georgia Tech solution. It’s not a free degree or a $1,000 degree. It might be a $7,000 degree or a $14,000 degree. And that’s at the graduate level. It’s not clear that we know how to do this at the undergraduate level yet.

    But the more interesting implication vis-a-vis MOOC providers is that their value proposition starts looking more like that of modern courseware support with some services bundled in.

    As MOOC providers compete with conventional OPMs, there are entirely separate questions of financing the program development (via revenue share or some similar mechanism) and marketing. The MOOC providers have the advantage of their portals for marketing; students may go to Coursera or EdX to look for a credential program (as opposed to a full graduate degree program) before they’d go to their local university. I’ve not seen that proven, but at least it’s plausible. And the financing is what it is. Either you want a revenue share or you don’t.

    But as a genre of course, the population of xMOOCs is dying off. We don’t see it because we’re also calling the thing that is replacing them—which isn’t open—a “MOOC.” The collection of actual xMOOCs that are still functioning as full and (more or less) open courses is slowly shrinking to fit the size and shape of the professional non-degree credential market. Forking off from that is something that looks like a MOOC but is actually prepackaged courseware, to be licensed like a textbook and taught by individual instructors at different universities, with or without a face-to-face component. Then there’s this third thing—the affordable degree at scale—that is using MOOC and courseware affordances, which are increasingly the same affordances, to teach more students with similar learning outcomes at a lower cost. So far, only in professionally oriented graduate degree programs. And finally, there are zombie MOOCs that have no strong reason to exist and are being killed off by the platform providers for whom they are loss generators (sometimes to the dismay of the universities who invested considerable time and money in creating them).

    OK, maybe I was wrong. The word “disruption” is still relevant in at least one sense.

  • Content as an Instrument of Inquiry

    If you’re not writing clearly, then you’re not thinking clearly.

    Mrs. Galligani (My high school English teacher)

    In response to the posts so far in the series, John P. Mayer, Executive Director of the Center for Computer-Assisted Legal Instruction (CALI) tweeted,

    “This article mirrors what @caliorg Lessons are about – and have been for almost 40 years.”

    In the first post of the series, I explained the basic logic of the content design pattern to which John is referring:

    1. Start by articulating what you are trying to help students to learn as clearly as you can.
    2. Then figure out how you are going to be able to tell (or “assess”) whether the students students have, in fact, learned what you are trying to help them learn. Again, be as clear and explicit as you can in this.
    3. After you have taken the first two steps, then design learning activities—reading, watching, discussing, experimenting, and so on—that you believe will help the students to learn what you are trying to help them learn.
    4. As you design these activities and the content that makes it possible (e.g., readings for the reading activity), be sure to embed little low-stakes exercises throughout so that you and the students can track their progress toward learning whatever it is that you are trying to help them to learn. And, if necessary, to adjust the teaching or learning strategies.

    It’s not exactly rocket science.

    In the second post, I showed that we could derive pedagogical benefit when we formalize this content design pattern in digital curricular materials. If every bit of content and every assessment item is labeled with a particular learning objective (or learning goal), then we can track various kinds of student progress toward that learning objective. Have they started any activities related to that learning objective yet? Have they finished them? How well are they scoring on the assessment measures for that particular learning goal? How are they progressing through the stack of learning objectives that make up a larger curricular unit? Are any students stuck on one particular goal? Is the whole class stuck? We can get clearer answers to these questions, for ourselves and for our students by creating well-crafted curricular materials in the design pattern that I described above and creating very simple software capabilities for showing how students are performing on the assessments. Call this much “analytics” is a stretch. Statistical algorithms are optional and often superfluous to get us this far. We’re really just colating and visualizing the student activities and assessment results in ways that the content design enables us to do. In tech speak, we would call these “dashboards” and “visualizations” for the “data.” But your paper grade book (if you still use one) is a dashboard for the same data in a somewhat similar way. Honestly, once you throw in grading on curves, different point schemes, dropping the lowest score, and so on, some instructor grade books are algorithmically more complex than many of the courseware affordances I showed in my last post.

    The value of having many examples

    So far, I have been writing about the content design pattern as a tool for getting the most out of self-study curricular materials. That is still a good frame of reference for thinking about courseware or courseware-like content as static tools for teaching. But this content design pattern also has great value—in the long run, perhaps greater value—as a tool of inquiry for educators who want to improve their teaching craft.

    And this is the moment where I reveal my not-so-hidden agenda.

    My altar ego, Montgomery Burns

    If you thought this series was about courseware, then you were (mostly) wrong. If you thought this post was going to be about machine learning and algorithms and adaptive learning, then you were also (mostly) wrong. I will touch on those topics, but they are means to an end.

    And that end is to talk about increasing literacy and fluency among faculty in evidence based, digitally-enabled pedagogy. Just as the proliferation of commercial courseware provides us with a wide range of examples of professionally designed content which we can study to learn more about trends in curricular materials designs, the proliferation of analytics and adaptive learning provides us with an increasingly wide range of examples of professionally constructed analysis and inference techniques which we can study to learn more about current trends in such techniques.

    We have an ever-growing library of implementations—good, bad, and ugly—which educators could be studying and learning from if only the pedagogical design and function were not either opaque or invisible to them. Forget about the products as products for a minute. Set aside your feelings about the vendors for a moment too. We have digitally-enabled or enhanced teaching design patterns that have been evolving for at least 40 years. We are now are at a point where those patterns have reached critical mass, are proliferating at a rapid rate, and are being employed by academic and commercial course designers alike.

    Somebody should tell the faculty, don’t you think?

    The Empirical Educator Project (EEP) is going to undertake a major literacy effort to help increase awareness, literacy, and fluency in this design pattern, aided by some of the tools that Carnegie Mellon University has made available as part of OpenSimon and by the efforts of the wonderful people and organizations in the EEP network. I’ll have more to say about these efforts in the coming weeks. but for now, I am using this series to explain why I believe this is a fruitful area for educator literacy efforts.

    And it ain’t just about building a better textbook.

    Pedagogy as hypothesis

    Have you ever had one of those days where your carefully crafted lesson doesn’t go the way you thought you did? If you haven’t, then either you haven’t taught much or you’re doing it wrong. Because, given sufficient time in the classroom, every self-aware educator is going to have this experience. Something that you thought would be easy for the students turns out to be hard. An explanation that you thought was crystal clear confuses them. You thought the students were following along great until they bombed the unit test.

    In the classroom, most educators experience these problems and make adjustments as needed. They may not test for these problems programmatically. They may not conduct formal experiments to fix problem spots in their courses. But many, many educators do this in some form to some degree. Whether consciously or unconsciously, deliberately or instinctively, they are teaching by hypothesis. They think something will work, test it, and if they are wrong, they will try something else. Teaching isn’t assembly line work. It’s knowledge work. Done right, it requires an endless amount of creating problem-solving and tinkering.

    In the days of analog textbooks, it was a lot harder to do this kind of tinkering with student self-study work because educators were mostly blind. They couldn’t see what students were doing and they often didn’t even get to directly observe the results. They almost never got a chance to create a rapid feedback cycle with individual students to try a few different things and see what works. That changes with digital—if you have the right content design.

    As with the last post, I’m going to show some examples from Empirical Educator Project sponsors to show this content design pattern makes new teaching insights possible in a digital environment.

    Is your content working?

    Suppose you want to find out if the content in your digital curricular materials is actually helping students to learn what you are trying to help them learn. You could look at page views and other Google Analytics-style information to see what content students are spending time on. And you could look at the assessment scores. Take a moment and think about what you could learn from those two pieces of information.

    Eh, not much.

    Now suppose your curricular materials were created using backward design. For each content item, you know which assessment questions it is designed to prepare students for and which learning objective it is ultimately designed to teach.

    The golden triangle of instructional design

    You now have the ability to link page views to assessment data (and both to learning goals). Did your students spend a lot of time on a piece of content and still perform poorly on the associated assessments? Then something probably wrong with your design. Did they score well on the assessments while skipping over the associated content? Again, there’s a potential opportunity for improvement.

    What I’ve just described to you is the essence of the RISE framework, which was developed by Lumen Learning. As I described in a previous post, Lumen worked with Carnegie Mellon University to integrate RISE into one of CMU’s open source learning engineering tools. Now, anyone using any platform that can export data about page views and assessment items that are associated with a particular learning objective can generate a simple graph with page views on one axis and assessment performance on the other. It’s a simple and intuitive tool that any educator could use to get insights into whether their curricular materials are as effective as they could be. At last spring’s Empirical Educator Project (EEP) summit, Lumen and CMU demonstrated this process working using content from D2L Brightspace.

    Here’s a panel discussion we had about RISE and CMU’s OpenSimon LearnSphere at the EEP summit:

    RISE and Shine panel at the 2019 Empirical Educator Project summit

    Note that the content doesn’t fix itself through the magic of machine learning. Rather, RISE helps educators focus their attention on areas of potential improvement so that the humans can apply their expertise to the problem. This is true even in the most sophisticated commercial products. However fancy the algorithms may be, behind the scenes, experts are using the data to identify problems that still require humans to solve.

    Publishers that really understand the digital transformation have subject-matter experts who are pouring over the data and making changes on a regular basis. When Pearson announced that they would be moving to a digital-first model so that they could update their products more frequently I wrote,

    [I]n the digital world, there are legitimate reasons for product updates that don’t exist with a print textbook. First, you can actually get good data about whether your content is working so that you can make non-arbitrary improvements. I’m not talking about the sort of fancy machine learning algorithms that seem to be making some folks nervous these days. I’m talking about basic psychometric assessment measures that have been used since before digital but are really hard to gather data on at scale for an analog textbook—how hard are my test questions, and are any of the distractors obviously wrong?—and page analytics on the level that’s not much more sophisticated that I use for this blog—is anybody even reading that lesson? Lumen Learning’s RISE framework is a good, easy-to-understand example of the level of analysis that can be used to continuously improve content in a ho-hum, non-creepy, completely uncontroversial way.

    It would be irresponsible for curricular materials developers not to update their content when they identify a problem area. It would be like continuing to give a lesson in class when you know that lesson never works. But the main point here is that the software helps the human experts perform better. It doesn’t replace them.

    Are your assessments working?

    Just because students are looking at a piece of content and still scoring poorly on the related assessment items doesn’t necessarily mean that the problem is with the content. What if the assessment questions are bad?

    Psychometricians developed statistical methods for evaluating the quality of assessments many decades before the advent of digital courseware. The term of art for the most widely used collection of methods is “item analysis.” Item analysis tools have been built into most mainstream LMSs for quite some time. (If I recall correctly, ANGEL was the first platform to do so.) Here’s an example of an item analysis graph from Brightspace:

    Brightspace item reliability visualization

    This particular visualization is showing the results of something called a “reliability coefficient.” It tells you how consistent the student answers are to questions within an assessment. I’ll ask you again to think for a moment about how useful that measure would be by itself.

    Now think about how useful it would be to see how consistent student answers are to questions about one particular learning objective. The more interrelated questions are within a group of questions being evaluated this way, the more consistent the student responses should be. If all of the questions assess mastery the same learning objective, then performance across those questions should have a fairly high degree of internal consistency. Yes, some questions may be harder than others. But there should be an overall pattern of consistency among the student answers to questions assessing the same learning goal. If there isn’t, then there may well be something wrong with the assessment.

    And speaking of difficulty, item analysis does provide the ability to assess the difficulty of questions relative to each other. When you design questions to assess a learning goal, you may deliberately write some questions that you believe will be trickier than others. But you may not always be right about that. Sometimes students can get tripped up by the design of question, making it more difficult than you anticipated. Or maybe you unintentionally gave a clue to the answer in the design of your question. Or maybe there’s nothing wrong with your individual questions per se, but the overall difficulty of them for one learning objective is higher than it is for the others in the unit. Or lower. Wouldn’t you like to know those things? Well, you can. Your LMS probably has the tools to help you learn these things about assessments within the LMS. But those tools are a lot more useful if you have grouped your assessment questions by learning objective.

    Are your learning objectives working?

    Maybe the reason that the students are struggling isn’t because your content is confusing, or because your assessments are poorly designed, but because the learning goal you’re assessing isn’t as well defined as it could be. Sometimes you discover that what you think of as one skill actually contains a second skill or knowledge component that you take for granted but that is tripping students up. Here’s how Carnegie Mellon University professor Ken Koedinger puts it:

    Ken Koedinger on hidden prerequisites

    If you have that problem, then there should be evidence in the assessment data. Students who are taking well-designed assessments that cleanly assess one learning goal should show improved performance on assessment questions over time as they progress toward mastery. Here’s an example of what Carnegie Mellon University calls a “learning curve,” showing exactly that trend in student performance in their OLI platform:

    OLI learning curve

    But what if your learning curve looks like one of these?

    Anomalous learning curves

    See the blip in the graph on the top right? That could suggest that one or more questions snuck in that test something other than just the intended learning objective. (Or they could have been poorly written questions.) The graph on the top left, on the other hand, may show that students have already mastered a learning objective, or that the questions are two easy. If you take some time to look at each of these graphs, they may suggest different questions to you about your curricular materials design. All of this is possible because assessment questions have been linked to learning objectives in the content design. And again, the solution when one of these anomalies appears is generally to have a human expert figure out what is going on and improve the content design.

    Are your scope and sequence working?

    Implicit in that last section is the notion that it’s possible for even good, experienced educators can miss prerequisite skills in their course designs from time to time. Very often, the algorithms behind skills-based adaptive learning platforms are testing for missing prerequisites. In the marketing, the emphasis is placed on the software’s ability to identify prerequisite skills that individual students may have missed along the way. But it’s important to understand what’s going on under the hood here and how it affects the content design and revision of these products by the human experts. One of the things that this type of adaptive software is really doing is finding correlations between students doing poorly on one skill and them doing poorly on prerequisite skills. In some cases, the prerequisites are well known by the content designers. In those situations, the software is checking to see if the student is struggling on a lesson because she needs to review a previous lesson. In these cases, the “adaptive” part of adaptive learning means that the system automatically provides students with opportunities to review the prerequisite lessons that they need to brush up on. ((This is not the only way that adaptive learning products can work, but it is a common approach.))

    But sometimes the software identifies a correlation between a skill and a previous skill that either the content designers weren’t fully aware was a prerequisite or did not realize how important of a prerequisite it is. The analysis shows that students who struggle to learn Skill F, perhaps surprisingly, often didn’t do so well with Skill B.

    Unfortunately I don’t have any screen shots illustrating this—maybe somebody reading this will send me one—but it’s an easy enough idea to grasp. And once again, the algorithms that drive this analysis only work when content is designed such that assessment questions are tied to specific learning objectives.

    Is your classroom pedagogy working?

    Now I’m going to explore a theoretical affordance made possible by this content design pattern. It’s not one that I’ve seen implemented anywhere. But we can get the outlines of it by looking at Pearson’s efficacy report and educator guide for their Revel psychology product. (Reminder: Pearson has paid me to consult for them on how to make their efficacy reports, including this one, as useful as possible.)

    One of the main efficacy measures in Pearson’s study was, essentially, the size of the improvement students showed from their formative assessments to their summative assessments. This was measured in a class where a professor was implementing specific pedagogical practices that are often lumped together under the heading “flipped classroom.” Teasing this out, we would expect students to experience some benefit from the formative assessments and the curricular materials themselves, and some benefit from the professor’s teaching practices that helped students take maximum advantage of what they could learn about their progress from their formative assessment performance that the courseware was giving them.

    Let’s think about the performance improvement measured in this study as a kind of a benchmark. It shows how much of benefit a class could experience given a particular set of teaching practices and students that tend to take that sort of class in that sort of university using that particular curricular materials product. What you might want to do, as an educator who is interested in active learning and flipped classroom techniques, is evaluate how much of an influence your classroom practices have on the benefit that students can extract from the formative assessments of the product.

    If both the formative and summative assessments are tagged with the same learning objective, then it should be possible to give every instructor with a gauge like the one that Pearson had to create in order to measure the impact of their product under close-to-ideal conditions. But we’d be flipping the analysis on its head. Rather than controlling for the teaching methods to measure the impact of the content, we’d be controlling for the content in order to measure the impact of different teaching methods. Instructors could try different strategies and see if they increase the benefit that students can get from the formative assessments.

    Yet again, this possibility for creating innovative “learning analytics” is becomes apparent once you do some hard thinking about the simple content design pattern and all the different kinds of insights that you can extract from implementing it.

    For the techies reading this, the lesson is that the value of educational metadata generated by human experts for the purpose of supporting their own thinking will exponentially increase the potential for your machine learning algorithms to generate useful educational insights. By itself, even the most sophisticated machine learning techniques will have sharply delimited applicability and severely limited value in a semantically impoverished environment. My favorite high school English teacher used to admonish, “If you’re not writing clearly, then you’re not thinking clearly.” Something like the inverse could be said for learning analytics. Clear writing is not evidence of clear thinking but rather the prerequisite for it. If the content authors do not provide the algorithms with indications of pedagogical intent, then the algorithms will not have the cues they need to make useful inferences. If content is infrastructure, then content metadata is architecture. It is a blueprint.

    For the educators in the audience, I hope it’s clear by now that this design pattern, which is ubiquitous in commercial digital curricular materials and quite possible to implement in platforms like LMSs, is something that educators need to be aware of and understand. We have a literacy challenge and a fluency opportunity.

    The opportunity before us

    In the coming days and weeks, I will be sharing details of several major initiatives from the Empirical Educator Project related to this challenge and opportunity. I haven’t even shared them with the EEP network yet because I am in the process of nailing down a few final details right now. But I am very close to nailing down those details. So close, in fact, that I expect to be able to share some of them with the public in the final installment to this series.

    Stay tuned.

    But also, y’all don’t have to wait for me or EEP. Many of you have deep expertise in this content design pattern and what can be done with it already. I’ve found that my biggest challenge in trying to talk to experts about this challenge is that they so take for granted the concepts I’ve been outlining in these last three posts that it’s hard for them to even think about them as something to be discussed and explored. I hope that this series has clarified the need for and value in talking about the content design explicitly. When we talk about the value of digital content in terms of algorithms and data and functionality and other digital terms, it is easy for educators lose the thread. But when those conversations are grounded in the design of content and courses and pedagogy, then the digital accoutrements become embodiments of teaching strategies that make sense to them and that they can think about critically as experts.

    More of that, please.

  • The Affordances of Content Design

    The Affordances of Content Design

    Content is infrastructure.

    David Wiley

    I opened my first post in this series with a statement about courseware and content design:

    An unbelievable number of words have been written about the technology affordances of courseware—progress indicators, nudges, analytics, adaptive algorithms, and so on. But what seems to have gone completely unnoticed in all this analysis is that the quiet revolution in the design of educational content that makes all of these affordances possible. It is invisible to professional course designers because it is like the air they breathe. They take it for granted, and nobody outside of their domain asks them what they’re doing or why. It’s invisible to everybody else because nobody talks about it. We are distracted by the technology bells and whistle. But make no mistake: There would be no fancy courseware technology without this change in content design. It is the key to everything. Once you understand it, suddenly the technology possibilities and limitations become much clearer.

    That’s all true. But this series isn’t really about courseware. It’s about the capabilities and limitations of digital curricular materials, whether they are products sold by vendors, OER, or faculty-developed. The content design pattern I’m exploring is neither unique to vended courseware products nor invented by commercial courseware providers. In fact, instructional designers and LMS providers have been desperately trying to convince faculty of the value of this course design pattern for a many years. But designing content this way takes a lot of work and lacking good examples of the return on that investment, most instructors have not opted to build their content this way.

    What the proliferation of commercial courseware provides that is new is a wealth of professionally developed examples that we can examine to better understand how this content design pattern works to support certain teaching and learning affordances in digital curricular materials. In this post and the next, I will draw on some of those examples, which happen to come from Empirical Educator Project sponsors, to show the design pattern in action.

    The most important message of this series, for both educators and technologists, is that real advances in educational technology will almost always arise out of and be best understood through our knowledge of teaching and learning. In this case, technological affordances such as learning analytics and adaptive learning are only possible because of the instructional design of the content upon which they operate. And we sometimes forget that “instructional design” means design of instruction. The baseline we are working from is instructional content, generally (but not exclusively) designed for self-study. How much value can students get from it? How far can we push that envelope? Whatever the fancy algorithms may be doing, they are doing it with, to, and around the content. The content is the infrastructure. So if you can develop a rich understanding of the value, uses, and limitations of the content, then you can understand the value, uses, and limitations of the both technologies applied to the content and the pedagogical strategies that the combination of content and technologies afford.

    The role of digital curricular materials

    Let’s start by looking at the holistic role that digital curricular materials play when implemented in a way that the design pattern supports. From there, we’ll back into some of the details.

    I’m going to ask you to watch a short promotional video from Pearson of a psychology professor who participated in one of their efficacy studies shares her experiences and observations about teaching with their courseware products. (You should know that Pearson has engaged me as a consultant to review their efficacy reports, including this one, to provide them with feedback on how to make those reports as useful as possible.) The fact that this professor’s story is part of a larger efficacy study means that it is richly documented in ways that are useful to our current purpose.

    As you watch, pay attention to Dr. Williamson says about the affordances of the content and how those affordances support her pedagogical strategies and objectives:

    Dr. Manda Williamson of University of Nebraska-Lincoln on her courseware experiment

    The first thing she talks about is layered formative assessments. Students are given small chunks of content followed by frequent learning activities. They then are prompted to take formative assessments which, depending on the results and the students’ confidence levels, may result in recommending additional activity. (The one mentioned in the video was “rereading.”) If your anchor point for the value of the product is the readings that you assign for homework, then you can see how interactive content that is well designed in this way might be an improvement over flat, non-interactive readings (or even videos).

    When the students come into class—and this is key—Dr. Williamson engages with them on the results of their formative assessments. She teaches to where the students are, and she knows where they are because she has the data from the formative assessments.

    How does that work?

    Those assessment items are tied to learning objectives. Skills and knowledge that have been clearly articulated. In well designed content, the learning objectives have been articulated first and the assessment questions have been written specifically to align with those learning goals. With this content design work in place, creating a “dashboard” is not technologically complicated or fancy at all. No clever algorithms are necessary.

    Suppose you give students five questions for each learning objective. One way you could create a dashboard is to show a line item for each learning objective and show what percentage of the class got all five questions right, what percentage got four out of five, and so on. I’ll show some example dashboards from other products later in this post. For now, the take-away is that the students are basically taking low-stakes quizzes along with their readings, and the instructor is getting the quiz results before the class starts so that she can teach the students to where they are.

    Hopefully the formative assessments don’t feel like “quizzes;” Dr. Williamson has positioned them as tools to help the students learn, which is how exactly how formative assessments should be positioned. But the main point is that the content includes some assessed activity which enables the teacher to have a clearer understanding of what the students know and what kinds of help they may need.

    As a result of adopting the digital content design and teaching strategies that the content and technology affordances supported, Dr. Williamson’s DFW rate dropped from 44% to 12%. Since her course is a gateway course, that number is particularly important for overall student success. So it’s a dramatic success story. But it’s not magic. If you understand teaching, and if you look at the improvements made in the self-study content and the in-class teaching strategies, you quickly come to see that it’s not technology magic but thoughtful curriculum design, solid product usability and utility, and hard work in the classroom that produced these gains. Technology played a critical but highly circumscribed supporting role.

    You can read more about Pearson’s efficacy study, ranging from an academic account of the research to a more layperson-oriented educator guide, here.

    Design details

    It might help to make this a little more concrete. I’m going to provide a few example screens in this post that are fairly closely tied to the basic affordances that I’ve discussed above, and then I’m going to explore some more complex variations in the next post in this series.

    I mentioned earlier that the formative assessments should function like quizzes but that students should not feel like they are being tested. This idea—that the assessments are to help the students rather than to examine or surveil them—is built into the design of good curricular materials in this style. For example, Lumen Learning’s Waymaker courses has a module that explicitly addresses this idea with the students:

    Lumen Learning “Succeeding With Waymaker” module emphasizes the value of formative assessment.

    The Waymaker product then uses the formative assessments the students take, tied to their learning objectives, to show students the associated content areas where they have shown mastery and others where they still need some work. This student dashboard is called the “study plan”:

    Lumen Learning’s study plan updates based on formative assessment scores.

    There are different philosophies about how to provide this kind of feedback. One product designer told me one philosophy he was thinking about is that the best dashboard is no dashboard, meaning that giving student little progress indicators and nudges are better. For educators evaluating different ways to deliver the content, the commonalities provide the tools for evaluating the differences. “Data” are (primarily) the formative student assessment answers. “Analytics” are ways of summing up or extracting insights from the collection of answers, either for an individual student or for a class. “Dashboards,” “nudges,” and “progress indicators” are methods of communicating useful insights in ways that encourage productive action, either on the part of the student or the educator.

    Speaking of the latter, let’s look at some educator dashboards. Let’s look at a dashboard from Soomo Learning’s Webtext platform. Even before you get into how students are performing on their formative assessments, you might want to know how far students have gotten on their assigned work. This might be particularly important in an asynchronous online course or other environment where you have particular reason to expect that students will be moving along at different paces. So this dashboard sorts student by their progress in a chapter:

    Soomo Learning Webtext dashboard shows percentage of questions answered in a chapter.

    Notice that progress here is measured by percentage of questions answered. That tells us something about where the product designers think the value is. A formative assessment isn’t only a measure of learning progress. It is also a learning activity in and of itself. We learn by doing. We learn more effectively by doing and getting instant feedback. So rather than measure pages viewed or time-on-page (although we do see a toggle option for “time” in the upper right-hand corner), the first measure in the dashboard is percentage of questions answered.

    Drilling down, Soomo also shows percentage correct by page:

    Soomo’s Webtext dashboard shows student score by page

    There’s a bit of a rabbit hole that I’m going to point to but avoid going down regarding how cleanly one can separate learning objectives. Does it always make the most sense to present one and only one learning objective per page? And if so, then what’s the best way to present analytics? Rather than explore Soomo’s particular philosophy on that fine point, let’s focus on highlights of the low scores. This is one detail that instructors will want to know at some fairly fine level of granularity. (If two learning objectives are on the same well-designed page, it’s usually because they’re closely related.) This dashboard enables instructors to see which students, both individually and as a group, scored poorly on particular assessments on a page.

    Again, there’s no algorithmic magic here. Let’s assume for the sake of argument that the content and assessments are well designed. Soomo is thinking about what educators would need to know about how students are progressing through the self-study content in order to make good instructional decisions. They are then designing their screens to make that information available at a glance.

    Now imagine for a moment that you have this kind of increased visibility on how students are doing with their self-study. You see that students are doing well on a learning objective overall, but they’re struggling with one particular question. In the old world of analog homework, you might not catch this sort of thing until a high-stakes test. But with digital curricular materials, where you can give more formative assessment and have it scored for you (within the bounds of what machines are capable of scoring), you might quickly find one particular problem in an assessment that students are struggling with. Is the question poorly written? Is it catching a hidden skill, or a twist that you didn’t realize made the problem difficult? You’d want to drill down. Here’s a drill-down screen from Macmillan’s Achieve formative assessment product:

    Macmillan Achieve question drill-down shows question-by-question performance.

    (You should know that I serve on Macmillan’s Impact Research Advisory Council.)

    This is exactly the sort of clue that an educator might want to look at while preparing for a class. What are the unusual patterns of student performance? What might that tell us about hidden learning challenges and opportunities? And what might it tell us about our course design?

    Hints of what’s coming

    I’ll share two more screen shots as a way of teasing some of the concepts coming in the next post. This first one is from Carnegie Mellon University’s OLI platform:

    Carnegie Mellon University OLI’s Predicted Mastery learning dashboard

    At first glance, this looks like another learning dashboard. What percentage of the class are green, yellow, or red (or haven’t started) for each learning objective? But notice one little word: “predicted mastery levels.” Predicted. Once you start collecting enough data, by which we mean enough student scores to begin to see meaningful patterns, we can apply statistical analysis to make predictions. There is a certain amount of justifiable anxiety about using predictive algorithms in education, but the problem springs from applying the math without understanding it. That’s what predictive algorithms are, at their most basic. They’re statistical math formulas. And honestly, many of the predictive algorithms used in ed tech are, in fact, basic enough that educators can get the gist of them. We’ve been taught to believe that the magic is in the algorithm. But really, most of the time, the magic is in the content design.

    And here’s a screen from D2L Brightspace:

    Brightspace conditional release tablet view.

    There’s a lot to unpack here, and I won’t be able to get to it all in this post. This is a tablet view of functions that Brightspace has been building up forever and a day. Since long before modern courseware existed as a product category. For starters, you can see in the top box that Brightspace can assign mastery for a learning objective. (In this case, the objective happens to be “CBE Terminology: Prior Knowledge.”) But what follows is a set of simple programming instructions of the form, “If a student meets condition X [e.g., receives less than 65% on a particular assessment] then perform action Y [e.g., show video Z].” In the olden days of personal computers, we would call this a “macro.” In the olden days of LMSs, we would call it “conditional release.” Today’s hot lingo for it is “adaptive learning” or “personalized learning.” Notice in this example that we are still starting with performance against a learning objective. We are still starting with content design.

    (Note also that many of the technology affordances built into vended courseware are also available in content-agnostic products like LMSs and have been for quite some time. Instructors can build content in this design pattern with the tools they have at hand and gain benefits from it.)

    In the next post, I’m going to talk about how advanced statistical techniques, including machine learning techniques, and automation, including what we commonly refer to as adaptive learning, are methods that digital course content designers use to enhance the value of their course content designs. But all of those enhancements still build off of and depend upon that bedrock content design pattern that I described in the first post of this series.

    The atomic unit of digital curricular materials design

  • The Content Revolution

    Content is infrastructure.

    David Wiley

    An unbelievable number of words have been written about the technology affordances of courseware—progress indicators, nudges, analytics, adaptive algorithms, and so on. But what seems to have gone completely unnoticed in all this analysis is that the quiet revolution in the design of educational content that makes all of these affordances possible. It is invisible to professional course designers because it is like the air they breathe. They take it for granted, and nobody outside of their domain asks them what they’re doing or why. It’s invisible to everybody else because nobody talks about it. We are distracted by the technology bells and whistle. But make no mistake: There would be no fancy courseware technology without this change in content design. It is the key to everything. Once you understand it, suddenly the technology possibilities and limitations become much clearer.

    For those familiar with course design lingo, the design pattern I am talking about can be summed up as backward design coupled with programmatic formative assessment. This post is the first in a series in which I will explain this design pattern, it’s possibilities and limitations, and the ways in which it makes possible a whole range of educational technology affordances.

    Backward Design

    “Backward Design” is a term that comes from a larger framework called “Understanding by Design,” (UbD) developed by Grant Wiggins and Jay McTighe and articulated in a book by the same name. While it was developed as K12 curriculum design approach, it has been widely embraced by curriculum and course content designers at all levels. Because the backward design practice as applied in courseware authoring necessarily requires what some might perceive as a “dumbing down” of the approach (for reasons I will get into later in this post), it’s important to understand the philosophical roots of UbD. On one hand, this is an approach that is grounded in the political reality of a K12 world that is driven by curriculum standards. Wiggins and McTighe are unapologetic about having defined curricular goals for students. On the other, UbD is intended to work against the tendency to memorization of facts and rote applications of lower-order skills, fostering critical thinking and knowledge transfer across domains. Three of the seven tenets of UbD (as articulated in this crisply written white paper by Wiggins) are as follows:

    • The UbD framework helps focus curriculum and teaching on the develop- ment and deepening of student understanding and transfer of learning (i.e., the ability to effectively use content knowledge and skill).
    • Understanding is revealed when students autonomously make sense of and transfer their learning through authentic performance. Six facets of under- standing—the capacity to explain, interpret, apply, shift perspective, empa- thize, and self-assess—can serve as indicators of understanding.
    • Teachers are coaches of understanding, not mere purveyors of content knowl- edge, skill, or activity. They focus on ensuring that learning happens, not just teaching (and assuming that what was taught was learned); they always aim and check for successful meaning making and transfer by the learner.

    UbD is explicitly not a paint-by-numbers approach to education. It is, however, a design-intensive approach to teaching that emphasizes the value of preparation and goal-oriented thinking as a key to unlocking teachable moments. This 10-minute video of Wiggins explaining the philosophy is well worth your time and provides a philosophical guide star to keep in sight as we navigate backwards design in general and its application to courseware design in particular:

    Grant Wiggins – Understanding by Design

    The upshot of his message here is that teachers and student continually need to be asking the question, both individually and together—what are the larger learning goals here?

    Backwards Design, at its most basic, is the idea that educators should be asking that question from the moment they start planning their course. Rather than starting with a collection of content and activities and putting it into a sequence, educators should start by articulating the end goals for the students (where an end goal is broad enough to encompass high-level and non-cognitive goals such as “a love of reading”). The three-step process of backward design is as follows:

    1. Identify desired results
    2. Determine acceptable evidence
    3. Plan learning experiences and instruction

    All content and activity choices flow from identifying the desired results and determining acceptable evidence of achievement of those results. This approach is “backwards” from the typical approach of starting with content that needs to be covered.

    Backward Design in courseware development

    Modern courseware, and many of the most highly touted technology affordances that come with it, flow from the Backward Design technique. But there are two additional constraints that are imposed by the medium. First, the activities in the courseware can only be activities that can be facilitated in an online medium—and, since courseware is modeled after the textbook, these are usually (but not always) solo activities by students that look like digital extensions of the kinds of exercises that you would expect from textbooks. Second, since key technological affordances of courseware come from its ability to auto-assess student progress, “acceptable evidence” generally must be machine-gradable evidence.

    Since this is Backward Design, these changes have implications up the chain to the first step in the process. Rather than “identifying desired results,” courseware designers have to think in terms of “learning objectives” that are realistic to achieve and measure given the limitations of the medium. The University of Central Florida (UCF) has posted a learning objective builder tool which, while not limited to the application of courseware design, begins to convey how courseware designers need to think about learning objectives in order to design content that will work in the courseware medium. The learning objective structure in the UCF example has four components:

    1. Condition, e.g., “Given a blank map of the United States…”
    2. Audience, e.g., “…the student…”
    3. Behavior, e.g., “…will identify all 50 states and capitals…”
    4. Degree, e.g., “…with 90% accuracy.”

    In comparison to Wiggins’ framing of UbD, this may feel starkly reductive to you. It’s important to keep in mind that the example was undoubtedly written for clarity rather than to illustrate how creative an educator can be while still staying within the bounds of the format. That said, there is no question that the format is limiting.

    And this, I think, is where a lot of unilluminating argument over the value of courseware originates. On the one hand, if the idea is that courseware will largely replace human instruction, then we have to recognize the gap between the learning objectives which the courseware can assess and the desired educational results which a human teacher can address and assess. On the other hand, it’s very hard to talk about that gap meaningfully and specifically when the entire course hasn’t been backward designed in the first place. If a course has clearly defined desired outcomes and clearly defined acceptable evidence of those outcomes, then it is a straightforward exercise to identify the subset of goals and evidence that courseware can address. But in absence of that larger course blueprint, educators who want to argue that courseware is too reductive start to get hand-wavy pretty quickly. We shouldn’t be surprised that interactive curricular materials are not complete substitutes for a classroom experience, but we should be able to clearly articulate what the gap is and how classroom interactions address that gap in ways that courseware can’t on on a course-by-course basis.

    The atomic unit of courseware content design

    Once we’ve translated the principles of Backward Design to fit the constraints of courseware, we end up with a tightly constructed content design:

    The content triangle of learning objectives, assessments, and instructional activities

    Again, this structure is not limited to courseware; it’s a good distillation of the results of backward design in general, using language that also translates well into courseware design. But when building modern courseware, this design is formal and structural. Every instructional activity (which, in the case of courseware, means interactive or non-interactive content items) and every assessment activity is tagged to correspond with a specific learning objective. As far as the software is concerned, this collection of items and metadata is a formal and atomic unit of instruction. McGraw-Hill Education even went so far as to name this collection a “compound learning object (CLO)“.

    As we will see in detail in the next post in this series, many of the technology affordances of modern courseware depend utterly on this formal structure. And once you understand the design pattern, you can see it everywhere in most curricular materials products and in an increasing number of courses designed on campuses with the help of professional instructional designers. I would go so far as to say that the formalization of this content structure, and not any fancy technology capabilities like adaptive learning algorithms or learning analytics dashboards, is the defining innovation in curricular materials over the last decade. It is the key to everything.

    Programmatic formative assessment

    There is one other defining content feature that is worth talking about before we explore implementation examples in the next post. A key educational affordance of courseware products that is often touted is instantaneous feedback. Since there is strong evidence that timely feedback is critical to the learning process, this is a key benefit (assuming that the feedback is meaningful). But instantaneous feedback on summative assessments—on assessments at the end that measure how much the student has learned before moving on to the next lesson—is not as helpful to students as it might be because…well…they’re moving on to the next lesson. They may or may not take the time to reflect on their incorrect answers. In contrast, feedback on low-stakes assessments, particularly when it is supported by feedback and support from the educator, can be very useful. In fact, I have long argued that this ability to have students practice their skills and test themselves—yes, before they take a summative assessment, but more importantly, before they walk into a class discussion—is a key value proposition for modern courseware. Class preparation.

    This is often billed as a technology affordance, but once again it is utterly dependent on the content design. Their analog…er…analogue is back-of-the-chapter homework problems. Practice problems that are linked to a skill or a bit of knowledge that will ultimately be assessed for a grade is not a new idea. The technology simply improves on the kinds of practice and feedback that were possible with flat textbooks. It can be given more often, in more interactive formats, with more timely feedback.

    Circling back to the Grant Wiggins video at the top of this post, students and educators alike need to constantly be asking the question “Why am I doing this now?” Whatever we are learning—or teaching—we should always also be studying whether our activities are aligned with our goals. Good educators are continually assessing their students in a variety of ways, starting with looking at their faces to see if they look like they are following, bored, confused, etc. They adjust according to what they see. Likewise, students need to be assessing their learning strategies and progress in order to get better at achieving their learning goals. One defining characteristic of modern courseware content design is creating as close to a continuous assessment feedback loop as possible.

    Content as infrastructure

    As I’ve stressed throughout this post, I don’t think it’s possible to overstate the role of this content design pattern—Backward Design plus programmatic formative assessment—in most of the recent innovations in digital curricular materials. In the next posts in this series, I will show concrete examples of how this design pattern makes various technological affordances possible as well as how it opens up new possibilities for tuning both courseware content and teaching strategies for continuous improvement. In the last installation, I will write about the need for and benefits of having content interchange and analytics interoperability standards that are tuned to this ubiquitous yet invisible content design pattern.

  • Pearson’s Born-Digital Move and Frequency of Updates

    Pearson’s Born-Digital Move and Frequency of Updates

    There’s been a bit of an uproar over Pearson’s announcement that they are switching entirely to a digital-first model and will be updating their editions more frequently as a result. ((Full disclosure: Pearson is both a current sponsor of the Empirical Educator Project and a current consulting client.)) The prevailing take in the media write-ups so far has been fear of increasing prices. I’m not doing as much of that sort of analysis as I used to anymore. As usual, if you want a clear write-up of it, a good place to check is with PhilonEdTech. I do have a somewhat different take on the economics than the current hand-wringing would suggest and will use a graph from Phil’s post to make a brief point about it. The middle of this post will be about the difference between print and digital and how that drives different reasons for updating an “edition.” And the last part will be about drawing larger lessons from the tendency in ed tech to tell stories that are more based on past traumas than analysis of current situations.

    The horse is out of the barn on pricing

    Phil’s post contains this graph showing the publishers’ competition with other sources for textbook rentals:

    Graph of textbook rental distribution from PhilonEdTech, sourced from NACS

    The textbook publishers have a lot to gain by controlling the distribution channel for their products. If they can cut out Amazon and Chegg, then they can get a larger percentage of each sale. So they definitely have a lot to gain economically from this.

    But I don’t get the sense that any of the major publishers believe they can raise prices again. They all read the OER faculty attitude surveys very carefully. The prevailing sense in the industry is that, in addition to the strong price sensitivity and ingenuity that has existed among students for some time, there is now increasing price awareness and sensitivity among academics that is not going away. I don’t think this is about that.

    In fact, while there are immediate economic reasons for publishers to make this move—if remember correctly, McGraw Hill made the same move a while ago—there are also product-related reasons for doing so.

    Analog vs digital updates

    Many folks in the sector have a reflex reaction to the phrase “textbook edition” based on the old print tradition of updating a book every three years whether it needed updating or not. In some subjects, a three-year update is warranted. Programming languages change, for example. Linear algebra, on the other hand? Not so much. Textbook publishers gained a somewhat justified reputation for updating books every three years just to thwart the used book market. Which they then reinforced by raising prices every year, driving students to the used book market and giving publishers further incentives to update editions for purely economic reasons.

    That said, in the digital world, there are legitimate reasons for product updates that don’t exist with a print textbook. First, you can actually get good data about whether your content is working so that you can make non-arbitrary improvements. I’m not talking about the sort of fancy machine learning algorithms that seem to be making some folks nervous these days. I’m talking about basic psychometric assessment measures that have been used since before digital but are really hard to gather data on at scale for an analog textbook—how hard are my test questions, and are any of the distractors obviously wrong?—and page analytics on the level that’s not much more sophisticated that I use for this blog—is anybody even reading that lesson? Lumen Learning’s RISE framework is a good, easy-to-understand example of the level of analysis that can be used to continuously improve content in a ho-hum, non-creepy, completely uncontroversial way.

    (I stress this because there’s been an elevated level of concern about intrusive analytics among a segment of the e-Literate readership and I want to be clear that one can do quite a bit without coming anywhere near the touchy areas.)

    At the same time, unlike paper books, digital products have functionality, and there is always a continuous list of features that students and teachers want or need. Just keeping up with accessibility requirements is a never-ending job. Then, instructors in different subjects want different quiz question types. Or the ability to author those question types. Or different configurations on the number of tries that students can have for the questions. Or the number of hints they can have. Or integration with some discipline-specific tool that they like to use. Or a tool that the students like to use. It goes on and on and on.

    Are all of these updates good updates? That’s an impossible generalization to make. People who have no use for a software product category in the first place generally tend to think that the updates to a pointless product are pointless. So if you’re already cynical about courseware, you’ll probably be cynical about courseware functionality updates. If you find courseware useful, then your attitude will be closer to that of any other software user, which is to say that you’ll look at whether the particular update fits your need. Since functionality in courseware platform often is differentially useful across disciplines, chances are that you will like some updates a lot and find others completely uninteresting (or even a step backwards for your particular needs). If you are a builder of digital platforms that need to support a wide range of academic disciplines, as Pearson now is, you have a lot of surface area that you have to cover. Hence the need for frequent updates.

    Motivation

    Ed tech news cycles often feel like fighting the last war to me. We’re always building our Maginot Line. Some vendor does something, and everybody rushes to write the hand-wringing story about how this might be just like the last disaster. The last trauma. Nobody asks, “What’s changed since then?”

    Analogously to default story trope with the LMS vendors, the default story tropes about the publishers all come from past traumas about pricing and don’t take into account how the economics of the industry have changed completely in the last decade, or how going from print to digital changes what it even means to update an edition.

    Nor do these trauma stories take into account what is changeable. They are typically written as if vendors are implacable forces of nature or all-powerful multi-national corporations. Pearson, one of the largest companies in the sector, is just 2.5% the size of Exxon Mobil. Even assuming the worst regarding the intent of the company management, the industry could not have maintained such high price points if faculty had simply started to select books based on price. Vendors respond to the signals customers send about what matters to them.

    Faculty had the power to change the industry. Whether knowingly or not, they chose not to exercise it. They chose not to even ask about the price in the majority of cases for many, many years. This is not to let companies off the hook for their decisions but rather to say that when we choose to retell trauma stories without interrogating them, we may miss opportunities to choose to play roles other than victims. And if you are going to choose to have a relationship with a vendor, why wouldn’t you choose for that relationship to be something other than victim?

    Five years ago, I wrote a post about how people who complained bitterly about how unhappy they were with their LMS vendors tended to ignore the procurement practices of their colleagues and their institution that all but guaranteed their dissatisfaction. That post was called Dammit, the LMS. It received quite a bit of attention. Even people who hated it admitted to me that they thought it was correct.

    Sadly, not much has changed since then.

  • OER Survey and Adoption Growth: It pays to check source material

    OER Survey and Adoption Growth: It pays to check source material

    I had a trip to the UK this month and only had time to read media coverage of the recent Babson Survey Research Group (BSRG) survey on open educational resources (OER). What a mistake. The Chronicle of Higher Education had a flawed description of a key question – actual and planned adoption of OER material by faculty – that misinformed readers like me who didn’t read the actual survey report, at least initially.

    The open-educational-resources movement, commonly known as OER, is an effort to encourage academics to use open-licensed materials in their classrooms as a way to lower costs. Some nonprofits, like OpenStax, have produced textbooks based on this material. The survey shows that OER has made inroads: 22 percent of people who teach introductory courses, subjects in which free textbooks are most commonly available, use it as required material, up from 15 percent last year.

    Yet the percentage of faculty members who say they will use, or consider using, open materials in the next three years actually dropped slightly, with the numbers now at 6 percent and 32 percent respectively.

    On the surface, this description indicates that OER adoption increased for faculty teaching introductory courses, but the second paragraph shows a potential drop in adoption over the next three years for all faculty. That would be major news showing that the OER movement hit an inflection point and is likely to drop soon, even though faculty have a natural affinity for the same issues that OER offers – lower cost and the ability to remix / reuse.

    It turns out this interpretation is wrong. The actual BSRG survey report states the following [emphasis added]:

    Each year, this survey asks faculty members who are not current users of open educational resources whether they expect to be using OER in the next three years.

    This question is only for non OER-adopting faculty. In other words, it measures growth potential, not total adoption potential. In fact, the percentage of faculty who used required OER material in any of their courses more than doubled this year, and based on the question above should continue to grow. BSRG further described the numbers for non OER-adopting faculty.

    There have been minimal changes in the proportion of faculty who report that they will use OER in the next three years, dropping slightly from 7% in 2015-16 and 2016-17, to the 6% reported this year. The number who report that they “Will consider” OER grew from 31% in 2015-16 to 37% for 2016-17, before dropping to 32% for 2017-18.

    This description is poorly worded and misses the context provided earlier, which I assume was part of the problem with Chronicle coverage.

    As for actual adoption, the category of “all faculty” grew significantly over the past year.

    Nearly one-quarter of faculty that teach large enrollment introductory courses report that they are using OER in some fashion, with more of these faculty responding that use OER as supplemental rather than as required materials. The rates are lower across all faculty, with 13% reporting using OER as required course material in at least one of their courses.

    These numbers represent a large increase over those in previous years, with the overall faculty rate of required OER use climbing from 5% two years ago to 6% in 2016-17, and then making a large jump to 13% this year. Given the sometimes vague understanding of the OER and its licensing, care must be taken in interpreting these results. Are faculty lumping any free resource into the OER category, even those that are not licensed as OER? Based on previous results we have to assume that there is some level of over-reporting in these figures of OER use; we just don’t know how much of an impact this is having.

    Chart showing growth of OER adoption

    What is interesting is that this adoption growth aligns with an independent source, the Cengage OER survey from Fall 2016.

    Open Educational Resources (OER) in higher education have the potential to triple in use as primary courseware over the next five years, from 4 percent to 12 percent, according to a survey of more than 500 faculty by Cengage Learning.

    What the data appear to show is significant growth in OER adoption for all faculty as well as for the subset teaching introductory courses. Adoption should continue, although it could be at a slower rate than was seen over the past year.

    It pays to read source material when the data describes important trends, especially when the results are surprising. I wish I had done this earlier.

  • Welcome Change: OpenStax using more accurate data on student textbook expenditures

    Welcome Change: OpenStax using more accurate data on student textbook expenditures

    Last week OpenStax, the Rice University-based publisher of open educational resource (OER) materials, announced that according to their data more than 2.2 million students at 48% of colleges in the US and 1,150 outside the US are using OpenStax free textbooks, saving an estimated $177 million.

    This is compelling data in its own right, and we are working on analysis around this organization and its model, but somewhat buried in the press release is another significant statement around what students currently spend on textbooks and what savings are possible with OER.

    “Our community is creating a movement that will make a big impact on college affordability. The success of open textbooks like OpenStax have ignited competition in the textbook market, and textbook prices are actually falling for the first time in 50 years.”

    As a result of the unprecedented downward shift in textbook prices, OpenStax will be decreasing its estimated student savings figure from $98.57 to $79.37 based on federal data. The U.S. Department of Education’s National Center for Education Statistics published a study in May stating the average undergraduate student spent $555.60 on required course materials for the academic year. Dividing that number by seven courses (the undergraduate average, according to enrollment data) comes out to $79.37 in savings for each student using an OpenStax book.

    I have long argued that OER groups and others arguing for making college more affordable should use baseline numbers based on what students actually pay for textbooks, rather than the all-too-common $1,220 – $1,420 per year numbers from a misuse of College Board budget numbers (see chart at top of page 10 in this document). With OpenStax moving to new federal data showing $556 average expenditures, we should start to see more reliable estimates of student savings. Kudos to them.

    However, this level of student spending should not be a surprise to anyone following the curricular materials market.

    Our 2015 post “How Much Do College Students Actually Pay For Textbooks?”, as well as a follow-up post, show in detail that we have had data for years showing that students roughly $600 per year on textbooks and related course materials, and that that number has been falling since at least 2008. Using data from the National Association of College Stores (NACS), we knew three years ago about the rough level of spending and the multi-year decline. NACS has continued to release annual updates, with the most recent public release from last summer:

    NACS data showing course material expenditure

    What OpenStax refers to, however, is the new National Postsecondary Student Aid Study (NPSAS) restricted-use data from the US Department of Education’s National Center on Education Statistics, showing $555.60 average student expenditures per year. Which is right in line with the NACS data.

    We plan to explore the NPSAS data in more detail, as it provides rich data for crosstabs and exploration of student expenses. But for now, kudos to OpenStax for this change in student savings estimates, even if it is years overdue. I would hope that other OER advocates would follow their lead.