e-Literate

Present is Prologue

Author: Michael Feldstein

  • The Crumbling of the OpenEd Coalition

    At the OpenEd conference this week, David Wiley made an announcement that was more significant than it may have sounded. Before I attempt to further characterize or analyze it, I think you should read it for yourself in its entirety, copied here from David’s blog: ((Disclosure: David is an employee of Lumen Learning, a sponsor of the Empirical Educator Project.))

    In 2003 I invited a small group of about forty people interested in open content and open courseware to Logan, Utah. Since then, this annual meeting has grown year after year to where we are today – 850 people interested in everything from open educational resources and open educational practices to sustainability and social justice. This annual conference has been a remarkable forum for the community to meet, share ideas, and foster collaborations, and the conference community is larger and more diverse than ever before. 

    With that growth comes change. That little meeting I convened 15 years ago has evolved organically in a way that has served the community fairly well. However, as currently constituted, the conference does not leverage all the energy, enthusiasm, passion, and leadership ability in our increasingly large and increasingly diverse community. And so the time has come for us to reconsider, as a community, how we wish to organize ourselves, learn from each other, and collaborate with one another. 

    In order to make the necessary space for that conversation, this year’s Open Education Conference is the last I plan to organize. As of this Friday afternoon the conference will be adjourned indefinitely. This is not a call for another person or organization to come forward to keep the same conference running the same way into the future. 

    Rather, it’s a call to reset and start over. To go back to the drawing board – as a community – and critically examine all of our assumptions about conferences, to grapple with a range of ideas about how we want to learn from and collaborate with each other, and to talk frankly about our end goals and guiding values. And when something like a consensus starts to emerge, the community can choose what to do next. To borrow a phrase, this is an opportunity to revise and remix what is, honestly, a very traditional academic conference, into something – or some things – far better.

    This reimagining must be owned by the community. It must be driven by the community. And it would be inappropriate for me to try to facilitate that process beyond extending a brief invitation. And so I invite all of you to use the breaks, lunches, dinners, and other free time you have here in Phoenix to engage in this critical reimagining and to explore with one another the possible shapes future meetings of our community – or communities – might take. I expect this will be a difficult, messy, and at times even painful process. But most things worth doing are.

    On a personal note, I want to thank the hundreds of people who have made the conference incredible over the years by presenting, reviewing proposals, convening sessions, volunteering at the information desk, and in countless other ways. Thank you to our Program Committees. Thank you to our sponsors. Thank you to Utah State University, Brigham Young University, BCcampus, and Lumen Learning for serving as the conference’s fiscal agents and logistics leads over the years. Most of all, thanks to each of you. Thank you for giving a bit of yourself to this community, for building each other up, and for moving the work forward. And, as always, thank you for everything you do for students.

    David Wiley

    I was at the conference for the first day and heard a range of different reactions to the news. Some said it was no big deal and that characterizing it as essentially a changing of the guard of the leadership:

    I think the framing of the conference as “ending” was not the best choice of wording. Really, the founder stepped down, that’s all. It’s new beginning for more diverse & inclusive model. And I say this as some1 who was participating in #oer since 2003 #opened19 #notgoinganywhere

    @lpetrides

    Others expressed more concern about the nature of the handoff from David to the next steward than Lisa did, but the fundamental view was the same.

    In my opinion, these reactions miss an underlying dynamic that makes this change more serious and difficult to patch over than it may appear on the surface. OpenEd was always at least partly an exercise in coalition politics. The attendees were a mix of people coming with different primary and secondary goals that overlapped enough for them to make common cause. That coalition has crumbled. In fact, it has been crumbling for some time. The idea that this conference could have been neatly handed off to some new steward as-is assumes that OpenEd is is otherwise tenable as-is. I don’t believe that is true.

    This is not ultimately about David Wiley or even Lumen Learning stepping away from organizing the conference (although the loss of an organization willing to manage an 850-person conference is a significant blow). My conversations with various OpenEd participants this week provided ample confirmation to me that there continue to be long-standing and deepening fractures within the OpenEd coalition, even though many of the participants are either not fully aware of them or not seeing how consequential they are in terms of how people may think about future convenings.

    I am by no means suggesting that OER or “open education”—whatever that is—is itself in danger of ceasing to exist. OpenEd is the only conference I know of that fell apart in a year when it had record attendance. If anything, OER adoption appears to be accelerating. In fact, I believe it is precisely OER’s success and growth that may have pushed this long-simmering problem to its breaking point. That growth has caused the balance of power within the coalition shifted in ways that created tensions among coalition members. Ultimately, those tensions proved insurmountable. The center did not hold.

    The OpenEd conference as we know it is dead. It may come back in some form, possibly even next year, and possibly even with the same name. But it won’t be the same coalition with the same focus and balance of interests. It won’t be the same OpenEd. That is neither necessarily bad nor necessarily good. But whether it is good, bad, or indifferent, it is a fact.

    This was foreseeable—and foreseen

    The OpenEd coalition has long consisted of (at least) three different groups with three different primary goals:

    1. Increase access to education by lowering cost of curricular materials
    2. Increase quality of education by increasing quality of curricular materials
    3. Promote values of education by fostering autonomy for educators and agency for learners

    Many—possibly most—of the OpenEd participants would likely say that they support all three of these goals. (I certainly do.)

    That’s good. Overlapping priorities are a critical success factor in building coalitions. But it’s not everything. Depending on how you interpret and rank these three priorities, your beliefs about strategy and values could be quite different. And there have long been signs that, in fact, there were very serious tensions among the views and priorities of the coalition members.

    In 2015, Phil Hill and I gave a joint keynote at the OpenEd conference in Vancouver. The theme of our talk was precisely that OpenEd was a brittle coalition that could fracture if the coalitional challenges were not addressed. Phil, in his part, talked about the challenge and opportunity that faculty surveys about OER demonstrated. There was a lot to be accomplished. My half of the talk was about my experience as a climate activist and how hard it is to build a coalition that holds together and accomplishes its goals over time (hint hint).

    Here are some of the questions that I encouraged the conference participants to discuss:

    • What is your goal?
    • What is your theory of change?
    • What is your strategy?
    • Who do you have to convince?
    • What are your obstacles?
    • What power do you have?
    • Who are your natural allies?
    • Who is persuadable?
    • What is your best message?
    • Who are you willing to let win?
    • What are you willing to give up?

    At the time, Mike Caulfield noted on Twitter that the question about who you are willing to let win was a particularly important one.

    Fast forward to a couple of weeks ago. A fight broke out about the planned conference programming that was bad enough to have caught the attention of the Chronicle:

    Less than two weeks before its 16th annual meeting, the Open Education Conference has canceled one of its keynote panels — “The Future of Learning Materials” — after facing a backlash on social media.

    The panel, which had been scheduled for November 1, was slated to include representatives from Cengage, McGraw-Hill, Lumen Learning, and Macmillan, all for-profit publishing companies, as well as the managing director of OpenStax, a nonprofit. It was supposed to explore the potential role of traditional commercial entities in the future of open education resources.

    “That role could be anything from ‘no role’ to ‘deeply committed participant,’” David Wiley, a member of the program committee and a co-founder of Lumen Learning, said in an email. Of the more than two dozen speakers and panels nominated for keynotes, the future panel was one of the top vote-getters on the program committee, he added.

    But the reaction to the panel highlighted the often contentious relationship between advocates for open education resources and commercial publishers, as open resources expand in the learning-materials market. The outcry also raised broader questions about the politics of providing platforms to those with opposing views and social media’s tendency to amplify outrage. Many open-ed advocates pushed back against the framing of the panel and objected to elevating profit-seeking entities with a keynote and prescreening questions audience members could ask.

    The conference’s program committee, comprising Wiley and 11 others involved in open education, said the decision to cancel had stemmed from “toxic behavior” on Twitter, adding that committee members had received “abusive and harassing” direct messages. Two panelists withdrew, according to a statement, and potential replacements declined to participate because of the tone of the discussion on Twitter. But some in the open-education community, both those who had pushed back against the panel and those who had stayed out of the discussion, said they did not see anything particularly troubling in the public posts.

    Wiley declined to share copies of the direct messages that committee members had received, or to comment on their nature. Several committee members and panelists did not respond to requests for comment.

    A Conference on Open Education Invited For-Profit Publishers to a Keynote. Then the Objections Began.

    To be clear, this latest episode is a symptom of the crumbling coalition rather than the cause of it. Phil and I saw this kind of tension—and occasional acrimony—as far back as 2015, which is what caused us to warn about it. Here’s one quote from the Chronicle article which marks one end of the spectrum of disagreement:

    In conjunction with the statement announcing the panel’s cancellation, the programming committee also released all of the questions that had been submitted to the panel. Of the 56 questions submitted, many were directed only toward the for-profit panelists and could be quite pointed. “Why should open-education advocates and OER publishers listen to the opinions of the commercial publishers, whose greed has directly caused the current textbook-cost crisis?” said one question.

    Others took the position that, while they were open in principle to commercial publishers participating, some of those entities did not meet key standards for good behavior and good will:

    Rajiv Jhangiani, associate vice provost for open education at Kwantlen Polytechnic University, in British Columbia, published a blog post outlining how he thought commercial entities had engaged with open education maliciously in the past, and how many of them were trying to improve how they work with the field.

    “While this may seem like a tricky balance, I see it as quite straightforward to criticize openwashing” — marketing as open education without offering a fully open product — “by a commercial player while also recognizing positive developments from the same actor,” Jhangiani wrote. “You see,” he wrote later in the post, “I do want to have these discussions with commercial players. I am interested in a diverse and healthy commons, and I take no joy in skewering for-profit actors publicly when they perpetrate harm and lie to advance their bottom line.”

    While it is not a 100% match, OpenEd coalition members’ views of this sort of question—who they are willing to let win (and under what circumstances)—tend to correlate with primary goals. On one end of the spectrum, those who see OER as a particularly effective tool for achieving their primary aim of improving quality of curricular materials (and therefore student outcomes) are likely to be most tolerant of commercial vendors whose tactics they don’t always agree with but who may contribute in multiple ways to improving student outcomes, both through OER and through other means. On the other end, coalition members with different primary goals tend to be least tolerant of the commercial vendors and less forgiving of their perceived bad behavior, in part because these advocates tend to hold a theory of change that is tied to correcting power imbalances. For example, the claim that the publishers’ “greed has directly caused the current textbook-cost crisis” entails the beliefs that the publishers have had all the power to control prices and that correcting the fundamental problem requires taking away power from the publishers. Different primary goals or different theories of change may lead people to different conclusions about who they are willing to let win under which circumstances.

    Making matters more difficult, as OER moved from a fringe interest to a more mainstream trend, the balance of attendees at OpenEd has changed from a high percentage individual contributors who sometimes felt alienated from the decision-making processes in their home institutions toward a higher percentage of participants who came precisely to advance institutional initiatives. As is often the case in coalitional politics, it is impossible to completely separate the political from the personal. People attending OpenEd come with both their professional responsibilities and their personal convictions, which tend to be deeply enmeshed among people who are passionate about what they do for a living. As the coalition came under increasing tension from the changing mix in priorities, personal tensions were also bound to increase.

    While I did not follow the public fight about the panel on Twitter and have no inside information about any private messages that were exchanged, Phil and I had both seen enough similar behavior by 2015 to have been concerned about whether this group would hold together over the long haul.

    What may come next

    As I have tried to say throughout this post, the current situation is not a signal of OER’s failure but rather a side effect of its success. Note that none of the three priorities I listed at the top of this post included “OER are intrinsically good and an end in themselves.” Even folks who tend to believe that openness is an important value that leads to other good things—and I count myself as one of those people—argue that a Creative Commons license is an end in itself. An open content license or, more broadly, a bias toward openness of various types can be useful in accomplishing a wide range of goals. And because interest in using OER has increased across a range of people who are pursuing different goals, it is no longer adequate to talk about “open education” and “open pedagogy” as nebulous things that we don’t all need to define and explicitly agree on because it is somehow obvious that we all agree enough. That simply isn’t true anymore, to the degree that it ever was true in the first place.

    The OpenEd conference as we know it is dead because it represents a coalition that has, at least for the moment, crumbled under the weight of its own growth and diversification. What comes next will almost certainly be one or more attempts to either reconstitute the old coalition or build new ones—or, most likely, both.

    I think it more likely than not that somebody will run a conference next year that will be called “OpenEd.” I also think it more likely than not that there will be one or more conferences—regional, national, and/or international—that attempt to build a new coalition that aligns more closely with one or more factions rather than trying to reconstitute the same blend as OpenEd as it has existed to-date. The efforts that succeed will be the ones that embrace the task of coalitional politics and develop a strategy that both recognizes the real differences among participating factions and develops an explicit strategy to attract and retain a coalition that can maintain alignment over time.

    There will doubtlessly be OpenEd participants who passionately disagree with this analysis, and some who may even be angry over it. Unfortunately, that is new to neither OpenEd nor e-Literate. A public platform is a blunt instrument. Sometimes the only way to both ethically and effectively get a message across in a venue like a blog is to be plain spoken and independent-minded at the risk of offending friends.

    The OpenEd coalition fell apart this week. It is not the end of OER or open education. But there are multiple paths leading forward from here. Finding the one that maximizes opportunities to do good will depend on the degree to which future coalition members are willing to take a hard look at what went wrong and learn from failures.

  • Announcing the Standard of Proof Webinar Series

    Announcing the Standard of Proof Webinar Series

    Licensed CC-BY-SA by The Blue Diamond Gallery

    In a recent post, I wrote about my experience on the EDUCAUSE exhibition hall looking for vendors with proof that their products actually help students and how this is an example of the more general problem:

    Right now, higher education has very poor signals to quickly distinguish between those vendors who can prove that their products are effective and those that can’t. Between those vendors who are contributing to the general state of knowledge about how to improve student success and those that aren’t. Until we can improve the signal-to-noise ratio, we are doomed to ride the roller coaster from hell that is the hype cycle until we pass out from exhaustion.

    I’m going to do something about improving the signal-to-noise ratio, and I will be announcing what that something is within the next week.

    Today I am pleased to announce the Standard of Proof ongoing webinar series, which attempts to do just that. The first is scheduled for November. But the plan is to have at least one a month starting in late January, after folks have returned from winter break.

    How webinar topics are selected

    For many years now, I have turned down requests to facilitate paid webinars because it’s very hard to do them without sliding down a slippery slope and either appearing to be a shill or actually crossing over the line and becoming one. And yet, I would like to give webinars that highlight genuinely good work, and I would like to get paid for my time so that I can afford to do this work.

    The Empirical Educator Project (EEP) has given me a novel way to thread the needle. Vendors don’t pay me directly for a webinar. Instead, they sponsor EEP. I vet prospective EEP sponsors—they have to have something to contribute beyond just money—and I also price sponsorship to incentivize active participation and contribution. Standard of Proof webinars are now another incentive for vendors to contribute.

    A Standard of Proof webinar is not an automatic benefit of EEP sponsorship. Each webinar is about not a company but a contribution to the work of empirical education. It can be, and often is, novel research conducted by the company that (a) contributes to our knowledge of how to help students succeed, and (b) adds to our knowledge in a way that is generalizable beyond their product or service. In other words, it is a contribution to the commons. But the contribution can also be a project that helps disseminate established research-backed practices or generally promote the culture of empirical education. A contribution should either be free if it has no maintenance costs or non-profit if it has does have a cost to maintain.

    Contributions also have to have been developed with some meaningful collaboration with or peer review from academics. As with the definition of “contribution” above, I am more interested in the spirit of meaningful collaboration than in setting up narrow rules. But there has to be an academic who was involved enough in the project that she or he is willing to speak publicly about their participation and their views of the project.

    While EEP sponsors are not guaranteed a minimum number of webinars in return for their sponsorship, neither are they limited to a maximum number. If they have made multiple contributions and I have time to cover them in the Standard of Proof schedule, I will. If that means running more than one webinar a month, I would be happy to have that problem.

    The end goal here is to help vendors that meaningfully contribute to education differentiate based on those contributions. Providing credible evidence that their solutions support student success is part of that work, but the bar is higher than that. The webinar series title, “Standard of Proof,” refers not only to the products but to the vendors. The series highlights acts of good citizenship that promote empirical education. By raising the profile of these acts and the vendors who perform them, I hope to influence the criteria by which prospective vendors are selected and therefore their incentives for behavior.

    Folks, this only works if you and your institutions decide that you will prioritize good behavior—including but not limited to credible proof that a product is effective—in your vendor selection. The most that I can do is help make that good behavior a little more visible.

    The first webinar

    As I wrote earlier in this post, the first webinar coming up later this month. I have two more lined up for early next year that I will be announcing at a later date and more in the pipeline.

    Summer melt: A randomized controlled trial

    We’ll be kicking off the series a randomized controlled trial by AdmitHub and its academic partners to advance our understanding of a critical phenomenon that influences the success of first-generation students.

    Folks involved with admissions at access-oriented schools may be familiar with the term “summer melt.” It’s when students are admitted to a university, say they are coming, and then never show up. And it tends to be a first-generation student problem. There are all kinds of logistical and bureaucratic hurdles in getting from admissions to first day of classes that may be harder for first-generation students, either because the processes are not designed to accommodate their situations or because they don’t have anybody in their lives who can help them. Maybe they need an immunization certification and they don’t know how to get it. Maybe they need the signature of a parent who they don’t know how to contact. This is a problem that is often caused by a number of small contributing factors, and it is hard to diagnose because the people who know what kept them from coming to college…never came to college. So having a dialogue with them after the fact is difficult to arrange (not to mention unhelpful for the student, who has already missed an opportunity).

    There are a number of products on the market, particularly machine learning-enhanced chatbots, that purport to address the summer melt problem. AdmitHub is one of them. But rather than just claim that they can help, they collaborated with their major customer, Georgia State University (GSU), and an academic researcher from the University of Pittsburgh who studies summer melt, to conduct a rigorous study of the impact of the AdmitHub-enabled interventions on summer melt at GSU. The results of the study? GSU was able to reduce summer melt by 20% using AdmitHub.

    In addition to providing credible evidence for the product’s effectiveness, this study’s results and experimental design both contribute to our general knowledge of summer melt. We’ll be talking about that as part of the webinar, which will include the PI Lindsay Page from Pitt as well as Tim Renick from GSU, who happens to be a personal hero of mine.

    Register for the AdmitHub summer melt RCT webinar on Monday, November 18th at 2 PM ET.

  • What I Didn’t See at EDUCAUSE

    I often get asked what themes I see coming out of EDUCAUSE. The last few years, the theme has been “there is no theme,” perhaps reflecting the fact that ed tech hype has been in remission. With the exception of the OPM product category, whose hype cycle from zero to peak to trough may have been the fastest we’ve seen yet, the market seems tired of the endlessly repeating roller coaster ride of hope and disappointment.

    This year was a little different. Overall, the conference was the healthiest that I’ve seen it in a while. Attendance seemed to be up. The balance between IT and learning presentations seemed good. There were more vendors on the floor, with more big booths (but no magicians, jugglers, or sword swallowers, thankfully). Overall, the organization seems to be undergoing a revitalization under John O’Brien.

    Within that big picture, there was some color. The major textbook publishers had very little presence, perhaps reflecting their current financial state or perhaps indicating that they’ve figured out the EDUCAUSE crowd isn’t really their target audience, for the most part. On the other hand, ERP/SIS vendors were back with bigger booths, including some names I didn’t recognize (like Unit4, which apparently has been around for a while but which hadn’t heard of before this year).

    There also was a proliferation of small vendors with various flavors of solutions aimed at improving student retention and graduation rates. Each one had its own take on the problem. One had an integration dashboard, pulling in data from the LMS, SIS, and other applications to tell advisors when students might be struggling and need encouragement. Another took a similar approach but focused on giving students nudges directly. Yet another focused on engaging students on social media. And so on.

    Whenever I saw one of these companies, I tried an experiment. I’d do a quick scan of their booth posters and video screens. Just enough to get a very rough sense of what they claimed to do. In other words, I looked about as carefully as the typical exhibition floor browser looks. Then I would approach the booth and say, “I have a rough sense of what your product does from what I see on your booth.” I have one question for you: How do you know that it works?”

    I got a range of different answers.

    “Our customers love it.”

    Bzzt. Wrong answer.

    “We have dashboards that show 67 points where students could be getting bogged down.”

    “OK, but how do you know that it works? I mean, have you tested it to see if it actually impacts outcomes?”

    “…Well…we have a white paper.”

    Pass.

    “We have a customer who improved their retention by 60%! But to be truthful, our product was one of a number of initiatives they put into place, so I don’t know how much of that improvement they would attribute to it.”

    Close, but no cigar. Points for honesty, though.

    “We have conducted multiple randomized controlled trials.”

    “Oh, really! What result were you testing against?”

    “Oh, right, of course. Year-to-year retention…”

    “Thanks, you can stop there for now. May I please have your business card?”

    I found one—just one—vendor who appears to be able to deliver the goods on evidence. I wouldn’t have picked up on that from my cursory scan of their booth. They didn’t look any different than any of the others I had spoken to. In fact, if you had asked me which of the products I looked at would have been likely to prove out as effective, I wouldn’t have ranked them near the top of the list. The trick that they advertise most heavily, which is that they repurpose existing advertising mechanisms on popular social media platforms to deliver productive nudges, seems clever but a little disconnected from problems specific to retention. While they mention “behavioral science” on their web site, they don’t provide a lot of detail. That said, if you read their marketing copy carefully, you’ll see that they do provide a rather stripped-down, easy-to-follow explanation of a randomized controlled trial that they conducted.

    The name of that company, by the way, is Motimatic. Have you heard of them? Because I haven’t. I’m going to look into them and find out more. Maybe you should too.

    Right now, higher education has very poor signals to quickly distinguish between those vendors who can prove that their products are effective and those that can’t. Between those vendors who are contributing to the general state of knowledge about how to improve student success and those that aren’t. Until we can improve the signal-to-noise ratio, we are doomed to ride the roller coaster from hell that is the hype cycle until we pass out from exhaustion.

    I’m going to do something about improving the signal-to-noise ratio, and I will be announcing what that something is within the next week.

    Stay tuned.

  • The MOOC-Courseware Convergence

    Now that Coursera for Campus—which I have occasionally erroneously referred to as “Coursera for Schools” in previous blog posts—has been launched, we have some more information about what it’s really about. (I’ll embed the launch video at the bottom of the post, but if you’d rather go straight to it, you can find it here.) I think it’s significant and portends significant trends in the sector.

    Before we get to that, let’s get one item of business out of the way related to my previous posts. As IBL Education notes, Coursera CEO Jeff Maggioncalda said, “Coursera for Campus is not a full-featured LMS. We expect many universities to stay on their LMSs.” IBL News further notes, “According to the company, Coursera for Campus’ LMS is designed to supplement the existing Canvas, Blackboard and Moodle systems.” [Emphasis in original.]

    So Coursera for Campus isn’t LMS play, at least for the foreseeable future. I was pretty harsh on Coursera’s SVP of Enterprise Leah Belsky for her disruption language, but as I’ll get into later in this post, this kind of category mistake is actually quite easy to make and one that other courseware providers have made.

    Because that is exactly what Coursera has explicitly become with their Courseware for Campus offering: a courseware provider. The less obvious part is that MOOC design and courseware design have been converging for some time now. Their increasing similarities in instructional affordances have been masked by the differences in the business models of their parent companies. And even less obvious is that the Venn diagram of courseware companies and OPM companies is starting to overlap significantly.

    “A 21st-Century textbook”

    In the launch presentation, Maggioncalda called Coursera for Campus, in part, a “21st-Century textbook.” Take that seriously. These are MOOCs repurposed as curricular materials. And it’s really not that big of a leap. Remember that the pedagogical model of the xMOOC course generally doesn’t require heavy participation from the instructor. Some instructors do participate heavily, while others, less so. A lot of instructor energy goes into course design and lecture video production. The actual live support could be from the instructor, but it also could be from TAs, or it could be self-study. Or, in the case of Coursera for Campus, it could be a different instructor. From its inception, the design model of xMOOCs began decoupling faculty course design from faculty course delivery.

    There is some messiness, of course. The biggest challenge in terms of textbook-like may be the lecture videos. Faculty may not love having some other personality featured as the star of the show. But there are two mitigating factors to that. First, Coursera’s design staff is likely guiding faculty toward authoring instructionally sound videos, which would tend to reduce the total amount of talking head content in any given course. Second, textbook-provided commercial courseware, which is still often written by star authors in their field, now also features talking heads. Here, for example, is the Cengage MindTap introductory video for Economics, by author Greg Mankiw:

    Greg Mankiw’s head, talking

    “But,” you may be thinking, “MOOCs are not designed like courseware.” One of my Twitter followers made a comment to the effect that at least courseware has an instructional design philosophy, unlike xMOOCs.

    That person is mistaken, and here is the proof:

    These analytics are only possible with backward design

    During the launch presentation, a Coursera executive made the statement that “any of the courses that have been authored on Coursera come with an out-of-the-box analytics platform.” If those analytics look anything like the picture above—and I would bet money that they do—then the courses have been built on a backward design philosophy like the one I have been describing in my recent post series on content as infrastructure.

    This shouldn’t be surprising. As I wrote repeatedly throughout that series, almost every professionally designed course uses that design pattern. And Coursera is known for having strong professional course design support.

    (I’m telling you, folks. That content pattern is the revolution of our time.)

    Update: Matthew Rascoff has reminded me to acknowledge that the professional course designers on campuses play large and critical roles in the design of these courses. The larger point is that, on both sides of the fence, there are trained, professional course designers who are applying this design pattern.

    While I haven’t looked at their catalog myself in quite some time, I would not be surprised at all if many of their offerings stack up pretty well against commercial courseware titles. For one thing, while xMOOCs have a bad reputation for anemic social interaction relative to other course models, they probably have more social interaction designed into them than many commercially published courseware titles.

    Courseware platforms vs LMSs

    It’s easy to get confused about the boundary between a courseware platform and an LMS. I know because I have worked on and consulted on both. The LMS companies inevitably start thinking, “Damn, there are so many courses that get delivered through our platform, and so much money made on selling them. And guess what? Everything that faculty build and deliver through our product is a course! The textbook publishers keep rebuilding what we’ve already built. Badly. I mean, have you seen their grade books? Why can’t we be the platform and get a cut of all that money?”

    So they try to displace the publishers. And they fail. Every time.

    The textbook publishers, meanwhile, start thinking, “Damn, our customers really hate their LMSs. We’ve built all these LMS features, and they keep asking for more. I mean, we’ve already built a grade book. Wow, that was painful. Why can’t we be the platform and get a cut of all that money?”

    So they try to displace the LMS companies. And they fail. Every time.

    Here’s the critical difference:

    LMS developers have to optimize for a wide range of faculty preferences, teaching styles, and teaching conditions. They have to accommodate every grading scheme imaginable. They have to handle huge classes and tiny classes. They have to deal with face-to-face, online, and blended. Constructivist, lecture, and whatever else. That’s why they spend so much time adding grade book micro-features and then optimizing the usability to handle all those micro-features without being totally overwhelming.

    Courseware developers, in contrast, have to optimize for the content. If the subject is software development, then you need an interactive code editor and test engine. If it’s accounting, then you need a test engine that looks like a spreadsheet. If it’s chemistry, then you need a molecule visualizer and manipulator.

    Which side of the divide do MOOC platforms land on? Here are a couple of slides from the Coursera on Campus launch:

    In-browser coding!
    Virtual labs!
    Data science notebooks!

    These subject-specific affordances, plus competency-based analytics, were the platform highlights of the Coursera for Campus presentation. Not the grade book that can do anything. Not the test engine that can provide any kind of feedback. Not announcements or an event feed. This was all about courseware.

    “But wait,” you say. “Courseware for Campus lets faculty author their own courses. Isn’t that different?”

    Yes. And no.

    Way back the better part of a decade ago, when I was at Cengage working on the MindTap platform, the company debated whether to open it up to customers and license it without content. (“We can disrupt the LMS!”) In the end, there were two major barriers. First, as a print-centric company in the midst of a transition, the authoring tools were not remotely faculty-friendly. And second, as a publisher whose bread and butter came from royalties, there was a fear of cannibalization of the business.

    Coursera has neither of those problems. It was born as a two-sided market, which means that it never owned the content to begin with and always had an incentive to make authoring as easy as possible. It may have taken some time to fully realize that vision, but we were destined to arrive where we are now.

    Further, as textbook publishers increasingly move away from celebrity franchises and toward fee-for-service contracts with their authors, they will have motivation to make similar moves. McGraw-Hill Education doesn’t advertise it widely, but they have been licensing the authoring platform for SmartBooks for several years now. Authoring support can work with a courseware platform as long as the range of course expectations for delivery models can be constrained. And MOOC courseware fits the bill. It’s a genre.

    Coursera for Campus is a harbinger of the future, not for the LMS industry but for the textbook industry. And they are an early mover with certain advantages in their business model.

    The courseware/OPM convergence

    One critical element that I don’t want to lose in all of this is the implications for the OPM market. Let’s not forget that (a) Coursera has been pushing into that market aggressively and successfully and (b) that market has been under massive pressure and upheaval lately. 2U has been the canary in the coal mine, having lost roughly four fifths of its market value since the beginning of the year. ((Disclosure: 2U is a 2019 sponsor of EEP.))

    To be clear, I think there’s some noise obscuring the signal. Two of the reasons why 2U took such a big hit are that the stock got way ahead of itself and some self-interested players have been extraordinarily successful at generating FUD around the market category in general.

    That said, there is no question that the hype around selling an infinite number of $40K masters degrees has met its demise. There are two barriers to OPM growth, which is another way of saying that there are two barriers to conventional online degree growth, and they have both proven formidable barriers to crack. The obvious one is cost. The less obvious one is geography. It turns out that, even in an era when people can take courses from anywhere in the world, they will tend to take them from their local institution or not at all. For all the talk of “national universities” and “mega universities,” it’s not clear that such beasts really exist. For the most part, big universities have proven exceptionally good at soaking up every ounce of demand for education in their local areas. So the next sustainability play is not so much about reaching students far away as it is about serving students you already reach for 40 years rather than for four.

    And interestingly, that is essentially the pitch that the Coursera executives made about Coursera for Campus—even when they were pitching in India. They weren’t making the pitch that the American stereotype would have predicted, of reaching the rural millions. They were talking about lifelong learning. Skilling and reskilling. From an OPM perspective, this pitch gives the company with the large catalog of low-cost and constantly refreshed inventory a competitive advantage.

    That said, there was definitely a bit of hand waving going on regarding completion rate. The Coursera executives talked about University of Illinois’ degree program as having over a 90% completion rate as if that remarkable achievement could be solely attributed to the fact that it was a degree program. I suspect there are some hard-working support staff at the University of Illinois who might quibble with that clean of a causal analysis. I don’t think the Coursera folks are directionally wrong, and I don’t think they were intentionally misleading, either. But I do think that they oversimplified, and that running a highly successful program at scale with a 90%+ degree completion rate entails a lot more than just handing out a piece of sheep skin at the end. The reason I bring this up is not because I want to snark on Coursera but because the part that is being glossed over represents a number of support areas that OPMs compete on (and that more traditional OPMs and OPEs pride themselves on excelling at).

    The gap between “MOOC” the course model and “MOOC” the courseware model is very much an open question in terms of student success. The MOOC courseware model, particularly as implemented into affordable degrees at scale delivery models by pioneers like Georgia Tech and University of Illinois, are creating an alternative delivery model that could start putting pressure on OPMs supporting more traditional models. Coursera on Campus, in addition to creating an additional revenue stream for the company and putting pressure on courseware providers, potentially ups the ante in the OPM market. But we need to look to those universities which are pioneering affordable degrees at scale to understand the service gaps, marketing gaps, and cost differences between MOOC courseware and the totality of what they are doing in order to understand the what it would take to replicate their success.

    In the meantime, consider the implications of the Coursera on Campus model for the future of companies like Pearson and Wiley, which own both courseware and OPM divisions, or McGraw-Hill Education, which has begun experimenting with opening up its platform for content authoring by customers. ((Disclosure: Pearson is a 2019 sponsor of the Empirical Educator Project.)) It’s not necessarily all bad, but it is a potential accelerant to change that is already in the wind.

    It’s a wild, wild, wild, wild world, my friends.

    Here’s the full launch event for your viewing pleasure:

  • Supporting Equity Doesn’t Mean Spending Blindly

    According to an article in Inside Higher Ed, California just modified its $475 million Student Equity and Achievement Program “to allow the funds to be used for emergency student aid.” Since these changes don’t entail new funding, much of the article was dedicated to hand-wringing about whether diverting existing funds from other priorities “like tutoring, peer-mentoring programs and equity-focused professional development for faculty” is, on balance, a good idea.

    On the one hand, there is evidence that giving students emergency financial support is both a needed and an effective intervention:

    Colleges in California and across the nation have created their own emergency aid programs. A Senate analysis of the bill notes that Pasadena City College and Grossmont College both fund their programs through external sources like foundations and fundraising.

    Amelia Parnell, vice president for research and policy at the National Association of Student Personnel Administrators, said the association found in a survey that most colleges feel they aren’t fully meeting students’ emergency financial needs.

    “Because emergencies are typically unexpected, it’s hard to find the right balance that’s needed,” she said, adding that she thinks the spirit of the bill is “consistent with what a lot of campuses have said.”

    According to the Senate floor analyses, Chiu cites as support a February 2017 report from the Institute for College Access and Success on college costs for low-income California students. The report found that low-income students at public colleges in California can’t afford college costs with the available grants, their own resources and some working income.

    It also found that community colleges sometimes have a greater net price for low-income students than four-year public schools due to the limited amount of grants available for community college students.

    Chiu argued that research shows emergency aid can keep students enrolled through unforeseen challenges.

    Research does show that emergency aid can keep students enrolled. See, for example, Georgia State University’s Panther Retention Grants.

    But on the other hand, the other interventions that the $475 million California program has been funding up until now are important too.

    However, the Senate grappled with questions of whether the Student Equity and Achievement Program funds would be best used for this purpose. The analysis asks if the bill would “set a precedent that dilutes student equity funds intended for critical academic support service,” and if expanding state financial aid programs would be more appropriate.

    The Senate Appropriations Committee said the bill could redirect funds away from other student support services, which could lead to “potentially significant … cost pressure” to maintain the state’s current level of student support services.

    What the article doesn’t mention is whether the legislature funded any significant research, either previously or going forward, that will help guide the colleges regarding which investments are likely to be most effective in meeting their equity goals. Because that’s the question, right? Colleges have options to spend their money to best serve their students. And given the total amount of money in play across the system—nearly half a billion dollars—one would think that a small amount of money invested in research would be a wise allocation of funding.

    Maybe it’s in there and just not mentioned in the IHE article. I hope so. Past experience with the California system suggests that (a) the legislature doesn’t think this way and (b) the California Community College System is not set up well to execute programmatic research of this kind even when they are given the funding and prioritization to do so—in part because they are not given the funding and prioritization to do so as often as they should be.

    If there is policy uncertainty about a consequential matter that impacts students in a meaningful way, then that risk should be approached with an experimental mindset. If you aren’t mindful about assessing the impact of different choices, then you’re just throwing dice.

  • MOOC Die-Off Coda

    In response to my last post, my friend Matthew Rascoff tweeted,

    https://twitter.com/mzrascoff/status/1180303464951250944

    Matthew makes some valid points, both here and further down in that thread. So I thought it would be worthwhile to take a moment to clarify a couple of the finer points of my post.

    First, I absolutely agree that there is a long-running narrative of “MOOCs are dead” that is not productive. My intention wasn’t to reiterate that narrative but rather to say that, given the very evolution of what is “coming next” that Matthew is interested in, the term “MOOC” itself has become meaningless. Design and usage of “MOOCs” have fragmented in really substantial ways as people have learned and tried new things. To call Georgia Tech’s Affordable Degrees at Scale, ASU’s Global Freshman Academy, and the new Coursera on Campus initiative all “MOOCs” or “MOOC-based” is to paper over the critical differences among them that are exactly the experimental factors that we should be attending to. Of these three programs, only Global Freshman Academy was something close to what I would call a MOOC in the original sense.

    The point isn’t to get into an academic debate (in the pejorative sense) over a term definition but rather to understand that there are very different kinds of experiments being run. The success or failure of one of these tells us very little about the prospects of the other two. When we call everything a MOOC, we obscure the differences among the experimental conditions and the possible reasons for the different results that are coming in from them. There are lots of interesting experiments happening on MOOC platforms right now. I’m not sure how to tell what works and what doesn’t because we’re not using precise language to disambiguate among the experiments. Die-offs are part of evolution, and we’re trying to understand the process of natural selection.

    Second and somewhat separately, I have a bugaboo about the term “disruption” being casually thrown about by ed tech companies. Disruptive innovation is a theory of change. We have lots of data about how the LMS market in particular changes—or doesn’t. At one point, Moodle was supposed to be a disruptive innovator. At another point, OpenClass was supposed to be the disruptive innovator. Both competed against non-consumption with a cheaper, simpler solution. Both got soundly beaten in developed markets by Instructure Canvas and, later, by other commercially hosted cloud-based solutions. There is no particular reason to believe that disruptive innovation works as a theory of change in the LMS market.

    Further, if you’re going to claim to be ready to “disrupt” a product category, then you damned well better have good understanding of the needs of the customers in that product category. It’s a matter of respect. There is a long history of vendors failing to show proper care in this regard, which in turn has created trauma and fostered dysfunction on the part of universities going through a procurement process. Those universities tend to develop a cynicism which, at its worst, manifests as a corrosive political nihilism in their approach to picking a product.

    Vendors have a responsibility to the sector to be humble. I don’t have any problem at all with Coursera’s ambitions for their Coursera at School offering, at least as far as I understand them. I do have a problem with the way that their executive characterized those ambitions in the EdSurge article. I think the language of disruption is both counter-productive for Coursera and harmful to the kind of conversations we want to foster in general about evaluating innovative offerings. As I said in my original post, it is long past time for companies serving the education market to strike that word from their public vocabularies.

    In general, I am spending less time picking at vendor language in this new incarnation of e-Literate, as I try to focus on incentivizing good behavior rather than punishing bad behavior. That especially applies to vendors using unfortunate but relatively inconsequential language. What Coursera is doing is generally much more important that what they are saying about what they are doing. That said, there are times when patterns of language use are genuinely harmful and need to be called out, not to punish a bad actor but to put a halt to counter-productive behavior that retards progress in the sector. This is one of those times.

  • Disruption Disrupted: The Great MOOC Die-Off

    Coursera has announced, with some fanfare, their Coursera for Campus initiative, which Jeff Young at EdSurge has characterized as an attempted entrance into the courseware market but which Coursera Vice President of Enterprise Leah Belsky described as aimed at the LMS market.

    From the EdSurge piece:

    Coursera for Campus is designed specifically with colleges in mind, says Leah Belsky, Coursera’s vice president of enterprise. That means the service includes new features tailored for use in an academic environment, including plagiarism detection to spot cheaters and integration with existing student gradebooks in the learning management systems (LMS) that colleges use.

    Meanwhile, Coursera is opening up its technology platform to any college to use for free to deliver course materials on their own campuses. That means that colleges could use the Coursera software as an alternative to their learning-management system. Belsky argues that Coursera’s system is better designed for delivering online courses and interactive lessons than most LMSes. 

    “We’re talking about a potential major disruption to the LMS market,” she says. “We don’t have all the features of an LMS but what we do have is all the tools to create cutting-edge interactive learning experiences.”

    This is bad framing from a PR perspective, but more importantly, it just plain misses the real potential value proposition and chases a plainly imaginary one instead.

    First, I hear a constant stream of complaints about both major MOOC providers having platforms that are not even fully adequate for their original purpose (though the situation does seem to be improving, particularly in terms of data analytics). At least one very high-profile customer, who I won’t name here, uses the MOOC platforms basically as a store front while putting their actual courses in an LMS. This comes at a time when some Blackboard customers still balk at switching from the classic to the Ultra experience because of feature gaps on level of granularity of test question feedback. MOOC platforms are interesting and have some innovative features, but they are neither mature for their original purpose nor tuned for the broad range of usage that a campus LMS must serve. Second, disruption talk is particularly tone deaf from anyone in a product category that was very recently known for hyping that they would be disrupting the university itself. And finally, disruptive innovation is an unfalsifiable theory that has thus far shown itself to have no predictive power in higher education. It’s a provocative idea that may have some generative intellectual value, but really, it’s well past time for every company that has aspirations in education to drop the word “disrupt” from their public vocabularies. Coursera for Schools may well have a decent value proposition, but this isn’t it.

    Jeff’s framing is closer to the truth: This is primarily a courseware play. There’s a little bit of gray area because courseware platforms and LMS platforms are slowly drifting toward each other in terms of design, but if they ever do converge, it won’t be in the next couple of years. No, this is about repurposing content. And the real story here is that the content needs to be repurposed because we have an overpopulation of MOOCs that are in the midst of a die-off. I’m not saying that MOOC companies are dying off. As far as I can tell, Coursera seems to be healthy. (I have less visibility into EdX’s financial status.) What I mean is that previous generation of the Stanford/MIT/Harvard-style xMOOCs, having failed to achieve either their mission or their sustainability goals, are now being repurposed into other things. Because we don’t have better names for those things, we still call them “MOOCs.” But they don’t meet the definition of Massively Open Online Courses. Even the Stanford/Harvard/MIT definition.

    Meanwhile, there are zombie MOOCs on these platforms that are in the process of getting killed off. Not too long ago, one campus stakeholder told me that their MOOCs basically serve the same purpose as their YouTube marketing videos, except that the YouTube videos get much better viewership and cost a lot less. I don’t expect the MOOC to die entirely, but two years from now, there will be a lot fewer of them than there are now. We just may not recognize that change if we insist on continuing to call any online enrollable thing with more than 30 students a “MOOC.”

    The known failure

    What’s weird is that everybody has known that the xMOOC was a failed experiment within 12 months of it reaching peak hype, and the widely known evidence has only mounted since then. In a January 2019 article in Science that was tellingly titled “The MOOC Pivot,” authors Justin Reich and José A. Ruipérez-Valiente write in the summary,

    When massive open online courses (MOOCs) first captured global attention in 2012, advocates imagined a disruptive transformation in postsecondary education. Video lectures from the world’s best professors could be broadcast to the farthest reaches of the networked world, and students could demonstrate proficiency using innovative computer-graded assessments, even in places with limited access to traditional education. But after promising a reordering of higher education, we see the field instead coalescing around a different, much older business model: helping universities outsource their online master’s degrees for professionals (1). To better understand the reasons for this shift, we highlight three patterns emerging from data on MOOCs provided by Harvard University and Massachusetts Institute of Technology (MIT) via the edX platform: The vast majority of MOOC learners never return after their first year, the growth in MOOC participation has been concentrated almost entirely in the world’s most affluent countries, and the bane of MOOCs—low completion rates (2)—has not improved over 6 years. [Emphasis added.]

    The entire (paywalled) article is worth reading, but honestly, is any of the above a shock to you? xMOOCs, as originally designed, are not replacements for face-to-face classes. They do not lead to reliable course or credential completion. They do not do a good job of serving underserved populations. And they do not create sustainability models by giving a way expensively produced courses and making up for the cost on volume. They have value for some folks. I’m glad they exist in the world. But as an alternative system of education, they are a failure.

    We see this proven out again and again, in multiple variations. Most recently, ASU has largely shut down their Global Freshman Academy, a large undergraduate MOOC experiment designed to give students credit for their first year of college and attract them to matriculate to ASU for a degree. From the IHE article:

    Of 373,000 people who enrolled, only 8,090 completed a course with a grade of C or better, just over 2 percent of all students enrolled. Around 1,750 students (0.47 percent) paid to receive college credit for completing a course, and fewer than 150 students (0.028 percent) went on to pursue a full degree at ASU.

    Was Global Freshman Academy an experiment worth running? Absolutely. In fact, I fervently hope that ASU will be more forthcoming than they have been so far with the lessons they have learned from the experience. Did students in those courses gain value? While that article doesn’t provide significant data on this question, I strongly suspect that many did. Were MOOCs an effective vehicle for saving freshmen roughly 25% of the cost of a college tuition while still getting them on to sophomore year? Clearly not.

    So what has worked as a MOOC-like alternative to the traditional degree? What’s the closest we’ve come to fulfilling the original vision? The most high-profile success has been Georgia Tech’s affordable graduate degrees at scale. Note: I have not seen them refer to these courses as “MOOCs.” They talk about “affordable degrees at scale.” Yes, these are big classes. And yes, Georgia Tech does use MOOC platforms as part of their delivery ecosystem. But their experiment was never about “massively open.” It was about “affordable at scale.” They wanted to see how inexpensively they could offer a degree while keeping some of the same structures and conventional quality checks that their face-to-face programs have. And for Udacity, Georgia Tech’s first MOOC partner, the degree program represented a pivot. (The first of several.) They essentially acted as a kind of Online Program Enabler, which is a weird category that crosses boundaries of platform, content, and services.

    When I look at Coursera’s latest announcement, I see an offering very roughly akin to the one that Udacity originally made for Georgia Tech, but with a heavier emphasis on prepackaged content—likely because Coursera already has a lot of content on the platform that may be quite good but is apparently not going to be disrupting universities, or conventional degrees, any time soon. If Coursera were to become (in part) a two-sided market for universities to buy and sell interactive curricular materials from each other, that’s not necessarily a horrible future for either the company or its customers. But it is suggestive of the collapse in the hype we’ve seen in both the MOOC and the OPM markets.

    OPMs, courseware providers, and “MOOCs”

    I want to return to the portion of that quote from the Science article about MOOC providers turning into OPM companies, which I highlighted but didn’t address. Again, that’s obviously true. All this talk about “micro masters” and “degree pathways” essentially amounts to a claim that many working professionals would like to pursue their post-graduate education in small, career-oriented (but still accredited and tuition-burdened) chunks. Likewise, MOOC providers like to talk about how MOOCs provide an inexpensive sales funnel to get degree students.

    While I haven’t seen direct hard data to support either claim, I’m more inclined to believe the former than the latter. A “micro-masters” program is essentially a large certificate program that can count toward a larger degree. Certificate programs have been around forever, and they sell. On the other hand, I’ve seen no public evidence that the MOOC, a course genre which has trouble getting students to the end of the first course, is going to be successful at getting students to matriculate to a program in significant numbers. The ASU Global Freshman Academy isn’t a direct comparison, since it is undergraduate, but the numbers are still pretty discouraging.

    Reich and Ruipérez-Valiente have their own opinion about the competitive advantage of MOOC providers in the OPM space, and it is revealing:

    The primary competitive advantage of MOOCs relative to established school-as-a- service providers involves cutting labor costs through automation. Many “traditional” online programs include small class sizes, synchronous sessions with instructors, and human-graded assignments. Many degrees offered by universities with the technology and support of Coursera and edX will be one- half or one-quarter as expensive as typical U.S. professional online credentials, with the bulk of savings coming from a combination of larger class sizes, fewer or no synchronous sessions, reduced contact with instructors, and more autograded assignments (12).

    This is precisely the value proposition that the digital homework solution—courseware’s older sibling—brought to the face-to-face lecture hall in survey-level courses. Digital assessment is what enabled those courses to swell to 500 or more students. The authors continue,

    Because MOOC platforms support programs that look more like “traditional” online higher education, the literature on online learning can provide guidance. By most indications, students typically do worse in online courses than in on-campus courses, and the challenges of online learning are particularly acute for the most vulnerable populations of first generation college students, students from low-income families, and underrepresented minorities (13). If low-cost, MOOC-based degrees end up recruiting the kinds of students who have historically been poorly served by online degree programs, student support programs will be vital. Some recent research has explored online and text-message–based interventions for supporting these students, but most research suggests that human connections through advisers, tutors, and peer groups provide the most important student supports (14). These human supports will push against lower tuition costs. MOOC- based degree providers may find that highly effective online learning for diverse populations costs about the same to provide as highly effective residential learning (12).

    When you start playing with this balancing act in order to arrive at…ahem…an affordable degree at scale and quality, you likely end up with something that looks very much like the Georgia Tech solution. It’s not a free degree or a $1,000 degree. It might be a $7,000 degree or a $14,000 degree. And that’s at the graduate level. It’s not clear that we know how to do this at the undergraduate level yet.

    But the more interesting implication vis-a-vis MOOC providers is that their value proposition starts looking more like that of modern courseware support with some services bundled in.

    As MOOC providers compete with conventional OPMs, there are entirely separate questions of financing the program development (via revenue share or some similar mechanism) and marketing. The MOOC providers have the advantage of their portals for marketing; students may go to Coursera or EdX to look for a credential program (as opposed to a full graduate degree program) before they’d go to their local university. I’ve not seen that proven, but at least it’s plausible. And the financing is what it is. Either you want a revenue share or you don’t.

    But as a genre of course, the population of xMOOCs is dying off. We don’t see it because we’re also calling the thing that is replacing them—which isn’t open—a “MOOC.” The collection of actual xMOOCs that are still functioning as full and (more or less) open courses is slowly shrinking to fit the size and shape of the professional non-degree credential market. Forking off from that is something that looks like a MOOC but is actually prepackaged courseware, to be licensed like a textbook and taught by individual instructors at different universities, with or without a face-to-face component. Then there’s this third thing—the affordable degree at scale—that is using MOOC and courseware affordances, which are increasingly the same affordances, to teach more students with similar learning outcomes at a lower cost. So far, only in professionally oriented graduate degree programs. And finally, there are zombie MOOCs that have no strong reason to exist and are being killed off by the platform providers for whom they are loss generators (sometimes to the dismay of the universities who invested considerable time and money in creating them).

    OK, maybe I was wrong. The word “disruption” is still relevant in at least one sense.