I’ll be giving a talk at the OpenSUNY conference tomorrow (Thursday) at 4:00 PM. The title of the talk is “Everything Old is New Again: What You Taught the OPM Companies.” But it’s about a lot more than that. SUNY was my first gig in higher education, after doing stints in K12 and corporate e-Learning, so it’s going to be a bit of a homecoming for me. I’ll be sharing some lessons learned over the intervening 14 years—for me and for them.
My last post was called “Is Ed Tech Hype in Remission?” It was about—surprise!—the interesting phenomenon of ed tech hype seemingly fading for the moment. I started the post by apparently breaking a little news. Civitas, the learning analytics company, had announced a new round of investment. A close examination of the details, coupled with some information from our sources, indicated that the company likely took a hit in valuation in that round. Since student success analytics has been a hyped product category, I used that bit of news as a jumping off point. And I used the phrase “fire sale” to characterize the downward valuation, although I was fairly clear that I didn’t even have enough information to confirm that it is a downward valuation. In and of itself, this downward valuation, if true—I’m fairly confident that it is—is mainly of interest to investors. I brought it up as an indicator the hype cycle in action rather than a signal of Civitas’ impending doom.
There were lots of interesting and nuanced reactions in the comments thread on the post, on Twitter, and on LinkedIn. Elsewhere, the reaction has been different. There was some press coverage, some of which was good, and some less so. These are pretty nuanced issues. The easy part of the story to cover is not about the hype cycle or how to think about solving education’s hard problems but about whether Civitas is doing awesome or terribly. Customers, ex-customers, and of course, competitors have plenty to say about that. Sales reps for Civitas’ competitors are already out on the streets, weaponizing that post.
Look, we have a sharp rhetorical style here on e-Literate. That isn’t going to change. We want to be clear, and we want to hold actors in this space accountable. But we also try to be nuanced. So here’s a tip for you: If you get a sales rep quoting us ripping one of their competitors, make sure you read the whole post. If it was primarily about holding that competitor accountable as a bad actor, that’s fair game. On the other hand, if it was an en passant observation made in the context of a larger discussion that wasn’t especially critical of the company—like my last post—then the use of our comment is more a reflection on the sales rep than of the company we were commenting on.
Civitas has a lot of mindshare in the student success platform market category. That market category was overvalued, not mainly because of anything Civitas said or did but because higher education and ed tech investors alike have had a tendency to look for technological magic bullets. In fact, Civitas has sometimes actively resisted that hype trend, even to the point of choosing a name that can be interpreted to mean “community.” The only choice they made that I focused on in my post was their decision to market themselves as a software platform. Which was part of my point. One reason a company inclined to name itself “Civitas” might focus on selling itself as a platform company is because the market (and funders) can only make sense of them as a technological magic bullet. It’s a systemic problem.
This wasn’t what I intended to write about, but it happens to fit perfectly with the main subject of the follow-up post I was planning to write. It was on my mind to say something about another sharp section of that post:
One could argue that we hit peak ed tech hype in 2012. The Year of the MOOC. Remember how there were only going to be 10 universities in the world, and only one lecture for every subject, given by the very best lecturer in the world? Remember how everyone was going to get a Stanford education for free?
Yeah. Good times.
Since then, the hype cycles have been shorter and less intense. Sure, there was the whole adaptive learning bubble (or “personalized learning,” as it is inaccurately called), but a lot of that was the knock-on effect of a flood of Gates Foundation money. I never got the sense that there were many True Believers in adaptive learning as a magic bullet. There are still some True Believers in learning analytics, but it’s a small group. In fact, the OER True Believers club may now be larger than the learning analytics club.
Mostly, people seem to be approaching all of these things—learning analytics, adaptive learning, OER, inclusive access, etc.—with a little more sobriety. These developments are all getting attention, but not a lot of hype (though not always for lack of trying). The general attitude among educators and institutions seems to be more like, “Huh. So that’s a thing now. Good to know. What can I do with it?”
Gone are the days—at least for now—when provosts or presidents emerged from their offices all across the country and proclaimed, almost in unison, “Hear ye, people! I hath spake with the good people from Coursera, and they have shared with me the miracle of recording lectures in four-camera studios and giving away the courses for free. Huzzah! Huzzah! Let us be fruitful and make MOOCs with great haste!”
Ouch.
It’s true that some of the stuff that happened around MOOCs was objectively dumb. Phil has a good run-down of the research from both now and then. There were schools that rushed into projects and people—most infamously, Sebastian Thrun—who displayed astonishing hubris. Many of the people at the heart of MOOC mania, whether or not they were actively or intentionally participating in the hype, were really, really smart, and some of them had the best of intentions. I never got to know Thrun or Anant Agrawal—I’ve briefly met them both—but I’ve spent some significant time with Coursera’s Daphne Koller and Andre Ng. I like them both. A lot. In fact, I worried that the two of them, but especially Andrew, were too idealistic and too focused on doing good in the world to hold onto the reins of power in a VC-owned company with the kind of growth expectations that were put on Coursera. (I was probably right.)
And the truth is that, in 2018, universities are still building, delivering, and experimenting with MOOCs. Students are still learning from the courses, and we are still learning from the form. There was and is nothing wrong with experimenting with MOOCs to see what we can learn about new ways to reach new students or better serve some of the students we reach today. Just as there is nothing wrong with experimenting with student success analytics to see what we can learn about new ways to identify students who need help sooner or find new ways to help them (or to enable them to help themselves).
But why does experimentation come with the insanity so often? Why were MOOCs accompanied by MOOC madness?
I don’t know for sure, and I suspect that the full answer is complex and related to pre-rational aspects of our thought processes as they evolved over millions of years. But here’s one simple and obvious part of the answer: 160,000. That’s roughly how many students took an early MOOC in artificial intelligence offered by Sebastian Thrun and Peter Norvig. It’s no coincidence, I think that the four founders of the pioneering MOOC organizations—Thrun, Agrawal, Koller, and Ng—are all computer scientists. For one thing, they all could do the math fairly quickly to recognize how long it would take them to reach that many students via more conventional means.
You probably should be knocked a little bit off your axis by the notion of reaching 160,000 students all over the world with one class, particularly if you are an idealistic educator. We live at the first moment in history when it is conceivable to enable every human being to have access to education equal to their intellectual potential. That is what “160,000” represents. The possibility of a new mission for higher education and the potential dawn of a new era for humanity. So yeah, some people lost their minds for a while.
It turns out that reaching all of those potential students effectively is not so simple, and doing so in a way that is organizationally sustainable is even harder. Heck, we haven’t even figured out how to fund educating all the people in our own states here in America. How are we going to fund educating everyone in the world? I’m not saying it can’t be done. I’m saying that we should have known it wouldn’t be so easy. And we should have known that video lectures wouldn’t be the answer. (Yes, yes, I know, many of us did. The point is, people lose perspective sometimes. I have over 15 years of blog posts on this site, so if anyone wants to point out times when I did, I’m sure they could find plenty of examples.)
This is not a sufficient explanation for the ed tech hype cycle. I could list other subjects of hype that were…shall we say, not as understandably inspiring of irrational exuberance. But it’s a place to start. Educators typically want to do good. That includes educational professionals who happen to work for for-profit companies, by the way. In order to do so, they often have to deal with organizational psychology, business process management, budgets, politics, market forces, and a whole host of confusing and interacting systems that human minds are not very good at modeling. So we tend to latch onto simpler, and often shinier, explanations. Technology will save us. Evil companies are killing education. Education need to be disrupted.
But the thing about chasing the hype is that it is exhausting and expensive. After a while, it wears you down. That’s what I think we’re in now. A period of exhaustion. We have enough people who have been burned enough times in rapid succession, and who are trying to solve enough serious and immediate problems, that they just can’t afford to be burned chasing the next shiny thing right now. They have to focus on solving the hard problems, because those are the real problems that just might move the needle for their respective institutions. That’s good news for almost everyone, from the students, to the faculty, to the universities, to the ed tech companies that want to do the right thing.
Last Wednesday, there was an announcement about an investment in Civitas Learning that appeared to escape the notice of the educational press. Even EdSurge and Education Dive, which are both pretty good about covering the investment side of ed tech, seem to have missed it.
And at first blush, it’s easy to see why. The press release reads like a generic funding round that was successfully closed:
Civitas Learning today announced a significant growth investment from Francisco Partners, alongside current education and impact investors including Rethink Education, SJF Ventures, and Lumina Foundation. The Austin-based company pioneered the use of advanced data science, design thinking, and machine learning to inform initiatives and improve student outcomes.
That’s the meat. The rest of the press release is the usual anodyne quotes from the CEO and investor, fun facts about the company, and so on. Blah blah blah.
But there’s more here than meets the eye. First, Francisco Partners is not a venture capital fund. It’s a private equity firm. If Civitas were experiencing enough success to raise a growth round of funding, then a PE firm with no obvious experience in higher education would not be a conventional candidate for a funder. And we at e-Literate have seen no evidence that Civitas is, in fact, growing in a way that would justify a next round of investment. We haven’t heard about any new Civitas customers in quite a while (though we’ve heard about some customer losses and some grumbling among their current customer base). A quick perusal of their press pages doesn’t show any announcements of big customer wins either. It does reveal an announcement of a partnership with recruiting and retention company Ruffalo Noel Levitz, which is interesting and which I’ll return to later in this post.
Put all this together, and the deal looks more like a fire sale than a growth round. We have some information from sources familiar with the deal which supports this inference, although we do not have enough independent sources to confirm it yet.
I have no major beef with Civitas on the fundamentals of what they’re trying to do. But the buzz they generated, the valuations they got from investors, and the cool kids’ club that seemed to hover around them for a while are all symptomatic of a company that, whatever the soundness of its fundamental aspirations may be, was hyped by the markets and the press. The bubble may be deflating now.
A smart friend commented to me the other day that there are fewer of these lo-how-the-mighty-have-fallen stories than there used to be. I think this is a profound observation. Something has been shifting in the ed tech markets over the past five years or so. The ed tech hype cycle seems to have at least partially and temporarily burnt itself out. While I have no illusions that hype cycle is dead, or that it will stay relatively dormant, the quality and intensity of it is definitely different than it was five or six years ago. And that change may be a visible symptom of some more fundamental changes that are happening in the educational landscape.
Remember the days?
Remember the days when EDUCAUSE was partly a kind of fashion show? I could always go there and come away with a good blog post about that year’s ed tech darling. One year it might be open source LMSs. (Did you blink? Ah, too bad. It was a sight to see.) The next year it might be Pearson’s OpenClass. (Remember that?) Or Knewton. (Yes, they do still exist. I checked.) But there was a moment about six or seven years ago—I remember talking to Josh Kim about it at the time—when the theme of EDUCAUSE became that there was no theme. It has stayed that way ever since.
That wasn’t the end of hype. It shifted to other places. The fashion show moved to SxSWedu for a few years. And online. And in the mainstream press.
One could argue that we hit peak ed tech hype in 2012. The Year of the MOOC. Remember how there were only going to be 10 universities in the world, and only one lecture for every subject, given by the very best lecturer in the world? Remember how everyone was going to get a Stanford education for free?
Yeah. Good times.
Since then, the hype cycles have been shorter and less intense. Sure, there was the whole adaptive learning bubble (or “personalized learning,” as it is inaccurately called), but a lot of that was the knock-on effect of a flood of Gates Foundation money. I never got the sense that there were many True Believers in adaptive learning as a magic bullet. There are still some True Believers in learning analytics, but it’s a small group. In fact, the OER True Believers club may now be larger than the learning analytics club.
Mostly, people seem to be approaching all of these things—learning analytics, adaptive learning, OER, inclusive access, etc.—with a little more sobriety. These developments are all getting attention, but not a lot of hype (though not always for lack of trying). The general attitude among educators and institutions seems to be more like, “Huh. So that’s a thing now. Good to know. What can I do with it?”
Gone are the days—at least for now—when provosts or presidents emerged from their offices all across the country and proclaimed, almost in unison, “Hear ye, people! I hath spake with the good people from Coursera, and they have shared with me the miracle of recording lectures in four-camera studios and giving away the courses for free. Huzzah! Huzzah! Let us be fruitful and make MOOCs with great haste!”
Don’t get me wrong; there are still presidents and provosts emerging from their offices and making pronouncements. But I’m seeing a lot less of it than I used to. The fever seems to have passed and left some antibodies in its wake.
The one area in the market where I see something approaching hype, which I would characterize more as “intense interest coupled with a lot of hand-wringing,” is in the Online Program Management (OPM) space.
It’s worth asking why this is so.
Operational excellence is the new hotness
I have an optimistic and a pessimistic take on why ed tech hype is cooling and focusing at the moment. The optimistic take is that the sector is finally learning that there are no magic bullets. There used to be a lot of frantic effort generated by of fear of being left behind. That fear is now balanced by a healthy fear of wasting time, energy, and reputation that could be better invested. The pessimistic take is that, as a wider swath of institutions faces the existential threat of financial insolvency, they don’t have cycles to waste on trying to do cool things. They need to be focused on effective things. When an elite liberal arts school like Hampshire College is teetering on the edge of extinction, you know that #*@!& is getting real. There is likely some truth to both of these takes, which tend to feed each other.
What do OPMs do for universities? The simplistic first-approximation answer is that they enable the schools to generate more live-and-thrive revenue by generating more enrollments. How do they do this? Again, the first-approximation answer is that they provide operational excellence at building, launching, and filling new online degree and certificate programs. The good ones can do it quickly, efficiently, and with decent quality levels relative to the current baseline of quality in online learning. And if you believe that the OPM solution category is partly defined by revenue sharing (which I do), then true OPM offerings include a financing service, which enables the schools to get more programs up more quickly (albeit potentially at a higher long-term cost).
This value proposition is a far cry from robot tutors in the sky that can semi-read your mind. It’s less sexy, more grounded, and more strategic.
This shift toward practical operational services is mirrored by shifts in the capital markets such as the recent Civitas development. I have always felt that the potentially fatal flaw in Civitas was that it should have been a service company but chose to position itself as a platform company in order to compete for capital. VCs love platforms because they can grow very big without adding the cost of a lot more humans to run them. You know, like Google and Netflix. If you want to get a lot of money from VCs, it helps a lot to look like a platform company (although I get the sense that’s beginning to change in the education investment space).
If you want a picture of what Civitas could have been (and maybe should have been) in an alternate universe, then look at EAB. It’s positioned as a service company with some tech. Its big-picture value proposition to customers is basically operational excellence at recruiting and retaining students. Because the EAB management never had to raise venture capital, they were free to be what they needed to be in order to support their customers. They didn’t have to pretend to have a magic platform. They could be some very smart people who have some useful skills at improving end-to-end student success and who have some software that helps with that (which might also be a more honest description of Civitas).
I doubt that Civitas’ valuation is that high, particularly after this latest cash infusion. Their partnership with Ruffalo Noel Levitz is particularly interesting in this context. What do you get when you combine Civitas with a recruiting and retention services company? A more service-oriented offering that looks a little more like EAB—and a little more like an OPM or an Online Program Enabler (OPE).
Maybe now, with a more realistic valuation, a more service-oriented value proposition, and hopefully some patient capital, Civitas can realize its potential. I’m not making any predictions, but this feels like a course correction which both reflects the current realities in higher education and has the potential to bring the company more in line with those realities.
Operational excellence at supporting student success is the Next Big Thing
So colleges and universities are getting more focused on developing and supporting solid online programs that serve untapped student needs well enough to generate reliable additional revenue for the institutions. There are whole product categories of companies that are succeeding by providing various kinds of operational support for this growing focus. The variations among the commercial offerings are diverse and fast-growing enough that it is creating some confusion in the market, but a lot of that naturally comes with rapid growth and the bare-knuckled competition it engenders among vendors.
The nascent area that shows major yet poorly defined potential for growth is in helping universities improve the baseline for (online and on-premise) student success, whether that means college completion, career advancement, or something else. I see a lot of innovation at individual institutions and from individual vendors that gets at pieces of this problem. But the sector still lacks end-to-end methodologies for restructuring our colleges and universities to optimize themselves for this purpose and continually improve at it. Learning analytics are cool. But you know what’s really cool? Students who graduate, on time, for less money, and whose education enables them to live better lives.
That’s the next frontier in higher education. It’s a hard one, and there won’t be any magic bullets that “fix” or “disrupt” all of education. But there will be large gains in significant pockets. Universities were not designed to serve this primary purpose with excellence and efficiency in a 21st-Century world. They do surprisingly well given that fact, but we will discover some big opportunities for gains similar to the ones we see when we put an electric drive train into a conventional automobile.
“Guess what? All that up-and-down with the cylinders, all firing at exactly the right millisecond, and then the gears and contraptions to turn the up-and-down into round-and-round? Gone. Oil changes? Gone. You have a battery, you have electric motors directly attached to wheels, you have absurd amounts of torque, and very few moving parts to wear out. The power goes straight to the wheels. All that internal combustion stuff did a great job for the last 100 years, but we can propel our wheeled vehicles with more efficiency and less complexity now.”
We will discover opportunities to rethink and return to first principles in higher education. They won’t be tech-only; the machines we are talking about are the universities themselves, and the changes will be ones of process at least as much as of tech. These changes won’t work everywhere equally well to solve all educational problems. But the fact that colleges and universities have not been consciously and continuously optimized for their new role (and sustainability needs) means that we will find many gaps where simple changes will make outsized differences.
This is already happening in many individual places. You will likely find at least one such story on any given week reading Inside Higher Ed or EdSurge. What we haven’t seen a lot of yet is a knitting together of the individual innovations into a methodology for operational excellence at supporting end-to-end student success. EAB is one harbinger of things to come in that regard.
Mark my words: The institutions that figure out how to do make this transformation, and the companies that figure out how to support it, will tend to thrive in the long term. In my entire career, the only thing I’ve ever found that has come close to living up to its hype is a good education.
Carnegie Mellon University and Duke University have shared newly available free tools that will significantly lower the barriers to conducting ethical educational research. The two universities contributed the tools through e-Literate’s Empirical Educator Project (EEP), an effort to promote broader adoption of evidence-based teaching practices and foster a culture of empirical education across higher education.
As with all academic research involving human subjects, educational researchers must have their experimental designs approved by their university’s Institutional Review Board (IRB). If a researcher wants to study students or their work, they must explain how they will get the students’ informed consent to participate.
This can be a major barrier that often prevents research from being undertaken. Teaching faculty who may be interested in conducting a study may decide that the bureaucratic burden is more than they can take on. Multiple universities that want to collaborate on cross-institutional studies will have to get approval from each institution’s IRB in an environment where there are no widely adopted standards for reviewing and approving educational research by these bodies. Educational technology companies that want to be more transparent and collaborative with universities about their own research into product efficacy can find the IRB process impractically time-consuming. As a result, far less educational research gets conducted in ways that are both reviewed for ethical practices and shared as credible research that contributes to the state of the art in learning science.
Through e-Literate’s EEP, learning science researchers at Carnegie Mellon and Duke Universities discovered that each institution had developed a solution for part of this problem. Carnegie Mellon University has developed templates approved by their IRB that they estimate will accommodate approximately 80% of classroom research use cases. Meanwhile, Duke University has developed language and a process approved by their IRB for requesting and tracking informed consent from students.
The two universities have released the tools under a Creative Commons Attribution (CC-BY) license and provided “train the trainer” support for the use of their templates and protocols. Together, these contributions could enable many of the educators and product designers who are already conducting informal educational research all over the world to participate in the same sort of social fabric that has enabled communities of researchers in other human sciences to tackle problems from cancer to Alzheimer’s disease.
Jeff Young has a great piece up about this release at EdSurge, and I believe we will see something from The Chronicle in their teaching newsletter on Thursday. I’d like to give you my own take on the reasons why this an important milestone.
It’s an example of untapped inter-institutional opportunity
Carnegie Mellon and Duke are theoretically peers. To use one very imperfect measure, they are both ranked in the top 25 national universities by US News and World Report. They are both more specifically ranked in the top 15 such schools for undergraduate teaching. Both are seriously concerned with improving undergraduate education through research-based practices. The two institutions have been working on similar yet complementary efforts to make that research easier. You would think that they would have had opportunities to share their work with each other, or at least know about what the other is doing.
Before EEP, they didn’t.
On closer examination, the complementarity of the two efforts suggests the kinds of opportunities that higher education is missing. Carnegie Mellon University has one of the broadest, deepest, most impressive, and most historic learning science research programs on the planet. There are only a handful of universities that are even in their league, and Stanford may be the only university that rivals them in depth and breadth. Further, the university has made a commitment in the form of the Simon Initiative to “[p]rovide accessible tools and methods with which any person or institution can adopt and advance CMU’s approach to learning engineering, improving outcomes for their own learners,” globally, in addition to improving teaching and learning at Carnegie Mellon University itself. (Not that this is distinct from but complementary to their Eberly Center for Teaching Excellence, which is similar in purpose to the centers of teaching and learning at many universities.) To get a flavor for who they are and how they think, here’s a playlist of three e-Literate TV videos that highlight a few of their faculty members:
Now, Carnegie Mellon is known more generally for its engineering prowess, so it’s no surprise that the folks at the Simon Initiative talk about “learning engineering.” (The initiative is named after the late Herb Simon, a cognitive scientist and CMU luminary who coined that term.) That mentality, as well as the orientation of an institution known for its learning science research, shows in their contribution. They studied the IRB applications submitted for educational research by their faculty—not just their learning science faculty, but all faculty interested in publishing their research projects, identified common traits, and developed a template that they estimate covers somewhere in the neighborhood 80% of those projects. They then had their IRB review and approve the template. Now, any educator at CMU who wants to do publishable educational research and can use the template gets their IRB application fast-tracked.
Without taking anything away from Duke’s own learning science research capabilities, one of the things that Duke is among the very best in the world at is making their undergraduates’ educational experience life-changing. As one example, the university has a large number of “professors of practice.” In many big research universities, a tenure track is a kind of death trap. Faculty are worked to the bone for three to five years in the hopes of achieving tenure when, in reality, most of them will be sent packing in the end. And while they are on that treadmill, they have a disincentive to invest too heavily in their teaching lest they neglect the publications and grants that will increase their odds of not being shown the door. That’s not how Duke rolls. Rather than exploiting young faculty by dangling a carrot that will forever be out of reach, they offer many a professor of practice position. Such faculty get basically everything except tenure, including long-term employment, a decent salary scale, and a say in shared governance. But they must not only show that they are excellent educators but also conduct research in education. In effect, rather than “physics” or “art history” being their discipline, it’s “physics education” or “art history education.” I don’t know whether Duke or their faculty would put it quite this way, but I intend it to be a compliment. The point is that they incentivize faculty to become disciplinary experts in supporting their students through evidence-based practices. I have been to one of their teaching and learning conferences and interacted with these faculty members. It was heavily attended and the atmosphere was electric. I have been to many of these kinds of events; yet I have rarely seen faculty who were more actively engaged in asking good, probing questions about teaching practice than I did at Duke.
This different context from CMU’s has resulted in a slightly different approach to the same problem. Like everyone else, Duke’s educators who want to publish their research have to go through IRB approval. And given that Duke has a world-class medical research program, their IRB is both very tough and very focused on privacy concerns. So their Learning Innovation center developed an informed consent tool called WALTer, where WALT stands for “we are learning too.” WALTer sits right inside the LMS for any course in which research is being conducted. Faculty who want to conduct such research are led through a decision tree that produces IRB-approved language for informed consent, based on the conditions of the experiment. Having this form in place helps the instructor get faster approval of the project. And once that approval is in place, WALTer helps ensure that students are given the opportunity to provide their informed consent (or not). Since WALTer has been launched, the Learning Innovation team has seen an increase in the number of faculty expressing interest in conducting educational research and a decrease in the time it takes educational research applications to be approved by Duke’s IRB.
Peanut butter, meet chocolate. CMU’s IRB template and Duke’s informed consent template are completely complementary. But before EEP, the peanut butter was in the fridge and the chocolate was in the drawer. They just didn’t meet.
It’s an example of untapped intra-institutional opportunity
The Duke feedback that they are getting more educational research interest and getting that research approved faster is illustrative of a larger point. The current institutional structures and processes of colleges and universities are not designed to facilitate the development, testing, sharing, and adoption of evidence-backed practices which improve student success. The IRB process is just one (very painful) example. The disincentive for faculty on the tenure treadmill is another. There are many more.
The question that I have heard asked ad nauseam for years and years now, from a wide range of people, is “How can we make faculty care more about teaching?” That is the wrong question on every level. A better question would be, “How can we design universities such that focusing on improving excellence at supporting student success is less painful and more rewarding?”
Think about just about every hot ed tech-related trend you can think of. Retention early warning analytics. Adaptive learning. Competency-based education. Stackable credentials. MOOCs. All this activity (and money) is swirling around the problem of making it easier for students to learn and succeed at school. Now think about all the hot trends that focus on making it practical and rewarding for faculty to focus their energy and considerable intellectual talents on solving the same problem. Can you name one?
An IRB form for educational research may sound like a small and boring thing, but it is a piece of cultural and institutional infrastructure that makes it a little more practical for motivated faculty to focus energy and intellectual talent on learning how to better support student success. “We Are Learning Too” indeed. This is the kind of work that will make college education better and, in the process, make all of those ed tech tools more useful. A laser scalpel doesn’t do much to promote health unless it’s wielded by a physician who knows how, when, and why to use it. ((Is a “laser scalpel” a thing? I may have just made that up. Anyway, you get the point.)) Without that knowledge, it risks doing more harm than good.
It’s an example of untapped multi-institutional and institution/vendor collaboration opportunity
So yay for Carnegie Mellon and Duke. What about everyone else?
Well, for starters, they have both contributed their language under a Creative Commons license. (Duke’s WALTer tool is built using a third-party proprietary platform, so contributing the source code wasn’t an option. But somebody else could easily build a tool that supports the workflow in the release document.) So there’s that.
IRBs are notoriously idiosyncratic. (Some might say arbitrary. I’m not saying that. But some might.) So you could see an IRB at an institution that’s very different from CMU or Duke having something like the following argument:
IRB Member #1: Hey, CMU and Duke are super-rigorous research institutions that are way more focused on these sorts of things than we are. If it’s good enough for them, it should be good enough for us.
IRB Member #2: Actually, exactly because CMU and Duke are super-rigorous research institutions that are way more focused on these sorts of things than we are, what makes you think that what makes sense for them will also make sense for us?
These are both reasonable starting positions. The conversation that should flow from this is an examination of the templates in which an IRB member who wants to change a part of it will need to provide a justification for doing so. This is exactly the next step we want to foster for this project, preferably at multiple institutions. Where we’d like to end up is at a toolkit in which an institution of any type can look at variations—and justifications for those variations—provided by peer institutions so that they adapt and adopt them. We’d also like to supplement what we already have with some more fleshed out student data privacy guidelines that are, once again, education-appropriate. Data privacy in this sort of research is just as important as it is in, say, medical research. But the specific concerns and methods for dealing with them aren’t necessarily identical. We should be developing a sector-wide consensus on ethical practices that can augment what is already in the Duke and CMU IRB contributions.
This would hopefully help to lower one barrier to conducting educational research everywhere. But it could potentially do much more than that. Under the law, if researchers want to conduct a multi-institutional research project, then each institution’s IRB must approve the project. And since no two IRBs use the same standards and most don’t have particular guidelines for educational research, doing research across two institutions is more than twice as hard. Three is more than three times as hard. Doing large-scale, multi-institution research quickly becomes impossible in most scenarios. Unless you’re a vendor, in which case it is trivial, because you are not required to go through IRB for your own research—as long as you don’t publish it in a journal. So vendors can conduct research on students in multiple institutions for their own proprietary purposes easily, but if they want to do the right thing by going through IRB approval and sharing what they’re learning through peer-reviewed journals, it’s way, way harder.
Imagine if it were easier for everyone to do the right thing and submit an application for IRB approval, knowing that they will be going through a streamlined but academically validated process. Imagine the kind of research opportunities that could open up. Now imagine further if there were some technology infrastructure behind this. Imagine if we could track IRB approval and informed consent across institutions, gather appropriately anonymized student data, and share it among researchers in a repository that is designed to respect the student privacy constraints dictated by the approved IRB applications while giving more researchers access to more research-relevant data—including data that are gathered through student interactions inside vendor tools. Technologically, this is quite practical. The hard part is getting the policy infrastructure solid and widely adopted.
This is what EEP is about. The problem isn’t that higher education is failing to innovate or that professors don’t care about teaching well. The problem is that we are flushing 99% of the existing efforts, potential opportunities, and good intentions down the toilet because we don’t have the cultural institutions and social infrastructure to support and sustain them. But that can change.
Kudos and thanks to the good folks at CMU and Duke. We have more folks working in EEP—from a diverse range of institutions—on a wide range of projects. And we’re still learning how to work together effectively. Expect more to come.
Our favorite technology industry blog—Ben Thompson’s Stratechery—has a great piece up about Amazon’s relationship to open source that also explains a lot about the tectonic shifts in the LMS market. The story he’s interested in telling is about the dynamics behind Amazon’s move to essentially copy and abandon a popular open source database called Mongo DB. His introductory analogy to the music business is revealing and worth quoting at length:
In 1999, music industry revenue in the United States peaked at $14.6 billion (all numbers are from the RIAA). It is important to be precise, though, about what was being sold:
$12.8 billion was from the sale of CDs
$1.1 billion was from the sale of cassettes
$378 million was from the sale of music videos on physical media
$222.4 million was from the sale of CD singles
In short, the music industry was primarily selling plastic discs in jewel cases; the music encoded on those discs was a means of differentiating those pieces of plastic from other ones, but music itself was not being sold.
This may sounds like a stupid distinction, but it explains what happened after that peak:
Music industry revenue plummeted, even as the distribution and availability of music skyrocketed: the issue is that people were no longer buying plastic discs, which is what the music industry was selling; they were simply downloading music directly.
Selling Convenience
The problem is that recorded music has always been worthless: once a recording is made, it can be copied endlessly, which means the supply is effectively infinite; it follows that to capture value from a recording depends on the imposition of scarcity. That is exactly what plastic discs were: a finite supply of a physical good differentiated by their being the most convenient way to get music. Pirating MP3s from sites like Napster or its descendants, though, was even more convenient — and cheaper.
As you can see from the chart, the industry started to stabilize in 2010, and in 2016 returned to growth; 2018 looks to be up around 10% from 2017’s $8.7 billion number, and it seems likely the industry will pass that 1999 peak in the not-too-distant future.
What happened is that the music industry — prodded in large part by Spotify, and then Apple — found something new to sell. No, they are still not selling music; in fact, they are beating piracy at its own game: the music industry is selling convenience. Get nearly any piece of recorded music ever made, for a mere $10/month.
I don’t agree with some aspects of his analysis of open source in the rest of the article, but Thompson’s introductory framing is brilliant. Sometimes we get hung up on the thing that we think is the product in way that blinds us to the critical aspects that are valuable to the customer. These blind spots can cause us to miss potential points of instability in a seemingly stable market landscape. And cloud computing is a classic example of a thoroughly unsexy idea that can sneak into one of those blind spots. It certainly did in the LMS market.
The Instructure surprise
Instructure’s rise is a perfect example of one of those surprises. Let’s think back to the late Noughties, when the company was founded. This predates Phil’s coming to blog on e-Literate, so we’ll have to look at a 2009 version of his famous squid chart that he posted on the blog of his former employer:
2009 Squid Chart
Instructure, having been founded the year before this version of the chart was made, had not yet scored its first big deal with the Utah Education Network. They were literally not on the map. Blackboard was a juggernaut, having successfully swallowed two of their most formidable North American competitors, WebCT and ANGEL. Blackboard was also in the midst of a patent infringement case with Desire2Learn (now known as D2L), having won their case in 2008, only to lose upon appeal in mid-2009. Even after the suit was over, nobody knew how well Desire2Learn would bounce back after seeing their sales largely freeze during the year when it looked like Blackboard would win. eCollege was doing…fine, but it was serving the niches of for-profits and small schools, and there was no real sign that it was going to break out into the general market.
The real action in 2009 appeared to be in open source. Moodle, having picked up the smaller customers that Blackboard had deliberately driven away because of their low profitability, was beginning to score bigger wins. The change was most visible in the Cal State system, where Moodle was spreading there like a virus. Sakai’s growth by institutional adoption numbers was not nearly as dramatic, but in contrast to Moodle, they were rich in prestigious R1 university adopters. In 2005, when I was working at SUNY, I remember my boss at the time telling me, “They have MIT, Stanford, Michigan, Indiana…they can’t fail!” There was a widespread feeling among academics that the only way to escape being a Blackboard hostage…er…customer was to run an open source LMS that the company couldn’t buy. (Blackboard didn’t acquire the largest Moodle hosting provider in the United States until 2012.) It felt like the battle was going to be between Blackboard and open source.
Two years later, Phil’s squid diagram in his inaugural e-Literate post wasn’t much different:
You can see at the top of the diagram that there was increasing speculation about whether other companies—mostly big, established ones—might enter the market. There were, as always, a few startups that popped up in that time period, only to fade away, either by dying or by pivoting. (Remember Epsilen?) But by and large, the fight continued to be perceived as Blackboard vs. Open Source, with D2L and eCollege—by then rebranded by Pearson as LearningStudio—doing fine but not setting the world on fire. Instructure is still not yet on Phil’s map. We had noticed Instructure and written a few posts about them, but honestly, neither of us thought that a new proprietary entrant could break its way into the market, particularly if it lacked the muscle of a big player like a major SIS vendor or textbook publisher.
But by 2013, the picture had changed:
2013 squid diagram
Canvas was growing. Fast. Here’s the squid diagram a year later:
2014 squid diagram
Look at that Canvas line.
Whoah.
It was really only in 2013 and 2014 that most of us started to realize that Canvas was not only going to survive but might provide significant competition to the incumbents. Why did it take us so long to see it coming?
I would argue that we undervalued three of Instructure’s core strengths: usability, customer service, and reliability. We’ve written here before about how Instructure’s usability was a step function better than its competitors at the time. The early but seminal example was Speed Grader, Canvas’ grading app that greatly increased ease of use of the grading function and was the first mobile LMS app that demonstrated the potential of tablet computing. Blogs and wikis, which LMS providers had begun to add, only to find them barely used, were not considered valuable by most faculty. Giving them back hours of their time that they would have spent entering grades, on the other hand…. Likewise, we’ve written about how Instructure quickly established itself as the “uncola” of LMS companies by providing excellent customer service that was inextricably tied with their “not-Blackboard, not-Oracle” brand identity. Both of these non-features turned out to be more valuable to many customers than the bazillion features that were beginning to encrust the older, more “mature” LMSs.
But at least usability, customer service, and branding are all visible to end users in some tangible sense. In contrast, cloud computing was most valuable for something that it made invisible. Specifically, downtime. By 2012, the LMS had become a mission-critical application. Online learning was in full swing. For-profit universities like the University of Phoenix as well as public (mostly Sloan Consortium-funded) public universities like University of Maryland, University College (UMUC) had reached impressive scale in around 2008, right when Instructure was being born. In the intervening four-year period, many colleges and universities were chasing that scale. In 2010, Western Governors University founded its first offshoot campus in Indiana. The Online Program Management (OPM) business was hitting its stride. Academic Partnerships and 2U, two of the most successful OPM companies, were both founded the same year as Instructure. By 2012, both were surging. (Forbes named 2U one of the “10 startups changing the world” that year.)
And online usage in general was surging. Gmail exited beta in 2009. The two remarkable aspects about Gmail were that it provided full, intuitive functionality in any browser and it never went down. Not for crashes, and not for upgrades. It was just always there. Occasionally you would log in and there would be a new feature. But most of those upgrades were invisible to the user. A friend of mine used to love to ask a question during this period to make this exact point: “What version of Google are you using?” Not having to know your version number, having the software be always there and always up-to-date, turned out to be a killer capability. The most important feature turned out to be the one that you never noticed, because it turned your app into something that end users could come to take for granted. As more people were using rich web-based applications—remember “Web 2.0”?—they started having higher expectations for online usage in general. If buying stuff online became a normal, everyday thing, then why wouldn’t checking your course grades online? Even in a traditional, face-to-face classroom, students were starting to expect the convenience of the web to just be there for them in their classes. Documents should be there. Announcements should be there. Schedules should be there. Grades should be there. All the time. Increasingly, when the LMS went down, it was like the ATM machines going down. Before ATMs existed, life went on. Nobody died without them. Nobody noticed the lack of convenience. But afterward, since people have grown to take them for granted, any outage is an outrage.
I honestly didn’t understand why Instructure was making such a big deal about the cloud when they first launched. Neither did their competitors. And it took them quite a long time to figure it out.
Blackboard’s Hopes
Let’s fast-forward now to the relatively recent 2017 version of the squid diagram: ((It’s worth remembering that these diagrams are of market share for the US and Canada only.))
2017 squid diagram
It’s a different world. Sakai and Moodle, having peaked at slightly different points, are in decline in US and Canadian higher education. Canvas’ growth has been off the charts, and has only really slowed down in the year since this chart was made, as LMS adoptions in general have slowed in this market. D2L’s Brightspace bounced back after the patent suit and is holding their own.
But the biggest change is that Blackboard, far from being dominant, is a shadow of its former self in terms of its share of this market. These days, their press releases about customer “wins” in the United States are about Blackboard customers who decide not to leave after conducting an evaluation. We’re not seeing new customer wins in this market.
One of the reasons that we’ve been arguing that SaaS adoption is more important to Blackboard than adoption of its new(er) Ultra user experience right now is for the same reason that SaaS was so important to Instructure. The most important feature for Blackboard right now is the one that the end user doesn’t see. LMS migrations may be easier than they used to be, but they still require significant time and, often, pain on the part of the users who experience the transition. Most of Blackboard’s most risk-tolerant customers have already migrated to a newer, shinier alternative. The company’s remaining North American customer base is heavily risk-averse. It’s exactly that risk aversion that makes SaaS attractive. Blackboard is saying to these customers, essentially,
Hey, migrating is hard. Why don’t you just switch to SaaS? It can be invisible to your end users if you want it to be, but you won’t be in the firing line anymore with the painful downtime that you always get blamed for. And once you’re on SaaS, you can try out Ultra at your own pace. If the change is too scary, then don’t worry. You don’t have to make it. If you want to try it, slow or fast, with a few classes or with all of them, you’re in control. But let’s get rid of that pesky downtime for you. And let’s make upgrades less painful too. Just sign here to extend your contract for a few years and we’ll make those nasty surprises go away for your stakeholders.
It’s the SaaS, rather than Ultra, that is the primary driver for contract extensions. Which is likely why the company’s latest “Hey, we’re doing great!” press release was entitled “SaaS Deployment of Blackboard Learn Continues to Gain Momentum Around the World” rather than “Ultra Deployment of Blackboard Learn Continues to Gain Momentum Around the World” even though both headlines could equally fit the text of the press release. If Ultra adoption creeps along for another couple of years, it probably wouldn’t hurt Blackboard too badly. But if SaaS adoption creeps along, that would be a lot more serious.
Moodle’s worries
The SaaS shoe is on the other foot for Blackboard when it comes to Moodle. While the company has been playing catch-up to Instructure with their 3-year-old SaaS Learn offering, they are doing to Moodle something like what Instructure did to them with their Blackboard Open LMS offering. When Moodlerooms, which was the largest Moodle hosting provider in the US, was acquired by Blackboard in 2012, they had already built out a highly scalable SaaS version of Moodle. (In fact, Blackboard Learn’s SaaS architecture is based on lessons the company learned from studying the Moodlerooms SaaS architecture.) In global higher education markets outside of North America, Moodle is still a formidable player. In fact, it is the dominant player in many places. But the core open source Moodle project has been slow to roll out true multi-tenant SaaS capabilities. This has created an opportunity for Blackboard to roll into markets that are heavily saturated with self-hosted Moodle and say, essentially,
Hey, moving is hard. Why don’t you just switch to our Moodle-based SaaS LMS? It can be invisible to your end users if you want it to be, but you won’t be in the firing line anymore with the painful downtime that you always get blamed for. And once you’re on SaaS, you can try out our enhancements at your own pace. If the change is too scary, then don’t worry. You don’t have to make it. But if you want to try it, slow or fast, with a few classes or with all of them, you’re in control. But let’s get rid of that pesky downtime for you. And the upgrade cycles. Just sign here to have us get you on an Open LMS contract and we’ll make those nasty surprises go away for your stakeholders.
Those self-hosted Moodle customers that aren’t moving to Blackboard’s Open LMS are generally moving to…wait for it…the SaaS versions of Blackboard Learn, Instructure Canvas, or D2L Brightspace. Self-hosting as an option is fading away in the LMS market, for the same reason that fewer and fewer organizations are hosting their own email servers. The market has apparently decided that self-hosting these applications brings them a lot of pain without a lot of gain. And they may be willing to trade off functionality and autonomy in return for the perceived reliability that comes with SaaS.
This brings us full circle to Ben Thompson’s blog post about Amazon and Mongo DB. Amazon basically built its own database that runs natively on the company’s cloud platform and implements an older version of Mongo’s APIs. The threat to Mongo is that customers will find Amazon’s hey-it-just-works offering to be more attractive than Mongo’s more up-to-date APIs or than any benefits, either ideological or practical, that come from adopting open source. We don’t know how this will play out with Amazon and Mongo yet, but we’ve certainly seen how it has played out (and continues to play out) in the LMS market. It’s easy to get distracted by shiny feature-driven trends like competency-based learning, adaptive learning, or learning relationship management. These may or may not turn out to be important. But if you undervalue the boring and often invisible product attributes of “easy” and “reliable,” you can easily miss a sharp turn in the road.
Before I move on to my next case study in academic institutions moving toward operational excellence at supporting student success, I want to revisit a section toward the end of my last post on the California Community Colleges Online Education Initiative (OEI). I was looking at the alignment that has to be achieved at various levels in the academic organization in order to encourage all the stakeholders to embrace this collective mission, with all the changes to their day-to-day work and even professional identities it would entail. Much of the piece is about the work that had been done so far to get alignment at various levels within the administration. But toward the end of the piece, I speculated a bit on potential opportunities for fostering faculty alignment through a course peer review process using a common rubric:
[T]o me, one of the most interesting vectors for culture-building is the course exchange course quality rubric. Every course on the exchange has to be evaluated against a rubric of evidence-backed effective online teaching practices. As the pace at which exchange courses are developed increases, OEI will not be able to keep up with demand to evaluate these courses using central staff. So they are creating a peer reviewer mechanism in which faculty on the campuses are trained on the rubric and presumably compensated to review courses that are candidates for the exchange.
This opportunity fascinates me. We know that faculty who go through an expert-supported course redesign process often experience intellectually deep and emotionally moving shifts in their teaching strategies. Is the same true when faculty are trained reviewers of their colleagues’ redesigned courses? What effect will simply exposing faculty to more and different course designs have? How will their role as reviewers and critiquers shape or enhance that effect? Can a continuously improved and updated rubric become a vector for sharing new research-supported processes across the system on an ongoing basis? Will the impact be broad and deep enough to foster new kinds of intra- and inter-campus faculty dialogs about the scholarship of teaching and learning (SoTL)? Will these cultural changes help to foster alignment around continuous operational improvement for enabling student success? This is the last mile problem of higher education. Operational excellence at student success cannot be achieved unless it is infused in the daily operations in individual classrooms. That requires affirmative faculty buy-in, support, training, and embedding in a culture that invites them into the larger conversation.
Unpacking this a bit, what does it really mean to build a culture of operational excellence in supporting student success? What kind of change would be necessary at the individual level to achieve change at the organizational level?
Organizational learning
There is a useful concept in organizational psychology called “double-loop learning.” I’ll give a simple example of a non-academic organization first to make the concept clear. Suppose your company manufactures smartphones. You want supply to match demand almost exactly as possible. If you manufacture too many phones, then you will sink expense into building units that will sit on the shelves and fairly quickly become obsolete. But if you manufacture too few, then you won’t have phones to sell at the moments that people need to buy them, thus encouraging them to buy a different (more available) phone instead. In a single-loop model, you have one lever to pull, which is how many phones you produce at a given time. It’s a like a thermostat: If the room is too cold, then turn on the furnace. If the room is warm enough, then turn off the furnace. If there are not enough phones on the shelves, then turn up production. If there are too many phones on the shelves, then turn down the production line.
The problem is that there’s a significant lag between when the order is given to produce more phones and when they arrive on the shelves. During that time period, demand can change. Maybe by the time the new phones the company produces during a period of high demand actually land on the shelves during the beginning of a recession, or right after a competitor releases their hot new model. The single-loop, thermostat-like model doesn’t work very well.
Of course, the people who run the company are smart enough to know this, so they come up with all sorts of work-arounds. They build warehouses to hold excess phones near where they are built, since holding onto the phones that way is cheaper than shipping them halfway across the world and negotiating with the retail stores that are selling them and may want to ship excess inventory back. They build sophisticated forecasting models that account for factors such as the economy and competitor behavior, so the chances of them being badly wrong are reduced. These are all work-arounds to a fundamental problem regarding the costliness of being wrong in your demand forecasts. And this is exactly the way manufacturers of all kinds of complex items, including smartphones, used to operate in the old days.
But then somebody somewhere questioned a fundamental premise that drove so much effort and activity: Does it have to take so long from the time the company order new products to be manufactured until those products reach the retail shelves? Maybe there’s some part that often holds up the whole product; if we could only use a different part, or make the part ourselves, then we could get rid of a lot of the delays. Maybe the places where those component parts come from are farther away from our factory than they need to be; if we could just get them to move closer, then we could cut down on the lags. Maybe we have extra steps in our manufacturing process, or use outdated equipment; if we could only make some updates, then we can shorten the lag. And maybe if we do all of these things, as well as more generally finding and making changes anyplace where the process bogs down, then maybe we don’t have to put products on shelves at all. Maybe we can get the delay between order and manufacture short enough that we could start manufacturing the device when the consumer orders it and get it assembled and shipped fast enough that the consumer would tolerate the delay.
This is double-loop learning. Organizations not only use processes that allow them to make adjustments but also regularly examine the assumptions behind those processes that may be unnecessarily getting in the way of achieving organizational goals. We assume that we have to develop processes to mitigate bad product demand forecasts because we assume that those costs will be high because, in turn, we assume that manufacturing the product will take a long time once we decide to do it. But what if we’re wrong?
Double-loop thinking is a reasonably simple concept to understand but very hard to execute well and consistently. In the smartphone manufacturer example, think about all the many kinds of assumptions in the way things had always been done that would have to be identified, questioned, and replaced with a better-designed alternative. Particularly in the early days, when there weren’t models to copy or lessons learned elsewhere, no one person who could see all the changes that would have to be made. There would be many people across the organization—in manufacturing, product design, contract negotiation, shipping, retail relations, and so on—who would each be able to spot an individual sub-optimization in her daily work experience. And then more people would have to be involved in designing a solution to each sub-optimization, including accounting for all the ripple effects across other aspects of the organization. It would be an all-hands-on-deck sort of affair. Everyone would be needed to find problems, identify potential solutions, check those solutions for side-effects, and then implement them well.
Double-loop learning in academia
Now think about a few of the many questions that are starting to be asked about the operating assumptions about the education-related processes of colleges and universities:
Why must students stay in a course for a set number of weeks, regardless of how quickly or slowly they are capable of learning the material?
Why are students only able to register for and start a course at most a couple of set times in the year?
Why some very common teaching modalities based on the default assumption that all students learn roughly the same way and encounter roughly the same rough spots?
Why do we define the minimum math literacy for a college degree as basic algebra rather than, say, statistics?
We do we believe that professorial training requires at least five years of deep disciplinary education and at most one course in pedagogical education?
Why do faculty gain job security through research excellence far more than through teaching excellence?
Why do we assume that students know and understand everything they need to do from the moment they receive their college acceptance to the moment they arrive on campus for the start of their first semester of class?
Why do we assume we can know when individual students are in trouble and need help from the academic institutions when no employee of that institution sees the student for more than a few hours in the week—at most—and there is no good mechanism for sharing concerns and observations among the people who have contact with that student?
Think about the people who were in a position to spot each of these assumptions. Think about all the people required to design, troubleshoot, and implement alternatives that arise out of questioning the assumptions. If we want to reliably create student-ready colleges, then we need to be able to identify many unwarranted assumptions and design many alternative ways of doing things in ways that will deeply affect the ways in which academic institutions—and the people employed by them—work. To change everything, you need everyone. That specifically includes faculty.
A rubric as a vector for change
Way back in late 2013, I wrote about Pearson using a rubric to try to catalyze this sort of broad-based organizational shift in (critical) thinking: ((Pearson is a sponsor of e-Literate’s Empirical Educator Project.))
So if you’re the CEO of major textbook publisher and you want to unite the entire 45,000-employee company around a plan to transform the way the company does business, what do you do? Surprisingly, Pearson’s CEO John Fallon’s answer was, “I’ll create a rubric.”
I’m not going to analyze Pearson’s rubric in detail here…. I’ll say this much about it: It’s nothing special. It’s not bad, but it’s not genius either. There are plenty of flaws and limitations you could find if you worked at it and applied it broadly enough. There is no magic in it.
But here’s the thing: There is neverany magic in a rubric. The magic, when there is any, happens from the norming conversations that the rubric engenders. It happens when one colleague says to another, “What do you mean by ‘quality of evidence’?” Or “I scored that course a 2 on effectiveness. Why did you think it was a 4?” To the degree that the Effectiveness Framework proves to have any magic for Pearson, it will be in the norming conversations that it engenders across the company. Like our hypothetical Berkeley president, Fallon is working with diverse groups within an institution that has a culture of independence and Balkanization. Some of this is for good reason; conversations about effectiveness in chemistry education should look very different from conversations about effectiveness in fine arts education. Some of the fractiousness is about lack of a common culture and language necessary to discuss what otherwise arecommon challenges. And some of it is just human territoriality and self-interest. The first two challenges might be addressed by having a deep and wide ongoing norming conversation about a rubric that is general enough to cover a wide range disciplines and products but focused enough to provoke important discussions. The goal is for that conversation to become the basis for a new culture. The third challenge might be addressed by reinforcing that culture through your HR and other business practices.
Since I wrote that post, Pearson has developed a set of rubrics for evaluating whether a given product supports research-backed learning design principles. They have rolled those rubrics out to every product team and trained their product teams on how to use them. They have released them under a Creative Commons license and are. (For more on both the resource itself and Pearson’s interest in working with academics to make them more useful to academia, see the talk given by Pearson’s Global Head of Efficacy and Reach at last year’s Empirical Educator Project summit.) So Pearson continues to use what is essentially an academic strategy, not that different from the one being rolled out by California OEI, to build a double-loop culture around designing educational content and software functionality that are more effective at impacting student outcomes.
The rubric development, training, and norming processes are necessary but not sufficient. As I suggest in that last sentence of the Pearson post quote, other organizational processes need to be put in place as well in order to get the desired effect. It would be easy to get faculty thinking that the new practices are baked into the rubric, and as long as everybody is aligned with them, you’re good. The organization goes through the double-loop, but only until the norming process is complete. This is, in fact, what happens in many colleges and universities that adopt course quality rubrics. The institution has to mindfully employ the rubric updating, retraining, and renorming processes as methods for collaborative innovation. The rubric needs to be designed at a high enough level that it invites discussion and thought rather than rote implementation. The processes around it need to be collaborative rather than broadcast-only. And many other processes—like compensation for time invested or rewards for innovation, to take a couple of obvious examples—need to be created or modified to support and dovetail with the rubric processes.
There are lots of organizations that implement course quality rubrics. Enough that we should be able to start gathering stories and effective practices for using them to foster continuous organizational improvement. If anyone has a good example, please let me know.
If you’re a regular e-Literate reader, you know we have a macro thesis that the higher education sector is in the early stages of an evolution from having a philosophical commitment to student success toward having an operational commitment to student success. In other words, colleges and universities are starting to approach student success systematically, not as the natural by-product of hiring good faculty but as something that every student-facing aspect of the institution needs to be optimized for.
There is no road map for making this transformation and a number of formidable obstacles to it. First, academia was simply never designed for this purpose. The civilizational goal of empowering every human to live up to her or his potential via access to higher education is very new. Much newer than higher education system itself. In fact, it’s almost a thousand years newer. The University of Bologna in Italy, which is the world’s oldest university in continuous operation, was founded in 1088. The Morrill Land Grant Act, which created the first public universities in the United States, was passed in 1862. The G.I. Bill passed in 1944. Pell grants were created as part of the Higher Education Act in 1965. In 2018, achieving the as yet unrealized ambition of access to higher education regardless of income is very much a live political discussion. Just this month, the Sacramento Bee reported on a poll showing that 58% of Californians view college affordability as “a big problem,” with another 25% saying it is “somewhat of a problem.”
Even newer is the idea that we should not only be giving universal access to higher education but also taking responsibility to ensure that, once students have access, the institution is maximizing their chances of success (as opposed, for example, to the much older and still much more common idea of elitist “weeder” programs that filter for the “best” by failing out most). The deep structure of academia, from its governance to its professional training to its funding structure to its culture, is the evolutionary product of serving different missions than the one which we are now asking it to serve.
Second, even if we agree to embrace the mission of universal access and affirmative responsibility for student success in higher education writ large, how that plays out is very different at, say, Stanford, Loyola Marymount, UC Berkeley, Cal State Chico, and Los Angeles City College. The requirements for access are different. The definitions of and requirements for success are different.
And then their are the students, each of whom comes with her own definition of success, life goals, strengths, needs, and life context.
This is a hard problem. One that drives a lot of our work and our thinking. In a series of posts, I’m going to try to lay out what that shift looks like in a variety of academic contexts, how a successful shift across the sector would impact the future various ed tech product categories, and how the Empirical Educator Project (EEP) is intended to foster a methodology for empowering that shift.
In this first case study, I decided to start with the California Community Colleges Online Education Initiative (OEI). ((Disclosure: CCC OEI is a consulting client of ours.)) In fact, much of the structure of this post is drawn from an analysis we wrote on their behalf for the California State Legislature. I’m interested in extracting some generalizable lessons from OEI’s design. It’s important to be clear that the story I’m telling here is compatible with but not quite the same as OEI’s official position as represented in the report that they submitted to the legislature. OEI is also an interesting place to start this post series because, as we will see, it operates in an extreme environment that makes it particularly instructive.
This is a story about the whole being greater than the sum of its parts. OEI has put together a number of pieces that other institutions also have put in place, either individually or in various combinations. But they have done so with larger strategic vision for the future of California Community Colleges firmly and consistently in mind. It is also the story of a work in progress. California OEI has some impressive early successes under its belt. But it is also a hugely ambitious effort with much still to achieve. (Its in-process merger and rebranding with California Virtual Campus (CVC) is one example of forward-looking plans that I will touch on later in this post.)
The California Community Colleges is the largest system of higher education in the nation, with 2.1 million students attending 115 colleges. Our colleges provide students with the knowledge and background necessary to compete in today’s economy. With a wide range of educational offerings, the colleges provide workforce training, basic courses in English and math, certificate and degree programs and preparation for transfer to four-year institutions.
That’s a lot of students and a lot of colleges. One more college than last year, in fact. The legislature just approved the creation of a 115th campus (which will be virtual). California Community Colleges cover a lot of ground—literally as well as metaphorically. If you were to drive from College of the Siskiyous, which is about an hour south of the Oregon border, to Imperial Valley College which is about 20 minutes from the Mexico border, you would have to travel over 820 miles. They serve the top 100% of students. A lot of ground indeed.
There’s one word you won’t find in that rather dramatic description of California Community Colleges: “system.” Many state college and university systems are pretty big on local control, but California Community Colleges takes that principle to an extreme. For example, despite being a program intended to serve the entire system, OEI is run out of the Foothill-De Anza Community College District (after winning a competitive grant) because the Chancellor’s Office of California Community Colleges is more or less forbidden the legislature from running it centrally. California’s legislators are fiercely protective of the autonomy of their home districts. And as far as I know, OEI has negligible power to compel campuses to do anything.
In that environment, how do you help the entire 2.1-million-student, 115-campus, 820-mile-long “system” move together toward better operational excellence in enabling student success?
Academics tend to bristle at terms like “business drivers” and “business processes” when applied to academia, and there are good reasons to be cautious about using them. I’m applying the terms narrowly here because terms like “sustainability” are less effective at focusing people’s thinking about the machinery of balancing budgets. At the end of the day, colleges and universities need to take in as much money as they spend in order to keep fulfilling their mission. If you want to think clearly about how the sustainability machine works, then business is not a terrible metaphor. People have a basic, intuitive sense of what kind of machine a business is. The same sort of machinery is obscured the moment you start using words like “university” or even “institution” (or “sustainability”).
What do businesses (or sustainability machines) need? Money. There are a number of ways to have more money. One is to spend less of it. So one of OEI’s first moves was to offer to pay for the campus’ LMS, thus relieving each campus of the need to spend that money. LMS licensing may be an insignificant expense for an R1 university with a big endowment, but for a community college, it matters. There is no wiggle room in the budget. Hard choices have to be made—choices that impact student access and student success. We found anecdotal evidence that campuses have been using the money freed up by OEI’s LMS subsidy to invest in student success.
Our courses are much improved. In one year we have had 75% of current online instructors are fully certified. Almost 80 additional faculty are in process of being certified. We have approved 46 online course sections and reviewed or are currently review this semester another 35-40 courses. Without the resources from OEI and @one, we could not have made this happen.
– Faculty Senate Curriculum Chair, College of the Desert
Funding that would be used for [the common course management system] can be redirected to training for faculty who need extra help learning HOW to teach online.
– Dean, Business, Technology, and Career Technical Education, Ohlone College
Part of the people/resources that Coastline was able to shift, include our new Faculty Success Center, whose staff was able to create a new online course template in Canvas that helps faculty design a quality course. In addition, we were able to devote trainers to help faculty learn to use [the common course management system]. In this environment, our Academic Senate then felt comfortable mandating training for online instructors, something we never had before. I believe all this would not have happened if we had to pay for the [the common course management system] license….
So, continued state/OEI support for the…license will be critical for us to continue to train/support faculty and disseminate the use of these [OEI support] apps and support services….
One thing we were able to do, due to the free license, is pay all District faculty a stipend for the completion of [course management system] training.
– Associate Dean, Distance Learning, Coastline Community College
So just saving the campuses more money, by itself, led to actions by at least some campuses invested in improving their operational excellence at enabling student success. But that was really just the beginning. First, to get the subsidy, the campuses had to agree to do certain things. One of which was to adopt the same LMS. There were reasons for this, which I’ll get to shortly. For now, consider the likelihood of getting that many resource-strapped community colleges to migrate LMSs. How hard would it be? How long would it take?
All 114 campuses signed the contract agreeing to move to the common LMS, and many moved quickly to implement. In fact, the migration proceeded so far ahead of schedule that OEI had to go back to the legislature and request additional funding to cover the unanticipated extra subsidies. Saving these campuses money was a powerful motivator. And, as we’ll see, OEI accomplished a lot more than meets the eye with this one seemingly prosaic move of subsidizing an important but work-a-day piece of enterprise software.
How else can businesses make more money? By doing a good job of aligning their investments with their business opportunities. A grocery store doesn’t want to overstock with produce that will spoil on the shelves. But it also doesn’t want to run out of that produce when there’s high demand. And demand is variable. Demand for produce the week before Thanksgiving is likely to be different than the week after.
Colleges have an inventory management problem too. Sometimes courses are under-enrolled; other times they are over-enrolled. Both represent money problems to the campuses. One of the reasons that OEI wanted all the colleges on the same LMS—not just the same brand, but the same instance—was to create a course exchange. Balancing course “inventory” in a single community college is tough. Room availability, instructor availability, changes in the job market and economy, and the unpredictability of part-time student enrollments all work against you. But balancing “inventory” across 114 community colleges is less hard (once you can figure out how to get it to work in the first place). One campus may be over-enrolled in macroeconomics, but chances are pretty good that one of the 113 other campuses is under-enrolled in the same course. If you can get enough campuses to put enough courses on the online course exchange, then you can solve a “business” problem for all of the campuses. And the more courses there are on the exchange, the more valuable it becomes to the campuses. Thus, colleges have incentives to create courses for the exchange, and the more courses that are created, the more incentive the colleges have to utilize the exchange.
You could tell this same story from a student access perspective. Over-enrolled courses prevent students from taking them in a timely way. If the course is required, this could force them to delay graduation (and a full-time or better paying job), take on additional unneeded courses in order to qualify for financial aid, take extra financial aid from the state and federal governments, take up an enrollment space that might have gone to other students, and increase the risk that they will not graduate. Under-enrolled courses risk cancellation, with many of the same knock-on effects. I don’t mean to neglect or downplay this portion of the story.
But the focus on business incentives lets us think more clearly about the machinery of the institution itself. Which, in turn, helps us to think clearly about how that machine works and how it can be tuned. OEI designed a machine to drive operational excellence at enabling student success across the largest community college system in the country. And it runs on only positive incentives because. This design constraint immediately rules out copying some of the most frequently cited examples of innovative universities which, through one mechanism or another, can exert varying degrees of top-down control. At ASU, President Michael Crow has an unusually strong hand to play within a reasonably traditional structure of faculty shared governance. (Ithaka S+R has some interesting and revealing interviews of some of ASU’s top leaders that give some hints about how that governance works.) Western Governors University is more extreme; there is no faculty senate and no shared governance. SNHU’s Paul LeBlanc has tried a combination of strategies, working with with the faculty senate on governance of the traditional college while separating out their College of Online and Continuing Education (COCE) and running it in a way that is only loosely coupled to the shared governance of the rest of the university. Like many universities and systems, OEI cannot redesign the machine from the top down. so thinking about the live-or-die campus sustainability incentives that could be used to drive collective action has been a central principle that influenced the rest of OEI’s design.
Creating the infrastructure
The desire to move all campuses in the system to one LMS wasn’t just for the sake of contracting convenience. It accomplished a variety of goals. First, it became a foundational layer of software infrastructure for rolling out other system-wide capabilities and services, from plagiarism detection to online tutoring to faculty training and help resources. Having everybody on the same instance of the same platform—cloud-hosted Instructure Canvas—made it much easier to do this. In the old world, where campuses were on a hodgepodge of different self-hosted and vendor-hosted LMSs, the best the system could have accomplished would have been common contracting. It would still be up to each campus to integrate and support the tools and services. After all, the way a tool looks and works in Moodle can be different than in Brightspace. Centralized support would have been a nightmare. And remember, these campuses are very tight on resources. Supporting add-on tools and services is costly to them.
In addition, sharing one LMS made it easier for OEI to create faculty training that could be shared across the system, and for campuses to do the same. As with system-wide licensing for LMS-connected tools and services, it’s not impossible to do this in a system with different LMSs. But the added friction makes it less likely to happen. I’m going to use the “B” word again: academia needs to think about business processes. Once again, stripping away the culture- and mission-inflected language lets us see the machinery more clearly. A business process is the way in which a business accomplishes something that is important for the business. For example, how does a business make sure that all its employees have up-to-date software, including critical ones like system updates and the latest anti-virus software? Sure, they could leave that to the individual employees to do. We’ve all updated our software on our personal computers; it can be done. But how likely is it that everyone will do so in a fashion that is timely, reliable, and consistently correct? And what work are all those employees not getting done while they are wrestling with software updates? It’s better to develop a business process for pushing out those updates from a central IT group so that employees don’t have to worry about them. Likewise, there are effiency benefits to centrally rolling out and support services for 115 campuses than to have each campus IT support person duplicate the effort. From a perspective of strengthening the business drivers that hold the group together, all of these benefits can be boiled down to saving money by providing additional capabilities with reduced cost to on-campus resources (in direct licensing fees, support staff time, or both). As we have already seen, the campuses tend to invest the money they’ve saved in enhancements that are specific to their local needs and that benefit their students.
The common LMS also helps with the over- and under-enrollment problem. Having all course exchange courses on a single instance of a common LMS made it easier both to provide more data to the campuses that would help them with their planning and to reduce friction in expanding the course exchange. If everybody is using the same system, that’s one less thing for faculty and students to learn, less help desk support, and more productive support (for both course delivery and course design) because the OEI staff don’t have to try to accommodate multiple flavors of learning environments.
Of course, there are trade-offs, the biggest one being autonomy. In OEI’s case, for example, all the campuses had to agree to use the same LMS rather than choosing their own. Anyone who has run a campus LMS selection process knows it can be an exercise in delicate diplomacy. Imagine doing the same with 114 campuses. As we’ll see, OEI turned this challenge into an opportunity. I’ll have more to say about that in the next section.
Anyway, once you start seeing infrastructure as the structure “underneath” (i.e., “infra-“) that supports business processes, two things immediately start to happen. First, your definition of success changes. You can no longer declare victory just because you successfully installed the software and got people to start using it. You have to start looking at whether it successfully enabled or improved the business processes you were intending to support.
Our Professional Development Coordinator is encouraging the creation of Pro Dev workshops in [common course management system] Canvas, such as health and wellness (“dealing with difficult people”), how to create Open Educational Resources, how to use [Student Learning Outcomes] for better teaching, and a lecture on science and its assumptions. What is developed at Butte can be instantly shared with other schools, and vice versa. I see a renaissance of Pro Dev opportunities!
– Technology Mediated Instruction coordinator, Butte College
The second thing that happens when you start thinking in terms of business processes is you identify new problems as well as rethinking and reprioritizing old ones. For example, OEI is in the process of revamping (and merging brands with) California Virtual Campus (CVC). Why? CVC exists today. It’s a web-based catalog of online courses students can across the California Community Colleges and California State University System, which is many more than the handful of OEI exchange courses. As of 2017, CVC included 23,445 online courses and 1,376 degree programs. So it’s big. If you think of infrastructure as a thing to have, then you might think of CVC as a massive success.
But if you think about CVC as the structure underneath that supports the critical business process of students finding, (wisely) selecting, and registering for online courses across the many campuses represented in the CVC course catalog, then you start to develop different metrics for success. And if you also think about the back-end process of making sure the right institutions get the registration information, tuition, and transcript information (respectively), then CVC becomes both a critical priority and a tough piece of infrastructure to build well, particularly across so many different campuses that are not all migrated to a single instance of common Student Information System (SIS) software. If students fail to register for online courses that they need because the process is too cumbersome, or if they don’t get properly credited by their home institutions after taking an exchange course, that is bad for the long-term health of both the students and their institutions. This is what it really means to say that some infrastructure is “mission-critical.” CVC is a big catalog with lots of courses, but it does not yet do a great job of fulfilling its mission-critical role of helping students find, register for, get credit for, and pay for their courses. Each of those is a business process that CVC should support. And the number of courses in the catalog tells us very little about how well the software is supporting those processes for the students and the campuses.
If you think about infrastructure in this way, then your communications to your stakeholders will also change. Here’s an explainer video they had us create for them in order to help communicate that message to the various campus folks:
(Source video: https://youtu.be/1DdlaIZYiDI)
Why was this so important to communicate? OEI could have gone to their constituents with a message of “Here’s a bunch of great free stuff for you!” Instead, they chose a much more challenging message to communicate; one about the ripple effects of having shared infrastructure. That wasn’t an obvious choice.
If you’ve read any one of a million articles on how to succeed with any major campus-wide initiative, you will have read the cliché about how important it is to “get buy-in.” Most of the time, “getting buy-in” is interpreted as “handling objections” or “reducing resistance.” It is an obstacle to get past. But the video above shows that the OEI leadership has interpreted the term differently. You only communicate to your stakeholders in this way if you believe that buy-in is infrastructure.
Fostering a culture
In our consulting work, we facilitate LMS selection processes reasonably often. The more forward-thinking institutions view these processes as opportunities. How often do you get to gather a group of faculty and other academic stakeholders from across your institution in one room and have them talk to each other about how they teach and what they need to serve their students well? A search for a product like an LMS can become a rare opportunity for focused, intensive, purpose-driven community-building. Yes, it lowers resistance, enabling people who might not be happy with the final decision to at least feel like they were heard. But it also begins to foster familiarity and dialog that can help foster a broader and more lasting community of purpose. When you’re trying to build such a community across 114 campuses as part of an ambitious and completely voluntary system-wide effort, taking that view of LMS selection is even more important. Needless to say, it wasn’t easy. Or seamless. That said, both during the selection process and and afterward when communicating the results, OEI worked toward building affirmative buy-in—not just lowering resistance but increasing the sense of goodwill and common purpose. For example, the selection committee was near unanimous in its selection, with the sole dissenter acknowledging the importance of what the committee was doing together and supporting the final decision.
The program design elements I’ve described so far helped OEI to foster increased organizational alignment at two levels across campuses. The campus executives who are responsible for financial health and sustainability of their campuses are aligned through infrastructure subsidization and the course exchange. In 2018, the state legislature decided to augment the initiative with an additional $35 million funding. California Community Colleges has chosen to invest that extra money capacity-building. In particular, the money will go toward grant programs intended to enable the campuses to launch more online courses on the OEI-CVC infrastructure, thus further strengthening this alignment while aiming to serve more students effectively. The common infrastructure and the culture-building process around it has helped to build a culture among the academic and technical support staff across campuses. The hoped-for consequence is that improvements on one campus will travel more quickly and easily to others:
I now know that I can ring up any other [Distance Education] Coordinator and we’ll be speaking the same “language” regarding the use, training, and administration of the [course management system]. I’m also really looking forward to faculty being able to share ideas and resources via Commons.
– Director of Distance Education, Santa Rosa Junior College
Building a similar sort of sharing network among the faculty in the system is an even larger challenge. The culture-building work has been ongoing work for years now, but one can acknowledge all the hard work and progress to-date while still also recognizing that this is a huge project that has barely begun. Certainly, having common resources and common platform supported by OEI has provided a boost, as has the inclusion of faculty voices in the OEI planning process. The augmentation grants, which will likely include instructional design support, represent another opportunity. But to me, one of the most interesting vectors for culture-building is the course exchange course quality rubric. Every course on the exchange has to be evaluated against a rubric of evidence-backed effective online teaching practices. As the pace at which exchange courses are developed increases, OEI will not be able to keep up with demand to evaluate these courses using central staff. So they are creating a peer reviewer mechanism in which faculty on the campuses are trained on the rubric and presumably compensated to review courses that are candidates for the exchange.
This opportunity fascinates me. We know that faculty who go through an expert-supported course redesign process often experience intellectually deep and emotionally moving shifts in their teaching strategies. Is the same true when faculty are trained reviewers of their colleagues’ redesigned courses? What effect will simply exposing faculty to more and different course designs have? How will their role as reviewers and critiquers shape or enhance that effect? Can a continuously improved and updated rubric become a vector for sharing new research-supported processes across the system on an ongoing basis? Will the impact be broad and deep enough to foster new kinds of intra- and inter-campus faculty dialogs about the scholarship of teaching and learning (SoTL)? Will these cultural changes help to foster alignment around continuous operational improvement for enabling student success? This is the last mile problem of higher education. Operational excellence at student success cannot be achieved unless it is infused in the daily operations in individual classrooms. That requires affirmative faculty buy-in, support, training, and embedding in a culture that invites them into the larger conversation.
This is highly reminiscent of the cultural transition that doctors had to make from the mid-Nineteenth through the mid-Twentieth Century. In the 1840s, one could begin practicing as a physician with no medical training at all, just as one can start practicing as professor with no pedagogical training today. Doctors learned medicine from whomever they happened to train with and whatever they read in the newspaper ads about cures and treatments ranging from early antiseptics to leeches and literal snake oil, with no easy way to distinguish them. There were no major conferences or respected, peer-reviewed journals. There were no standards for quality research or convincing evidence. There were a handful of teaching hospitals and medical colleges of wildly varying quality that touched only a small minority of practicing physicians. All of these institutions, all of this social infrastructure, needed to be built and bought into by physicians before antiseptics could be differentiated from snake oil, the signal separated from the noise, regarding “progress” or “innovations” that might help their patients’ welfare.
OEI has accomplished some remarkable early successes in an extremely challenging context. But the degree to which they are able to move 114 California community colleges toward better support of student success as a group may well depend on the ability of the social infrastructure they are creating to reach faculty, be embraced by them, and and foster a culture in which academics collaborate differently and more intensively in their day-to-day work of helping students to succeed, one student at a time.
