I know, I know. This is exactly the sort of clickbait headline that e-Literate normally eschews. But bear with me, please.
Because my wife and I live in a rural area and are seriously thinking about going even more rural in our next phase of life, I have been researching rural broadband. I can learn to live with a long trip to the nearest real town, and I might even be able to learn to live with composting toilets. (Maybe.) But no broadband? No way.
So I’ve been researching the various trends in rural broadband. The short answer is that they do not look good. The economic incentives are not strong for either cable or phone companies to extend their networks. Could that change with Federal subsidization? Sure, to a degree, but only so much. And laying new cable or putting up new cell towers takes time. There’s a thing called “fixed wireless internet,” but it’s limited by line-of-site connections, so if you live in the mountains, then you can probably forget about it. Traditional high-orbit satellite internet connections are limited by really horrible latency, pretty much ruling out living in the Zoom world many of us have been inhabiting lately. I don’t see any of the usual suspects solving this problem at any time soon.
The next possibility is new low-orbit satellites. These are much, much closer to the ground than the internet/tv satellites of today, so that helps a lot with the latency problem. In fact, they may not be satellites at all. Google’s Alphabet parent company is looking at using balloons, while Facebook is researching high-flying drones.
Many of these efforts are still glorified research projects. But Elon Musk’s Starlink effort, which is currently operated by his privately owned SpaceX company, appears to be much further along. SpaceX has been launching satellites in batches of about 60 at a time. They are scheduled for their tenth launch this week, bringing their total close to 600 satellites launched so far. To put that in perspective, Musk thinks the minimum number of satellites for a commercially viable service is between 800 and 1,000 (though the company has filed with the FCC for permission to eventually launch 30,000). The company claims that once the service is operational it will provide gigabit speeds—for comparison, that’s ten times faster than my quite satisfactory cable broadband connection—and latency that’s certainly low enough for videoconferencing and will eventually be low enough for high-end video gaming.
Starlink claims it will be able to offer service to “Canada and the Northern United States” by “the end of 2020.” Obviously, even assuming that they meet their schedule, there’s a lot of ambiguity here. What does “Canada and Northern United States” mean? When exactly is “end of 2020”? The last article I linked to above claimed that the company will start accepting beta testers by the end of the summer.
There are significant technical hurdles that the company will have to overcome. First, Starlink hopes to compete in a Federal funding auction for $16 billion to apply toward rural broadband access, but they need to demonstrate that they can provide service with latency under 100ms. The FCC has expressed serious doubts about whether a satellite service such as Starlink can achieve this benchmark (which is significantly less stringent than the 8ms latency that Musk claims the service will eventually be able to achieve). The due date for that proof point is July 15th, so we’ll know very soon whether Starlink can cross that hurdle. Second, the receiver boxes needed by consumers for this service are very sophisticated and therefore expensive. The initial cost to build them could be in the $1,000 range, and Musk has said his goal is to get the cost down to $300 by 2023. None of that tells us how much they will actually cost consumers. First, since this is a subsidiary of a privately owned company, Musk has latitude to subsidize the costs. Second, particularly with the COVID situation, there could be considerable additional funding coming from the Federal governments (particularly after the election in the US). Musk is initially focusing on rural areas that don’t get broadband service, so a combination of subsidies to his satellite service for those areas and the more traditional providers for low-income families in areas that get traditional service could be a reasonable approach.
To sum up, while it is highly doubtful that SpaceX will provide a universal broadband solution for the US in time for the September start of the school year, it is not crazy to imagine that he could provide a technically practical solution over the course of the next school year that could be made affordable to rural students via government funding and subsidization by the company. Needless to say, this could be a game-changer for online learning. It’s just one piece of the equity problem, even if it’s universally available and affordable. But it would be a big step in the right direction. And I don’t see any other plausible solution candidates in the near term.
We will be reaching a major proof point in about a week.
If your main intellectual response to the current health crisis is to double down on your previously held beliefs, then you are probably not thinking deeply enough. For example, suppose that we develop both treatments and a vaccine for COVID-19, but the immunity conveyed by that vaccine only lasts for a season (much like other flu vaccines). Suppose that flu strains become permanently more virulent and more deadly. Some of the changes in the healthcare system that we are seeing now—in point-of-care, supply chain, and research efforts—become permanent. The risk of getting together physically in groups is permanently altered. Not to the degree that it is now, but significantly higher than it was. Think about how that permanent change would impact all the many businesses that are being impacted now. The growth sectors of our economy would change. The geopolitical landscape would change. Everything would change. Much of what we thought we knew about the future of work would have to be thrown out and rethought. And much of what we thought we knew about the future of education—what jobs we need to prepare students for, how we should be teaching, what sustainability looks like for an educational institution, and so on—would also have to be thrown out and rethought.
It is hard to think such thoughts. I find my mind desperately casting about for reasons to believe that this scenario is not plausible. That life will someday, somehow, return to normal. That the world has not changed.
But it has. And we must all learn how to think the unthinkable. We have to find ways to let go of the comforting assumptions that enable us to glide through the day on autopilot, somehow not swerving off the road while we fiddle with the car radio or talk on the phone or think about what will happen when we arrive at where we are going. We have to look at the road with fresh eyes. See what is in front of us. Prepare our minds for unexpected obstacles. The adaptive cruise control and lane-assist and other comforting tools that we have built to protect us from the need to be in the moment will not serve us well on this road we have found ourselves on. In fact, they provide a dangerous illusion of comfort. We must seize the wheel and learn to be navigators again.
Right now, as we respond to an unprecedented crisis, a lot of the safety mechanisms in the system are being thrown out even as we face massive uncertainty. Multi-trillion-dollar legislation is being passed through Congress incredibly quickly. The medical system is finding various shortcuts to get around the system of checks that are designed to make sure treatments that tinker with your immune system are safe. These safeguards were created for good reasons, one of which is to mitigate the impact of sloppy or reflexive thinking on consequential actions. If we are going to remove them, then we must think harder.
Even under normal times, I would have been disturbed by the Christensen Institute’s argument that “California should double down on Calbright College—by leaving it alone.” The post author Richard Price asserts that California’s new, experimental online community college, which received a $100 million initial allocation and a $20 million annual budget from the California State Legislature, is already fabulously successful. And it will be even more successful if only it is freed from the shackles that the other 114 California community colleges manage to live by and State Senators would back off of their call for Calbright to be audited for its finances, progress, and its compliance with the conditions under which it was authorized. He makes these assertions without evidence or even much in the way of argument to back them up. While reading the piece, I can’t escape the feeling that I am in the back seat of a car with a habitually distracted driver who is reflexively trusting his GPS to guide him through territory that has not been mapped.
I have been patient with the more careful proponents of disruptive innovation in education, despite my skepticism, because it is not their fault that the term has been appropriated by testosterone-addled Silicon Valley teenagers, and because they have made cogent (if unconvincing) arguments. But the opening line of Price’s Calbright post, while sounding completely anodyne, alarms me:
As the COVID-19 outbreak increasingly prompts universities to move instruction online, California lawmakers may want to reconsider their deep skepticism of the state’s first fully online community college—the young institution is proving itself more vital than ever.
There are just so many things wrong with this sentence. For starters, the author links to an article about lawmakers’ “deep skepticism” without mentioning anywhere in his own piece that the article describes the reasons why lawmakers are skeptical. He certainly doesn’t attempt to refute those reasons. To the contrary, he asserts that the “young institution is proving itself more vital than ever” without ever really explaining what that means. Second, the sentence conflates skepticism about Calbright with skepticism about online education. Reading it, one would never guess that the California Legislature has poured something like a quarter of a billion dollars into online programs for the California Community College System alone, plus hundreds of millions more into the Cal State and UC systems. Lawmakers are not demonstrating skepticism toward online education. Rather, they are demonstrating skepticism of Calbright’s implementation as the best application of the substantial investment they are making in online education.
Worst of all, the introduction sets the tone for the article, which is to accuse others of dogmatism while itself making its case on pure dogma. In doing so, it links to another post on the Christensen Institute’s site by another author about another system. This article, unfortunately, also gave me the feeling of being an uncomfortable passenger sitting beside a driver who is barreling down the road toward a destination that he is determined to reach, heedless of what may lie between here and there.
The end result is a specious chain of arguments made from a position that indirectly has Harvard’s imprimatur—because Christensen was a noted Harvard professor—arguing that the state of California should commit to throwing tens of millions of dollars at an unproven program. And that it should do so while declining to hold that program accountable or even expecting it to follow the rules that all the other community colleges manage to live under. The main justification for this position is a theory that was designed to explain what happened to steam shovels and old disk drive technologies and is now being applied to a dramatically different domain without attention to context or evidence.
If you are going to argue for removing safeties at a time of historic emergency, then you must think harder.
It is time for the more serious proponents of disruptive innovation in education to take a good look in the mirror and ask themselves if they have been thinking hard enough. As their predictions about mass closures of universities and other dramatic changes have failed to materialize repeatedly over many years, their assertions about the future have become increasingly strident and sloppily constructed. It is time for them to seriously consider the possibility that they are just plain wrong, and that their attachment to cherished ideas is doing more harm than good.
My own prior assumptions
If I’m going to be fair, I should lay out my own prior thinking about both disruptive innovation and Calbright so that you can decide whether I am the one who is stuck in old thinking.
I am a fan of Clay Christensen’s work. I have three of his books on my shelves, all of which I have read at least once, and two of which I find myself occasionally dipping back into for reference or inspiration. I find his basic thesis to be clever, well-argued, reasonably grounded in evidence, and provocative. That said, his theory is interesting because of the inference chain he constructs rather than because any of his individual insights are particularly novel in and of themselves. For example, one central concept in disruptive innovation starts by “competing against non-consumption,” which is a fancy way of saying you should go to where the competition isn’t by figuring out who isn’t being served well. While this assertion is worth noting, it is the opposite of revelatory. It borders on the banal. Another stepping stone in his argument is that successful organizations will tend to resist ideas that are different than their historically successful ways of doing things. We nod at this without pausing. Christensen’s argument pulls us along not because the steps along the way stop us in our tracks but, to the contrary, we glide down an easy path and suddenly end up somewhere unexpected. As Price puts it in his article, “In the Innovator’s Dilemma, Clayton Christensen made an unsettling observation—CEOs of established companies that fell to disruptive entrants often did everything right, and for that very reason failed to save their companies.”
Disruptive innovation theory boils down to the idea that companies that are fat and happy tend to result in products that are fat and customers that are less happy. At some point, Microsoft stopped building features into Office that were useful to everybody and started focusing on features that were useful to their most lucrative niche customers. Office customers who were not in those niches became less happy as the software became more expensive and harder to use. But the company was focused on maximizing revenues from its best customers.
So they didn’t react much when Google came out with Docs, Sheets, etc. This was a perfectly reasonable business decision for them to make, since Google was not an immediate threat to their customer base and some very profitable corporate customers wanted specialized features. Google’s products were obviously inferior for mainstream word processing tasks. I tried and abandoned Google Docs a few times in the early years because it just didn’t meet my needs. But it got better over time, first satisfying needs for which Word wasn’t well suited, then nibbling at the simplest applications of Word, and eventually becoming sophisticated enough to challenge the dominant word processor for a substantial percentage of common tasks. Today there are a billion users of Google Drive. I am one of them. I have not abandoned Microsoft Office, but I have thought about it at times.
Disruptive innovation is both a particular kind of disruption and a particular kind of innovation. It is important to understand this. When we learn to trust a theory, we begin to take it for granted like we take our cruise control for granted. We stop thinking about it. But we need to understand how our tools work so that we can know when we are safe to trust them and when we are not. As I put it in 2014,
These days, “disruptive” and “innovation” seem to always come together in the same sentence. It’s a bit like “big galoot.” Theoretically, “big” is a modifier for “galoot.” But you never hear people talking about small galoots, average sized galoots, or galoots of indeterminate size. In modern common usage, galoots are always big. “Big galoot” has pretty much become an open form compound word, like “post office” or “real estate.” But “disruptive innovation” is not a compound word. Disruptive innovation is a particular kind of innovation, and a fairly narrow kind at that. Specifically, disruptive innovation is a phenomenon in which a new market entrant can overtake a leader in an established market by offering cheaper and simpler solutions. It’s important to remember that some of Clayton Christensen’s seminal examples of disruptive innovations were steam shovels and disk drives. This is not the same kind of innovation that produced the iPhone. It’s essentially about identifying the slow fat rich kid and taking his lunch money. To be fair, it’s not that inherently mean-spirited, because presumably one takes the rich kid’s lunch money (or market share) by providing solutions that consumers prefer. But the point is that disruptive innovation is generally not about solving new problems with brilliant out-of-the-box ideas. It’s primarily about solving old problems better because the old solutions have gotten overbuilt.
The reason that I have been skeptical about disruptive innovation’s applicability to education has to do with differences in both demand and supply from the situations that Christensen analyzed in his books. On the demand side, I argued in 2013 that education doesn’t fit the disruptive innovation framework because it is not a product:
Silicon Valley can’t disrupt education because, for the most part, education is not a product category. “Education” is the term we apply to a loosely defined and poorly differentiated set of public and private goods (where “goods” is meant in the broadest sense, and not just something you can put into your Amazon shopping cart). Consider the fact that John Adams included the right to an education in the constitution for the Commonwealth of Massachusetts. The shallow lesson to be learned from this is that education is something so integral to the idea of democracy that it never will and never should be treated exclusively as a product to be sold on the private markets. The deeper lesson is that the idea of education—its value, even its very definition—is inextricably tangled up in deeper cultural notions and values that will be impossible to tease out with A/B testing and other engineering tools. This is why education systems in different countries are so different from each other. “Oh yes,” you may reply, “Of course I’m aware that education in India and China are very different from how it is here.” But I’m not talking about India and China. I’m talking about Germany. I’m talking about Italy. I’m talking about the UK. All these countries have educational systems that are very substantially different from the U.S., and different from each other as well. These are often not differences that a product team can get around through “localization.” They are fundamental differences that require substantially different solutions. There is no “education.” There are only educations.
Students’ reasons for the college choices they make are much more complex and varied than their reasons for choosing a word processing program. So I’ve been skeptical about whether the nebulous cloud of ideas and ideals that we call “education” is a thing that can be disrupted in the Christensonian sense.
There are also mismatches to the theory on the supply side. Christensen’s theory of organizational behavior fits best with organizations that conform to economist Ronald Coase’s theory of the firm. Command-and-control organizations that are optimized for profitability and growth will be particularly vulnerable to the pathologies that he identifies. Most universities don’t function that way, and no university systems do. While universities do have to worry about optimizing for sustainability, that goal is complicated by their strong mission focus and their shared governance structures.
And university systems are even further from Christensen’s model because they are governed by political systems, which operate under entirely different decision-making processes than companies do. For example, with 114 physical community colleges grouped into 72 districts across California’s 58 counties, every state legislator has a local interest in protecting one or more of those colleges. There were mission-related reasons for this design decision, having to do with making education accessible in an era before modern online learning. Regardless of the degree to which those reasons may or may not still apply, decisions for the California Community College System are still driven by political representation of those county interests in the state’s legislature. Christensen wrote an entire book—The Innovator’s Solution—on how entrenched corporations can stay innovative. As far as I can recall, none of the example organizations in that book looked remotely like the California State Legislature. So at the very least, somebody would need to make the affirmative case that Christensen’s prescriptions for corporations can be applied in this very different context.
And remember, it’s the chain of inferences that makes disruption theory interesting. If any of the links break, then we have to question whether the theory holds. Cherrypicking evidence for one or a couple of the individual common-sense assertions in the inference chain puts us on a fast road to Stupidville, however our GPS display may label the destination.
I have historically been skeptical of Calbright for reasons that have less to do with disruptive innovation and more to do with math. COVID-19 has been giving us all a hard lesson on the mathematics of growth. If every person carrying the disease spreads to between two and three other people, then the number of people with the disease will double every few days. On the other hand, if the average infected person spreads the disease to at most one other person, then it will eventually die off.
The challenge of Calbright has been that the math never worked for the kind of growth that the legislature expected it to achieve. If the main goal was to create an incubator that would eventually teach us better ways to serve underserved students, that would be one thing. But the goal was to actually serve millions of underserved students in a relatively short span of years. Calbright’s growth rate would have to be unprecedentedly high in order to achieve that aim. Phil Hill broke it down in a post in 2017:
The maximum growth rate of [some] cherry-picked successful schools ranges from ~1,200 / year for Excelsior to ~7,700 / year for SNHU (note that Rio Salado at ~1,400 / year is the only public institution). Add to this the fact that all of these schools have been around for decades. No accreditation issues, no time-consuming establishment of core leadership team, etc.
There is a big difference in dealing with institutional issues and statewide issues, particularly in California. One in five US community college students in the US do so in California, and the statewide issues tend to come in large numbers. Statewide issues tend to come in hundreds of thousands while institutional issues tend to come in tens of thousands.
What this points to is that for a new fully-online institution to get to some meaningful level of enrollment (let’s say 20,000) in the same ballpark as these comparison schools, I estimate it would take a full decade at the least. This is the reason, by the way, that Mitch Daniels and Purdue University made the Kaplan University deal even though Kaplan’s enrollments are dropping. Daniels did not want to wait a decade to get to meaningful enrollment numbers for an online college serving working adults – if everything works out, within a year Purdue will have a fully-online institution serving 30,000+ working adults. That is a big if, by the way.
I have no problem with the idea of Calbright as an experiment in supporting underserved students. But the math doesn’t support the idea that it is going to have an impact at scale any time in the next decade, even under the most optimistic of scaling scenarios. So my concern with Calbright is that both California politicians and proponents of disruption theory have seized on it as a means for solving different problems than the ones it seems suited to solve.
I have one more predilection that you should know about, which is that I believe theories should be disprovable if they are to be credible. You can’t say that disruptive innovation is going to result in 50% of schools going bankrupt—no, wait, maybe it’s 25%—and that it’s MOOCs that are the disruption—or maybe just online learning in general, or possibly CBE, or maybe adaptive learning—and continue to be taken seriously. There has to be a way to arrive at the statement, “data X would disprove my hypothesis.” It can’t be the case that any old data can support your hypothesis while no data can disprove it. That’s not a theory. It’s dogma. My patience with proponents of disruptive innovation in education has worn thin over time because I haven’t seen them demonstrate the willingness to ask themselves the question, “But what if we’re wrong?”
Now that I have laid my own comforting assumptions on the table, it’s time to examine the assumptions that are evident in the arguments from the Christensen Institute.
Is Calbright a disruptive innovator?
Let’s return to the hyperlink in the very first sentence of Price’s post, which is attached to the words “their deep skepticism.” What is the skepticism described in the Education Dive article behind that link? First, the article emphasizes the question of whether Calbright would be best run in the existing structures rather than as an independent campus. That’s one surface resemblance to disruption theory. Again, you don’t have to believe in disruption theory to believe that sometimes a little autonomy from a large, entrenched bureaucracy might lead to fresh thinking. But at least it’s consistent.
On the other hand, Price neglects to mention other salient facts. Like the fact that the legislative analyst’s report in 2018 said there is no evidence that Calbright will solve the problems that prevent Californians from enrolling in college (or, in Christensonian parlance, that it can compete against non-consumption). Here’s the relevant paragraph from that report:
Unclear If Providing Online Offerings Will Solve Key Barriers for Target Student Group. One of the proposal’s goals is to increase educational attainment for adults who currently have no postsecondary credentials. Although this is a laudable goal, the administration has not provided any evidence that an online community college will address the key barriers for this potential student group. Although an online program can increase convenience, working adults may not be pursuing additional education for a number of reasons. The administration also has not provided evidence that those working adults who are interested in more education cannot access it through existing online or in‑person community college programs.
California LAO report
Is there reason to believe that the innovations of Calbright’s design will cause large numbers of students who were previously not enrolling in existing online offerings to enroll in Calbright’s? I don’t know. The California LAO’s office didn’t see such evidence, and Price fails to even mention it as a problem.
The Education Dive article also mentions that Calbright has duplicated existing academic offerings of other community colleges in violation of state rules. Here Price does have a rejoinder, though he does not dispute that Calbright has violated state rules:
[C]oncerns that Calbright’s courses may duplicate existing OEI offerings fail to acknowledge that competency-based, self-paced courses differ fundamentally from seat-time based courses covering the same material, and better serve time-constrained adult learners.
OK, that’s a claim. Where is the supporting evidence? Such programs exist in the world. Western Governor’s University, SNHU’s College for America, and others implement the instructional approach that Calbright has chosen. I’m not arguing that evidence doesn’t exist. But I do think that if somebody is advocating for a $100 million initial allocation plus a $20 million allocation in support of a new and unproven institution while simultaneously advocating that rules should be relaxed and oversight waved off, then one should present any evidence so that we can examine it together. The action Price is calling for requires less reliance on cruise control and more on the blind-spot camera.
Rather than focusing on making an evidence-based case for Calbright, Price chooses to make an argument against California Online Education Initiative (OEI):
Consider CCCS’s Online Education Initiative (OEI), a course exchange program, which Calbright critics point to as obviating the need for a separate, online college. In theory, students can enroll in online courses offered by other state community colleges. In reality, CCCS’s enrollment-based funding model discourages school participation, since only the college that offers the course earns enrollment revenue. This has limited course offerings and engagement overall.
Here is an example of how Calbright, operating under a different business model, could be an asset to CCCS as an autonomous institution. Free from the constraints of the funding model that has proven incongruent with expanding online course options, Calbright could work towards revenue and course enrollment models that best serve large numbers of job-seeking adult learners.
This is factually wrong on several levels. First, the course exchange is just one component of OEI. Luckily, MindWires—the consulting practice that I used to be a partner in and that Phil Hill still runs—produced this convenient explainer on behalf of OEI (which is less than four minutes and is easily findable on YouTube):
Intended Consequences
(Video link: https://youtu.be/1DdlaIZYiDI)
As Phil recently wrote, Calbright is currently serving about 450 students. In contrast, in the 2020-2021 academic year, OEI projects 99,000 enrollments in new academic pathways. (And word is that the OEI number is likely to be revised upward significantly based on more recent data.) Apples-to-apples, this translates into probably a couple of tens of thousands of students.
The course exchange part of OEI is one component. It may or may not succeed at scale. But the broader OEI program is indisputably reaching many more students than Calbright. Worse than getting the incentive mechanism wrong for the course exchange, Price apparently failed to conduct even cursory research regarding the nature and impact of the program he was critiquing. Even if the course exchange portion of OEI fails utterly, the overall program is still tracking to increase enrollments by two orders of magnitude more than Calbright will in the next year.
That doesn’t even count the ways in which Calbright’s progress toward (unspecified) milestones was accomplished, in part, by using resources built with OEI funding. Here is a screenshot of Calbright’s essentials course:
The link at the bottom of that screen leads to this page:
Of the Calbright students enrolled in October through December of last year, 449 enrolled in essentials courses, while 20 enrolled in a program pathway. In other words, the vast majority of students taking Calbright courses last term were taking courses that were using OEI content resources.
Also, while Calbright’s mission was specifically to reach underserved students that aren’t being reached via the 114 existing community colleges, the results so far are not promising on that score. For example, despite specifically targeting Latinx students, they represent 16% of Calbright’s enrollments—in contrast to 45% Latinx enrollment system-wide. Calbright’s students are, in fact, disproportionately white:
In addition to these reasons for questioning Calbright’s progress toward and ability to accomplish the mission set for it by the State Legislature, there’s also the fact that Heather Hiles, former Calbright CEO, resigned abruptly without public explanation less than a year after being hired.
So given these facts, is it reasonable for California to ask whether Calbright makes sense and is working as intended at this time? I think it is. While I don’t prejudge the outcome of that introspection, the legislators have a responsibility to at least ask the hard questions.
But, worries Price, whether or not OEI may work in practice, does it work in theory? He seems to take a kind of libertarian position that any program that is regulated will inherently be inferior to one that is not. He is aghast at the LAO’s concern that Calbright will eventually have to comply with the system’s collective bargaining agreements and spend 50% of its budgets on instructional salaries.
In other words, Calbright being embedded in CCCS subjects it to strictures that will eventually hamstring the school in its efforts to innovate for adult learners. The LAO report ironically makes the argument that even as an independent school, Calbright isn’t autonomous enough.
Does it? How much does it cost to run a Calbright-style CBE program with quality? How much of that money goes into instructional costs? How does that compare to the requirements that it is currently required to comply with and that OEI online programs do comply with? Again, the data exist, but Price either doesn’t believe he needs to justify his position or doesn’t have the data in question.
I don’t have it either. What I do have is a quote from the President of Southern New Hampshire University (SNHU), which the Christensen Institute valorizes without understanding—more on that in a bit—about the amount of human time and attention that is required to deliver quality CBE education:
I think there is a general lack of awareness of how rich now the underlying data analytics are. We monitor our students 24/7. We know when someone hasn’t logged on. We know when someone has struggled with a project. We know when performance has dropped off. We actually have closer to a 360-degree view of our students than most traditional institutions do. Then, when those students are engaged in the work, they have ready access to qualified faculty if they’re really stuck. We’re never going to let somebody get stuck on a math concept, for example, and just say, well, just figure it out. We’re going to get you help.
SNHU is putting significant time and money into instructor support. Is it 50%? I don’t know, and Price apparently doesn’t either. Is there a case to be made for granting Calbright some flexibility based on evidence of results in successful programs? Possibly, but Price doesn’t make it. Instead, he simply declares,
Without an autonomous Calbright, CA will struggle to properly serve its adult learner population. Innovators like WGU and SNHU will take good care of these learners, but CA will have ceded, rather than seized, the future of learning.
With little in the way of argument and nothing to speak of in the way of evidence, he concludes that (a) only Calbright can “properly” serve California’s adult learner population, (b) Calbright can only do so if it is “autonomous,” (c) Calbright represents “the future of learning,” and (d) if the California Legislature doesn’t cede oversight Calbright while continuing to fund it, then out-of-state competitors may disrupt or otherwise somehow surpass the California Community College System. It’s not at all clear how Price got us here, or even what he means by “properly” serving California’s adult learner population, or making Calbright “autonomous,” “ceding” or “seizing” the “future of learning,” or what role he believes that “innovators” such as WGU and SNHU will play in California’s future should the State Legislature fail to follow his prescription. He does link to (without really explaining the relevance of) another Christensen Institute post by another author about how the PASSHE system partnered with SNHU. I wish I could say that I found that article more persuasive.
Alas.
Is SNHU disrupting PASSHE?
Michael Horn’s post from February of this year bears the breathless title, “Why disruptive innovation is stealing Pennsylvania’s students.” The trigger for the article was an articulation agreement signed between Pennsylvania’s State System for Higher Education (PASSHE) and SNHU which enables PASSHE students to transfer up to 90 credits to SNHU and complete their bachelor’s degree at SNHU at a 10% tuition discount.
Horn emphasizes his belief that state regulations and slow-moving bureaucracy have hobbled PASSHE’s ability to innovate in online education, leading it to the necessity of outsourcing. He does so mainly by analogy to SNHU. There are several problems with this approach.
First, as I pointed out earlier, the idea of increasing autonomy to spur innovation was not newly introduced by disruption theory. Bell Labs embodied this strategy from its formation in 1925. Clayton Christensen wasn’t even born until 1952. Second, as I also pointed out earlier, the success of this strategy does not validate all of disruption theory, never mind its applicability to education. As Horn acknowledges in his post, PASSHE is not failing because its leaders did everything right in pursuing its most lucrative customers. (Not that we would want public education systems to do that anyway, which is yet another problem with disruption theory’s applicability.) To the contrary, PASSHE arranged for the articulation agreement because many of its (tuition-paying) students are not finishing their degrees, according to PASSHE Chancellor Dan Greenstein. Presenting those students with another reasonably priced online option was intended to increase student graduation rates, even at the possible cost of the system’s most lucrative customers. We have to follow Horn down a chain of several inferential steps—each of which is given with either thin factual support or none at all—to arrive at the implication that “disruptive innovation is stealing Pennsylvania’s students.”
Horn relies on a loose comparison to SNHU to do most of the work from him. To give him the benefit of the doubt, let’s stipulate several points:
SNHU’s growth has been undeniably impressive at a time when PASSHE has been experiencing a slow-motion crisis.
SNHU President Paul LeBlanc, like me, is a fan of Clay Christensen’s work. He has thought about it a lot and credits it with influencing his thinking.
Even so, it’s not that simple. LeBlanc hasn’t applied the separation principle dogmatically. SNHU operates under one shared governance structure. Autonomy is not the same as freedom from rules or oversight. Second, he has not hesitated to revoke autonomy when it wasn’t working. SNHU’s competency-based education unit, the College for America, used to be autonomous but has since been pulled back into the larger online learning unit.
Overall, Horn’s article is a confusing muddle. He readily acknowledges that the causes of PASSHE’s troubles look nothing like the causes of disruptive innovation described in The Innovator’s Dilemma. He also admits that better funding of the chronically underfunded system would help. He then swerves off into an explanation of disruption theory, connected only by the thread that SNHU employed the same basic strategy that Bell Labs used successfully nearly 30 years before Clayton Christensen was born. He ends with a call for state systems to create small incubators where new ideas can be piloted. I agree with that idea, but I don’t think it bears any relationship to the completely unjustified clickbait title of his post. I don’t think one has to believe in disruption theory to come to the conclusion that university systems have become sclerotic and need to find ways to get out of their own way. This could have been a perfectly good article if it weren’t trying so desperately hard to validate disruption theory. Horn’s rather modest call to action is not in any way proportionate to the dire pronouncement of the headline; nor does it justify Price’s call for Calbright to be exempted from oversight or collective bargaining rules before it has even proven that it can serve 1,000 students.
This is all hauntingly familiar. I mentioned that I own three of Christensen’s books, two of which I occasionally dip into to this day. The other one is Disrupting Class, which is the book that Christensen co-wrote with Horn about the application of disruption theory to education. In stark contrast to The Innovator’s Dilemma and The Innovator’s Solution, it was a disappointing mess. Parts of it attempt to stretch disruption theory beyond recognition in order to fit educational examples. Other parts contain thoughtful, reasonable observations that bear no credible connection to disruption theory. It doesn’t hang together as a cohesive work. I found the book unpersuasive when it was published 12 years ago and do not believe it has aged well. And in subsequent publications by disruption theory proponents in education, the flaws that were apparent in that original work have been papered over rather than addressed. Christensen has passed away, but the people carrying his torch in this domain have not done his legacy a service, however diligently and earnestly they may have tried.
Twelve years of trying is enough. It is time to think harder.
Grieving our errances
Alexander Pope wrote that to err is human, but he might have added that to err repeatedly due to habits of mind is particularly human. The disruption theory proponents love their idea. As a man who has written 7,000-word blog posts, I cannot credibly claim immunity from that particular siren song.
It is not a propensity to be ridiculed. Our capacity to fall in love with a thing as aetherial as a concept has been a key to our survival as a species, and to creating lives for ourselves that are about more than just surviving.
Sadly, we have not evolved to feel equal love for the process by which we produce, examine, and sometimes destroy ideas. We became toolmakers because tools enable us to accomplish more work with less effort. Ideas are tools that enable us to accomplish more work with less effortful thought. To err is human; to think is divine. These days, I am less inclined toward mocking people for being passionately wrong than when I was young enough to believe that only other people ever commit that sin. But I will attack their arguments without hesitation or mercy when I think they are harmful.
It has become clear to me that all talk of disruptive innovation in the context of education is harmful. Not just when applied thoughtlessly by people who don’t understand the theory, but even when applied diligently by people who do. It is harmful to advocate for a failed idea as if the fate of education depends on believing it is true.
Disruption theory in education is dead. Be sad if you need to, but move on. Seize the wheel and learn to be a navigator again.
As for Calbright, I remain agnostic. I believe it is a good idea in principle for California to create a safe space to experiment with alternative means of reaching the millions of Californians who need education but aren’t getting it. I worry that this characterization is not fully aligned with the stated goals of the Calbright program. I do not know whether the experimental college is being run well, or whether now is the best time to prioritize it over other educational exigencies of the moment. I do believe it is not only fair but important to ask these questions and insist on arriving at fact-based, well-reasoned answers.
According to an article in Inside Higher Ed, California just modified its $475 million Student Equity and Achievement Program “to allow the funds to be used for emergency student aid.” Since these changes don’t entail new funding, much of the article was dedicated to hand-wringing about whether diverting existing funds from other priorities “like tutoring, peer-mentoring programs and equity-focused professional development for faculty” is, on balance, a good idea.
On the one hand, there is evidence that giving students emergency financial support is both a needed and an effective intervention:
Colleges in California and across the nation have created their own emergency aid programs. A Senate analysis of the bill notes that Pasadena City College and Grossmont College both fund their programs through external sources like foundations and fundraising.
Amelia Parnell, vice president for research and policy at the National Association of Student Personnel Administrators, said the association found in a survey that most colleges feel they aren’t fully meeting students’ emergency financial needs.
“Because emergencies are typically unexpected, it’s hard to find the right balance that’s needed,” she said, adding that she thinks the spirit of the bill is “consistent with what a lot of campuses have said.”
According to the Senate floor analyses, Chiu cites as support a February 2017 report from the Institute for College Access and Success on college costs for low-income California students. The report found that low-income students at public colleges in California can’t afford college costs with the available grants, their own resources and some working income.
It also found that community colleges sometimes have a greater net price for low-income students than four-year public schools due to the limited amount of grants available for community college students.
Chiu argued that research shows emergency aid can keep students enrolled through unforeseen challenges.
Research does show that emergency aid can keep students enrolled. See, for example, Georgia State University’s Panther Retention Grants.
But on the other hand, the other interventions that the $475 million California program has been funding up until now are important too.
However, the Senate grappled with questions of whether the Student Equity and Achievement Program funds would be best used for this purpose. The analysis asks if the bill would “set a precedent that dilutes student equity funds intended for critical academic support service,” and if expanding state financial aid programs would be more appropriate.
The Senate Appropriations Committee said the bill could redirect funds away from other student support services, which could lead to “potentially significant … cost pressure” to maintain the state’s current level of student support services.
What the article doesn’t mention is whether the legislature funded any significant research, either previously or going forward, that will help guide the colleges regarding which investments are likely to be most effective in meeting their equity goals. Because that’s the question, right? Colleges have options to spend their money to best serve their students. And given the total amount of money in play across the system—nearly half a billion dollars—one would think that a small amount of money invested in research would be a wise allocation of funding.
Maybe it’s in there and just not mentioned in the IHE article. I hope so. Past experience with the California system suggests that (a) the legislature doesn’t think this way and (b) the California Community College System is not set up well to execute programmatic research of this kind even when they are given the funding and prioritization to do so—in part because they are not given the funding and prioritization to do so as often as they should be.
If there is policy uncertainty about a consequential matter that impacts students in a meaningful way, then that risk should be approached with an experimental mindset. If you aren’t mindful about assessing the impact of different choices, then you’re just throwing dice.
EdSurge has a good piece up about the U.S. Public filing submitted by the Scholarly Publishing and Academic Resources Coalition (SPARC) with the U.S. Department of Justice opposing the merger between Cengage and McGraw-Hill. In addition to the expected fare about pricing and reduced competition, there is a surprisingly fulsome argument about the dangers of the merger creating an “enormous data empire.”
Given that the topic at hand is an anti-trust challenge with the DoJ, I’m going to raise my conflict of interest statement from its normal place in a footnote to the main text: I do consulting work for McGraw-Hill Education and have consulting and sponsorship relationships with several other vendors in the curricular materials industry. For the same reason, I am recusing myself from providing an analysis of the merits of SPARC’s brief.
Instead, I want to use the data section of their brief as a springboard for a larger conversation. We don’t often get a document that enumerates such a broad list of potential concerns about student data use by educational vendors. SPARC has a specific legal burden that they’re concerned with. I’ll briefly explain it, but then I’m going to set it aside. Again, my goal is not to litigate the merits of the brief on its own terms but rather explore the issues it calls out without being limited by the antitrust arguments that SPARC needs to make in order to achieve their goals.
Let’s break it down.
When is bigger worse?
While I’m sure that PIRG’s concerns about the data are genuine, keep in mind that they have been fighting a long-running battle against textbook prices, and that the primary framing of their brief is about the future price of curricular materials. Their goal is to prevent the merger from going through because they believe it will be bad for future prices. Every other argument that they introduce to the brief, including the data arguments, they are introducing at least in part because they believe it will add to their overall case that the merger will cause, in legal parlance, “irreparable harm.” As such, that has to be the standard for them. It’s not whether we should be worried about misuse of data in general, but about whether this merger of the data pools of two companies makes the situation instantly worse in a way that can’t be undone. That’s pretty high bar. Each of their data arguments needs to be considered in light of that standard.
But if you’re more concerned with the issues of collecting increasingly large pools of student data in general, and if you can consider solutions other than “stop the merger,” then there is a more nuanced conversation to be had. I’m more interested in provoking that conversation.
What can be inferred from the data
One question that we’re going to keep coming back to throughout the post is just how much can be gleaned from the data that the publishers have. This is a tough question to answer for a number of reasons. First, we don’t know exactly everything that all the publishers are gathering today. SPARC’s doesn’t provide us with much help here; they don’t appear to have any inside information, or even to have spent much time gathering publicly available information on this particular topic. I have a pretty good idea of what publishers are collecting in most of their products today, but I certainly don’t have a comprehensive knowledge. And it’s a moving target. New features are being added all the time. I can speak a lot more confidently about what is being gathered today than on what may be gathered a year from now. The further out in time you go, the less sure you can be. Finally, while publishers—like the rest of us—have thus far proven to be relatively bad at extrapolating useful holistic knowledge about students from the data that publishers tend to have, that may not always prove to be the case. So with those generalities in mind, let’s look at SPARC’s first claim:
Like most modern digital resources, digital courseware can collect vast amounts of data without students even knowing it: where they log in, how fast they read, what time they study, what questions they get right, what sections they highlight, or how attentive they are. This information could be used to infer more sensitive information, like who their study partners or friends are, what their favorite coffee shop is, what time of day they commute from home to school, or what their likely route is.
How much of that “more sensitive information” that SPARC claims can be inferred really logical to fear right now? Most of the scary stuff they speculate about here is location-related. Unless the application page specifically asks the student’s permission to use geolocation and the student grants it—I’m sure you’ve had web pages ask your permission to know your location before—then the best it can do is know the student’s IP address, which is a pretty crude location method. None of the place-based information is really accessible via any data that is collected through any courseware that I’m aware of today. The only exception I know of is attendance-taking software. How much of an additional privacy risk it is to know the attendance habits of students who are already known to have registered for a class in virtue of the fact that they are taking and using the curricular materials associated with the class is an open question.
The other risk SPARC references specifically is knowledge of social connections. There are products that do facilitate the finding of study partners. Actually, the LMS market, which is roughly as concentrated as the curricular materials market, may have much more exposure to this particular concern.
While I certainly wouldn’t want these data to be leaked by the stewards of student learning information, I suspect there is much better quality data of this sort that is more easily obtainable from other sources. Even in the worst case, if they got misappropriated and merged with consumer data sets, the incremental value of this information relative to what someone with ill intent could learn from the average person’s social media activity strikes me as pretty limited.
Of course, the information value is a separate question from the responsibility of care. Students are responsible for the information that they post on their social media accounts. Educators and educational institutions have a responsibility of care for data in products that they require students to use. That said, we should think about both the responsibility of care and the sensitivity of particular data. Generally speaking, I don’t see the kind of location and and personal association data that publisher applications are likely to have as particularly sensitive.
Anyway, continuing with SPARC’s brief:
“We now have real time data, about the content, usage, assessment data, and how different people understand different concepts,” said Cengage CEO Michael E. Hansen in an interview with Publishers Weekly.135 McGraw-Hill claims that its SmartBook program collects 12 billion data points on students. Pearson now allows students to access its Revel digital learning environment through Amazon’s Alexa devices—which have been criticized for gathering data by “listening in” on consumers.
Once gathered, these millions of data points can be fed into proprietary algorithms that can classify a student’s learning style, assess whether they grasp core concepts, decide whether a student qualifies for extra help, or identify if a student is at risk of dropping out. Linked with other datasets, this information might be used to predict who is most likely to graduate, what their future earnings might be, how a student identifies their race or sexual orientation, who might be at risk of self-harm or substance abuse, or what their political or religious affiliation might be. While these types of processes can be used for positive ends, our society has learned that something as seemingly innocent as an online personality test can evolve into something as far-reaching as the Cambridge Analytica scandal. The possibilities for how educational data could be used and misused are endless.
I realize that this is a rhetorical flourish in a document designed to persuade, but no, the possibilities really aren’t endless. If you can’t train a robot tutor in the sky by having it watch you solve more geometry problems, then you can’t bring Skynet to sentience that way either. I don’t want to minimize real dangers. Quite the opposite. I want to make sure we aren’t distracted by imaginary dangers so that we can focus on the real ones.
I’m particularly concerned by the Cambridge Analytica sentence. “Something as seemingly innocent as an online personality test can evolve into something as far-reaching…”. The implication seems to be that Cambridge Analytica inferred enormous amounts of information from an online personality test. But that’s not what happened. The real scandal was that Cambridge Analytica used the personality test to get users to grant them permission to enormous amounts of other data in their profile. The kind of deeply personal data that people put in Facebook but don’t tend to put in their online geometry courseware. I don’t see how that applies here.
Of course, the data that these companies collect in the future may change, as may our ability to infer more sensitive insights from it. Writ large, we don’t have to make the kind of cut-and-dry, snapshot-in-time decision that a legal brief necessarily advocates. Rather than making a binary choice between either blithely assuming that all current and future uses of student educational data in corporate hands will be fine or assuming the dystopian opposite and denying students access to technology that even SPARC acknowledges could benefit them, the sector should be making a sustained and coordinated investment in student data ethics research. As new potential applications come online and new kinds of data are gathered, we should be pro-actively researching the implications rather than waiting until a disaster happens and hoping we can up the mess afterward.
Data permission creep
SPARC next goes on to argue that since (a) students are a captive audience and essentially have no choice but to surrender their rights if they want to get their grades, (b) professors, who would be the ones in a position to protect students’ rights, don’t have a good track record of protecting them from textbook prices, and (c) nobody has a good track record of reading EULAs before clicking away their rights, there is a good chance that, even if the data rights students give agree to give away are reasonable today, there is a high likelihood that they will creep into unreasonableness in the future:
Students are not only a “captive market” in terms of the cost of textbooks, they are a captive market in terms of their data. The same anticompetitive behavior that arose in the relevant market for course materials is bound to repeat itself in the relevant market for student data.
As the market shifts toward inclusive access fees and all-access subscriptions, students increasingly will be required to use digital course materials as a condition of enrolling in a course. Even if a student is not automatically subscribed, they may be enrolled in a course using digital homework, where a portion of a student’s grade depends on purchasing an access code, accepting the terms of use, and potentially surrendering data in the process of completing assignments. This is a new dimension of the principal-agent problem. In the same way that it is a foregone conclusion that students will need to purchase assigned materials regardless of the price, it is also a foregone conclusion that they will need to accept the terms of use.
The graph of textbook prices since 1980 in Section 1.1 illustrates what can happen when publishers engage in coordinated pricing practices in a market where consumers have little power, as we discussed in Section 4.1. The same problem could repeat itself in terms of the ever expanding permissions granted under terms of use. Just as professors are sometimes unaware when the price of a textbook goes up, they may not be aware when the terms of use change in a way that may be unacceptable to their students.
Therefore, there is potential for publishers to inflate the permissions they require students to grant in exchange for using a digital textbooks in the same way that they have inflated prices through coordinated behavior. Students will not only be paying in dollars and cents, but also in terms of their data.
I find the permissions creep argument to be compelling for several reasons. First, the question of whether people should have a right to control how their data are used is separable from the question of known harm that abuse of those data could cause. Students should have right to say how their data can be used and shared, regardless of whether that use is deemed harmful by some third party.
Second, there is an argument that SPARC missed here related to human subjects research. Currently, universities are required by law to get any experimentation with human subjects, including educational technology experiments, approved by an IRB. This includes, but is not limited to, a review of informed consent practices. Companies have no such IRB review requirement under current law. Companies with more data, more platforms, and bigger research departments can conduct more unsupervised research on students. For what it’s worth, my experience is that companies that do conduct research often try to do the right thing. But that should be small comfort, for a number of reasons.
First, there is no generally agreed upon definition of what “the right thing” is, and it turns out to be very complicated. When is an activity research “on” students, and when is it “on” the software? If, for example, you move a button to test whether doing so makes a feature easier to find, but awareness of that feature turns out to make a difference in student performance, then would the company need IRB approval? If the answer “yes,” and “IRB approval” for companies looks anything remotely like what it does inside universities today, then forget about getting updated software of any significance any time soon. But if the answer is “no,” then where is the line, and who decides? There is basically no shared definition of ethical research for ed tech companies and no way to evaluate company practices. This is not only bad for the universities and students but also for the companies. How can they do the right thing if there is no generally accepted definition of what the right thing is?
Second, if IRB approval specifically means getting the approval of one or more university-run IRBs, and particularly if it means getting the approval of the IRB of every university for every student whose data will be examined, universities have not yet made that remotely possible to accomplish. Nor could they handle the volume. I believe that we do need companies to be conducting properly designed research into improving educational outcomes, as long as there is appropriate review of the ethical design of their studies. Right now, there is no way of guaranteeing both of these things. That is not the fault of the companies; it’s a flaw in the system.
Fixing the student privacy permission problem would be hard to do in a holistic way. Some further legislation could potentially help, but I’m not at all confident that we know what that legislation should require at this point. I’ve written before about how federated learning analytics technical standards like IMS Caliper could theoretically enable a technical solution by enabling students to grant or deny permission to different systems that want access to their data, similarly to the way in which we grant or deny access to apps that want access to data on our phones. But that would be a long and difficult road. This is a tough nut to crack.
The research problem is also tough, but not quite as tough as the privacy permission problem. I’ve been speaking to some of my clients about it in an advisory capacity and working on it through the Empirical Educator Project. It is primarily a matter of political will at this point, and the pressure to solve this problem is rising on all sides.
More data means more privacy risk
For our purposes, I won’t quote the entirety of SPARC’s argument on this topic, but here’s the nub of it:
It is common sense that the more data a company controls, the greater the risk of a breach. Recent experience demonstrates that no company can claim to be immune to the risk of data breaches, even those who can afford the most updated security measures. The size or wealth of a company has proven no obstacle to potential hackers, and in fact larger companies may become more tempting targets. Allowing more student data to become concentrated under a single company’s control increases the risk of a large scale privacy violation.
As a case in point, Pearson recently made the news for a major data breach. According to reports, the breach affected hundreds of thousands of U.S. students across more than 13,000 school and university accounts. Pearson reports that no social security numbers or financial information was compromised, but this is not the only kind of data that can cause damage. Compromising data on educational performance and personal characteristics can potentially affect students for the rest of their lives if it finds its way to employers, credit agencies, or data brokers.
While state and federal laws provide some measure of privacy protection for student records, including limiting the disclosure of personally identifiable information, they do not go far enough to prevent the increased risk of commercial exploitation of student data or protect it from potential breaches.
While we should be very concerned about student data privacy, I don’t think the number of data points an education company has about a student is a good measure of the threat level. Again, a merged Cengage/McGraw-Hill would not have the same kind of data that Facebook would. We have to think very specifically about these data because they are quite different from data on the consumer web. The number of hints a student asked for in a psychology exercise or the number of algebra problems a student solved do not strike me as data that are particularly prone to abuse. These sorts of information bits comprise the bulk of the data that such companies have in their databases today. There may very well be extremely serious data privacy issues lurking here, but they will not be well measured by the volume of data collected (in contrast with, say, Google).
The point about the gaps in the laws is a much more serious one. Everybody has known for years, for example, that FERPA is badly inadequate. It is only getting worse as it ages. The Fordham paper cited by SPARC has some good suggestions. Now, if only we had a functioning Congress….
Algorithms
Again, I’ll excerpt the SPARC filing for our purposes:
Algorithms are embedded in some digital courseware as well, including the “adaptive learning” products of the merging companies and some of their competitors. These algorithms can be as simple as grading a quiz, or as complex as changing content based its assessment of a student’s personal learning style….
While algorithms can produce positive outcomes for some students, they also carry extreme risks, as it has become increasingly clear that algorithms are not infallible. A recent program held at the Berkman Klein Center for Internet and Society at Harvard University concluded categorically that “it is impossible to create unbiased AI systems at large scale to fit all people.” Furthermore, proprietary algorithms are frequently black boxes, where it is impossible for consumers to learn what data is being interpreted and how the calculations are made—making it difficult to determine how well it is working, and whether it might have made mistakes that could end in substantial legal or reputational consequences.
Let’s disambiguate a little here. There are two senses in which an algorithm could be considered a “black box.” Colloquially, educators might refer to an adaptive learning or learning analytics algorithm that way if they, the educators using it, have no way of understanding how the product is making the recommendations. If an algorithm is proprietary, for example, the vendor might know why the algorithm reaches a certain result, but the educator—and student—do not.
Within the machine learning community, “black box” means something more specific. It means that the results are not explainable by any humans, including the ones who wrote the algorithm. In certain domains, there is a known trade-off between predictive accuracy and the the human interpretability of how the algorithm arrived at the prediction.
Both kinds of black boxes are very serious problems for education. In my opinion, there should be no tolerance for predictive or analytic algorithms in educational software unless they are published, peer reviewed, and preferably have replicated results by third parties. Educators and qualified researchers should know how these products work, and I do not believe that this an area where the potential benefits of commercial innovation outweigh the potential harm. Companies should not compete on secret and potentially incorrect insights about how students learn and succeed. That knowledge should be considered a public good. Education companies that truly believe in their mission statements can find other grounds for competitive advantage. This is another area that EEP is doing some early work on, though I don’t have anything to announce on it just yet.
The second kind of black box—algorithms that are published and proven to work but are not explainable by humans—should be called out as such and limited to very specific kinds of low-stakes use like recommending better supplemental content from openly available resources on the internet. We should develop a set of standards for identifying applications in which we’re confident that not understanding how the algorithm arrives at its recommendation does not introduce a substantial ethical risk and does produce substantial educational benefit. If the affirmative case can’t be made, then the algorithm shouldn’t be used.
Data monopolies
I’m going to be a little careful with this one because, again, I am recusing myself from commenting on the merits of the brief, and this particular data topic is hardest to address while skirting the question before the DoJ. But I do want to make some light comments on the broader question of when combining different educational data sets is most potent and therefore most vulnerable to abuse.
From SPARC:
One lesson learned from the rise of technology giants like Facebook is that preventing platform monopoly from forming is far simpler than breaking one up. Given the vast quantity of data that the combined firm would be in a position to capture and monetize, there is a real potential for it to become the next platform monopoly, which would be catastrophic for student privacy, competition, and choice.
For decades, the college course material market has been split between three giants. There is a large difference between a market split three ways and a market split two ways. As these companies aggressively push toward digital offerings and data analytics services, a divided market will limit the size and comprehensiveness of the datasets they are able to amass, and therefore the risk they pose to students and the market. So long as publishers are competing to sell the best products to institutions, and there is significantly less risk of too much student data ending up in one company’s hands.
I won’t characterize the danger of combining publisher data sets beyond what I’ve already covered in this post. What I want to say here is that the bigger opportunity for potential insights, and therefore the bigger area of concern for potential abuse, may be when combining data sets from different kinds of learning platforms. I haven’t yet seen evidence that combining data across courseware subjects yields big gains in understanding regarding individual students. But when you combine data from courseware, the LMS, clickers, the SIS, and the CRM? That combination of data has great potential for both benefit and harm to students because it provides a much richer contextual picture of the student.
Irreparable harm
While nothing in this post is intended to comment directly on the matter before the DoJ, the phrase that frames the anti-trust argument—”irreparable harm”—is one that we should think about in the larger context. I believe we have an affirmative obligation to students to develop and employ data-enabled technologies that can help them succeed, but I also believe we have an affirmative obligation to proceed in a way that prioritizes the avoidance of doing damage that can’t be undone. “First, do no harm.” We should be putting much more effort into thinking through ethics, designing policies, and fostering market incentives now. I don’t see it happening yet, and it’s not even entirely clear to me where such efforts would live.
To paraphrase the intro paragraph from January’s post on the George Mason University report, another year month and another deeply flawed report about online education in US higher education, this time by Di Xu (assistant professor of educational policy and social context at the University of California
Irvine and a visiting fellow at AEI) and Ying Xu (Ph.D. candidate at the School of Education at the University of California Irvine). The report is titled “The promises and limits of online higher education: Understanding how distance education affects access, cost, and quality”.
While the supply and demand for online higher education is rapidly expanding, questions remain regarding its potential impact on increasing access, reducing costs, and improving student outcomes. Does online education enhance access to higher education among students who would not otherwise enroll in college? Can online courses create savings for students by reducing funding constraints on postsecondary institutions? Will technological innovations improve the quality of online education?
This report finds that, to varying degrees, online education can benefit some student populations. However, important caveats and trade-offs remain.
In many ways this report takes a similar approach to the GMU report and a prior one by Caroline Hoxby from Stanford University, which was subsequently withdrawn, in asking important questions but providing flawed analysis to support conclusions. The problems with the American Enterprise Institute (AEI) report lie in its description of the history of online education and the 50 percent rule, the usage of data to describe the “supply side” of online, and some misinterpretations of IPEDS data. The flaws are hard to overlook, which is a shame, in that much of the qualitative discussion on online education provides a nuanced set of answers to the questions posed above.
The Good
The AEI report takes a look at a little-used portion of the IPEDS data set – The Completions survey and its program-level data on whether an institution offers certain programs at all and whether they are offered as a fully online (distance education) offering. This data has its flaws, which we’ll get to below, but it was quite interesting to get a summary view at the program level.
After a relatively solid discussion of research findings on Online Education and Student Outcomes, which summarizes positive and negative results along with the context and limitations of the relevant research, the report presents its discussion of known strategies to improve online education. This is a welcome relief, as many studies view online as a conclusion to be made about online vs. face-to-face, while this one summarizes known methods to continue improvements of a necessary modality.
Based on the growing knowledge regarding the specific challenges of online learning and possible course design features that could better support students, several potential strategies have emerged to promote student learning in semester-long online courses. The teaching and learning literature has a much longer list of recommended instructional practices. However, research on improving online learning focuses on practices that are particularly relevant in virtual learning environments. These include strategic course offering, student counseling, interpersonal interaction, warning and monitoring, and the professional development of faculty.
The Bad
The introduction relies heavily on the “50 percent rule” and 1998 and 2006 changes to this rule as key points in the expansion of online education. This regulation did have an effect, but so did a number of other factors not mentioned in the report. To make matters worse, the wording of the rule conflates students and institutions. For example, in an email conversation with Russ Poulin from WCET, he noted how the following is inaccurate:
Similarly, the HEA also denied access to certain types of federal financial aid and loans for students who took more than half their courses through distance courses.
yet this statement is accurate:
…the rule dictated that institutions that offered more than 50 percent of their courses through distance education or enrolled more than half of their students in distance education courses would not be eligible for federal student aid programs.
The regulation applied to institutions and in no way measured this usage at the student level. I find that this article from New America does a much better job describing this regulation’s history and impact.
Update 3/8: Poulin also noted (see comment below):
After talking to you Phil, the oddity of the 50% discussion being front and center hit me even more. The lifting of the 50% rule had an impact on only a small number of institutions. Several for-profits and a small number of non-profit and public universities. The vast growth in distance learning has primarily been in institutions that get nowhere near the 50% mark, so the change in that rule was not a direct influence in their decision to enter the distance education market. To place it front and center seemed odd to me and not a real reflection of the motivations for most college leaders.
The report also confuses institutional vs. student level data in looking at per-state online statistics.
Finally, considering that state-level policies may shape online learning in unique ways, Figure 13 shows online enrollment by state in the 2016–17 school year. Unsurprisingly, the most populated states, such as California, Florida, and Texas, also had the largest number of online course takers. Once accounting for between-state differences in overall higher education enrollment, four states have the largest share of students who enrolled in at least one online course in 2016: Arizona (61 percent), Idaho (52 percent), New Hampshire (58 percent), and West Virginia
57 percent).
This might be nitpicking, but the IPEDS data referenced is for institutions located in each state, not students located in each state. But a report trying to make sense of a complex subject should get this information correct and not add to the confusion.
The Ugly
The worst aspects of the report can be seen in figure 1 and an attempt to summarize changes in the supply side of online education. The authors chose to define the supply side as number of institutions offering at least one online course or one online program, using the aforementioned Completions / program-level data.
I’ll wait while you take the necessary 5 minutes to decipher the worst color-legend usage in a chart that I’ve seen in years . . . Not yet? . . .
When I shared this image on Twitter, Kevin Carey pointed out some results that seems non-sensical.
Wait, so from 2012 to 2016, the percentage of public institutions offering at least one online programs went from 16% to 72%, whereas at for-profits it went from 4% to 33%?
The GMU report and the Stanford / Hoxby report made the more common mistake of essentially conflating online education with the for-profit sector, but this data makes little sense on the surface – implying that the for-profit sector offers relatively few online programs compared to public and private institutions. Looking at our 2016 IPEDS profile, you can see that 4-year for-profits by far have the greatest percentage of students in fully online programs (69%). How does AEI measure for-profits as much lower in offering fully-online programs?
It took a while to figure out, but I think the authors made two mistakes. One is that they combined all for-profits together (2-year and 4-year), which is confusing since 2-year for-profits have the lowest usage of online education and a bunch of really small schools. This combination cuts the for-profit numbers dramatically. Look at the 2012 summary data below, where I show data for each sector and then combining 2-year and 4-year sectors together for public, private, and for-profit.
The second issue is that simply measuring for-profits by institution using Completions program-level data is an unreliable approach to understanding online education supply, particularly for the for-profit sectors. Most for-profit systems own a number of smaller campuses, each with their own IPEDS code, yet the online programs are offered centrally by the system. And the Completions survey DE data has major holes in it. Consider South University (part of EDMC as of the 2012 data shown below):
All 21 online programs are offered through the online campus, with over 12,364 taking exclusively DE courses and 8,898 taking no DE courses. Using the AEI methodology, 13 of the 14 institutions have no online courses or programs – almost no supply of online education in their language.
Also consider DeVry University, which does not list a centralized online campus yet has significant online presence. For whatever reason, they report the student enrollment data per campus, but they did not fill out the Completions program-level data at all. Zero supply of online education in AEI’s approach.
My therapists jumped in at this point and convinced me to not fully duplicate the AEI findings (serenity now!!!). What’s important here is that the basis of AEI’s description of online education supply, using institutional metrics that are dubious and ignore how the for-profit sector works, is flawed and misleading. Technically they used data in IPEDS, but they misunderstood its usage and limitations.
Yes, there are valuable parts of this report. But like the GMU and Stanford reports, the flaws in analysis make it very difficult to separate the good from the bad and the ugly. This type of report from well-funded organizations aimed at policy-makers should inform, not confuse, but yet again we are faced with some serious flaws. We need better.
Carnegie Mellon University and Duke University have shared newly available free tools that will significantly lower the barriers to conducting ethical educational research. The two universities contributed the tools through e-Literate’s Empirical Educator Project (EEP), an effort to promote broader adoption of evidence-based teaching practices and foster a culture of empirical education across higher education.
As with all academic research involving human subjects, educational researchers must have their experimental designs approved by their university’s Institutional Review Board (IRB). If a researcher wants to study students or their work, they must explain how they will get the students’ informed consent to participate.
This can be a major barrier that often prevents research from being undertaken. Teaching faculty who may be interested in conducting a study may decide that the bureaucratic burden is more than they can take on. Multiple universities that want to collaborate on cross-institutional studies will have to get approval from each institution’s IRB in an environment where there are no widely adopted standards for reviewing and approving educational research by these bodies. Educational technology companies that want to be more transparent and collaborative with universities about their own research into product efficacy can find the IRB process impractically time-consuming. As a result, far less educational research gets conducted in ways that are both reviewed for ethical practices and shared as credible research that contributes to the state of the art in learning science.
Through e-Literate’s EEP, learning science researchers at Carnegie Mellon and Duke Universities discovered that each institution had developed a solution for part of this problem. Carnegie Mellon University has developed templates approved by their IRB that they estimate will accommodate approximately 80% of classroom research use cases. Meanwhile, Duke University has developed language and a process approved by their IRB for requesting and tracking informed consent from students.
The two universities have released the tools under a Creative Commons Attribution (CC-BY) license and provided “train the trainer” support for the use of their templates and protocols. Together, these contributions could enable many of the educators and product designers who are already conducting informal educational research all over the world to participate in the same sort of social fabric that has enabled communities of researchers in other human sciences to tackle problems from cancer to Alzheimer’s disease.
Jeff Young has a great piece up about this release at EdSurge, and I believe we will see something from The Chronicle in their teaching newsletter on Thursday. I’d like to give you my own take on the reasons why this an important milestone.
It’s an example of untapped inter-institutional opportunity
Carnegie Mellon and Duke are theoretically peers. To use one very imperfect measure, they are both ranked in the top 25 national universities by US News and World Report. They are both more specifically ranked in the top 15 such schools for undergraduate teaching. Both are seriously concerned with improving undergraduate education through research-based practices. The two institutions have been working on similar yet complementary efforts to make that research easier. You would think that they would have had opportunities to share their work with each other, or at least know about what the other is doing.
Before EEP, they didn’t.
On closer examination, the complementarity of the two efforts suggests the kinds of opportunities that higher education is missing. Carnegie Mellon University has one of the broadest, deepest, most impressive, and most historic learning science research programs on the planet. There are only a handful of universities that are even in their league, and Stanford may be the only university that rivals them in depth and breadth. Further, the university has made a commitment in the form of the Simon Initiative to “[p]rovide accessible tools and methods with which any person or institution can adopt and advance CMU’s approach to learning engineering, improving outcomes for their own learners,” globally, in addition to improving teaching and learning at Carnegie Mellon University itself. (Not that this is distinct from but complementary to their Eberly Center for Teaching Excellence, which is similar in purpose to the centers of teaching and learning at many universities.) To get a flavor for who they are and how they think, here’s a playlist of three e-Literate TV videos that highlight a few of their faculty members:
Now, Carnegie Mellon is known more generally for its engineering prowess, so it’s no surprise that the folks at the Simon Initiative talk about “learning engineering.” (The initiative is named after the late Herb Simon, a cognitive scientist and CMU luminary who coined that term.) That mentality, as well as the orientation of an institution known for its learning science research, shows in their contribution. They studied the IRB applications submitted for educational research by their faculty—not just their learning science faculty, but all faculty interested in publishing their research projects, identified common traits, and developed a template that they estimate covers somewhere in the neighborhood 80% of those projects. They then had their IRB review and approve the template. Now, any educator at CMU who wants to do publishable educational research and can use the template gets their IRB application fast-tracked.
Without taking anything away from Duke’s own learning science research capabilities, one of the things that Duke is among the very best in the world at is making their undergraduates’ educational experience life-changing. As one example, the university has a large number of “professors of practice.” In many big research universities, a tenure track is a kind of death trap. Faculty are worked to the bone for three to five years in the hopes of achieving tenure when, in reality, most of them will be sent packing in the end. And while they are on that treadmill, they have a disincentive to invest too heavily in their teaching lest they neglect the publications and grants that will increase their odds of not being shown the door. That’s not how Duke rolls. Rather than exploiting young faculty by dangling a carrot that will forever be out of reach, they offer many a professor of practice position. Such faculty get basically everything except tenure, including long-term employment, a decent salary scale, and a say in shared governance. But they must not only show that they are excellent educators but also conduct research in education. In effect, rather than “physics” or “art history” being their discipline, it’s “physics education” or “art history education.” I don’t know whether Duke or their faculty would put it quite this way, but I intend it to be a compliment. The point is that they incentivize faculty to become disciplinary experts in supporting their students through evidence-based practices. I have been to one of their teaching and learning conferences and interacted with these faculty members. It was heavily attended and the atmosphere was electric. I have been to many of these kinds of events; yet I have rarely seen faculty who were more actively engaged in asking good, probing questions about teaching practice than I did at Duke.
This different context from CMU’s has resulted in a slightly different approach to the same problem. Like everyone else, Duke’s educators who want to publish their research have to go through IRB approval. And given that Duke has a world-class medical research program, their IRB is both very tough and very focused on privacy concerns. So their Learning Innovation center developed an informed consent tool called WALTer, where WALT stands for “we are learning too.” WALTer sits right inside the LMS for any course in which research is being conducted. Faculty who want to conduct such research are led through a decision tree that produces IRB-approved language for informed consent, based on the conditions of the experiment. Having this form in place helps the instructor get faster approval of the project. And once that approval is in place, WALTer helps ensure that students are given the opportunity to provide their informed consent (or not). Since WALTer has been launched, the Learning Innovation team has seen an increase in the number of faculty expressing interest in conducting educational research and a decrease in the time it takes educational research applications to be approved by Duke’s IRB.
Peanut butter, meet chocolate. CMU’s IRB template and Duke’s informed consent template are completely complementary. But before EEP, the peanut butter was in the fridge and the chocolate was in the drawer. They just didn’t meet.
It’s an example of untapped intra-institutional opportunity
The Duke feedback that they are getting more educational research interest and getting that research approved faster is illustrative of a larger point. The current institutional structures and processes of colleges and universities are not designed to facilitate the development, testing, sharing, and adoption of evidence-backed practices which improve student success. The IRB process is just one (very painful) example. The disincentive for faculty on the tenure treadmill is another. There are many more.
The question that I have heard asked ad nauseam for years and years now, from a wide range of people, is “How can we make faculty care more about teaching?” That is the wrong question on every level. A better question would be, “How can we design universities such that focusing on improving excellence at supporting student success is less painful and more rewarding?”
Think about just about every hot ed tech-related trend you can think of. Retention early warning analytics. Adaptive learning. Competency-based education. Stackable credentials. MOOCs. All this activity (and money) is swirling around the problem of making it easier for students to learn and succeed at school. Now think about all the hot trends that focus on making it practical and rewarding for faculty to focus their energy and considerable intellectual talents on solving the same problem. Can you name one?
An IRB form for educational research may sound like a small and boring thing, but it is a piece of cultural and institutional infrastructure that makes it a little more practical for motivated faculty to focus energy and intellectual talent on learning how to better support student success. “We Are Learning Too” indeed. This is the kind of work that will make college education better and, in the process, make all of those ed tech tools more useful. A laser scalpel doesn’t do much to promote health unless it’s wielded by a physician who knows how, when, and why to use it. ((Is a “laser scalpel” a thing? I may have just made that up. Anyway, you get the point.)) Without that knowledge, it risks doing more harm than good.
It’s an example of untapped multi-institutional and institution/vendor collaboration opportunity
So yay for Carnegie Mellon and Duke. What about everyone else?
Well, for starters, they have both contributed their language under a Creative Commons license. (Duke’s WALTer tool is built using a third-party proprietary platform, so contributing the source code wasn’t an option. But somebody else could easily build a tool that supports the workflow in the release document.) So there’s that.
IRBs are notoriously idiosyncratic. (Some might say arbitrary. I’m not saying that. But some might.) So you could see an IRB at an institution that’s very different from CMU or Duke having something like the following argument:
IRB Member #1: Hey, CMU and Duke are super-rigorous research institutions that are way more focused on these sorts of things than we are. If it’s good enough for them, it should be good enough for us.
IRB Member #2: Actually, exactly because CMU and Duke are super-rigorous research institutions that are way more focused on these sorts of things than we are, what makes you think that what makes sense for them will also make sense for us?
These are both reasonable starting positions. The conversation that should flow from this is an examination of the templates in which an IRB member who wants to change a part of it will need to provide a justification for doing so. This is exactly the next step we want to foster for this project, preferably at multiple institutions. Where we’d like to end up is at a toolkit in which an institution of any type can look at variations—and justifications for those variations—provided by peer institutions so that they adapt and adopt them. We’d also like to supplement what we already have with some more fleshed out student data privacy guidelines that are, once again, education-appropriate. Data privacy in this sort of research is just as important as it is in, say, medical research. But the specific concerns and methods for dealing with them aren’t necessarily identical. We should be developing a sector-wide consensus on ethical practices that can augment what is already in the Duke and CMU IRB contributions.
This would hopefully help to lower one barrier to conducting educational research everywhere. But it could potentially do much more than that. Under the law, if researchers want to conduct a multi-institutional research project, then each institution’s IRB must approve the project. And since no two IRBs use the same standards and most don’t have particular guidelines for educational research, doing research across two institutions is more than twice as hard. Three is more than three times as hard. Doing large-scale, multi-institution research quickly becomes impossible in most scenarios. Unless you’re a vendor, in which case it is trivial, because you are not required to go through IRB for your own research—as long as you don’t publish it in a journal. So vendors can conduct research on students in multiple institutions for their own proprietary purposes easily, but if they want to do the right thing by going through IRB approval and sharing what they’re learning through peer-reviewed journals, it’s way, way harder.
Imagine if it were easier for everyone to do the right thing and submit an application for IRB approval, knowing that they will be going through a streamlined but academically validated process. Imagine the kind of research opportunities that could open up. Now imagine further if there were some technology infrastructure behind this. Imagine if we could track IRB approval and informed consent across institutions, gather appropriately anonymized student data, and share it among researchers in a repository that is designed to respect the student privacy constraints dictated by the approved IRB applications while giving more researchers access to more research-relevant data—including data that are gathered through student interactions inside vendor tools. Technologically, this is quite practical. The hard part is getting the policy infrastructure solid and widely adopted.
This is what EEP is about. The problem isn’t that higher education is failing to innovate or that professors don’t care about teaching well. The problem is that we are flushing 99% of the existing efforts, potential opportunities, and good intentions down the toilet because we don’t have the cultural institutions and social infrastructure to support and sustain them. But that can change.
Kudos and thanks to the good folks at CMU and Duke. We have more folks working in EEP—from a diverse range of institutions—on a wide range of projects. And we’re still learning how to work together effectively. Expect more to come.
Another year and another deeply flawed report about online education in US higher education, this time by Spiros Protopsaltis (associate professor and director of the Center for Education Policy and Evaluation at George Mason University, as well as former aide to Senate Democrats) and Sandy Baum (a fellow at the Urban Institute and professor emerita of economics at Skidmore College, as well as former advisor to Hillary Clinton’s presidential campaign). As Inside Higher Ed described the report, titled “Does Online Education Live Up to Its Promise? A Look at the Evidence and Implications for Federal Policy”:
Online education has not lived up to its potential, according to a new report, which said fully online course work contributes to socioeconomic and racial achievement gaps while failing to be more affordable than traditional courses.
The report aims to make a research-driven case discouraging federal policy makers from pulling back on consumer protections in the name of educational innovation.
In many ways this report takes a similar approach to the report by Caroline Hoxby from Stanford University, which was subsequently withdrawn, in asking important questions but providing flawed analysis to support conclusions. But unlike the previous report, the GMU one documents its sources well with 165 end notes, and for the most part this new report describes the underlying analysis accurately. Where the major problems arise is in conflating online education in general with the for-profit sector and in drawing conclusions that are not supported by the evidence.
The report is not easy to wade through, largely from its wide-ranging discussion of for-profits, online history, past federal policy, a snapshot of research on learning outcomes, and a discussion of current policy debates. Let’s take the primary conclusions and discuss the analysis provided.
“Online education is the fastest-growing segment of higher education and its growth is overrepresented in the for-profit sector.”
The report accurately describes the growth of online education, rising to point where one in three postsecondary students take at least one online course.
There is a disturbing tendency to describe this growth as “explosive” (mentioned five times in report) and an unexplained reliance in many cases on six year old data when new data exists. But the conclusion about growth is accurate.
The phrasing “overrepresented in the for-profit sector” and “concentration in the for-profit sector” in describing online education is very misleading, however. It is true that for-profit schools have a larger percentage of their students studying fully online, but the topic of the report is online education in general, and for-profits represent a rapidly shrinking minority of this case. Never mentioned in the report is the most salient point about for-profits – the sector is in major decline. As documented by IPEDS:
This decline seems relevant, even if you then look at fully-online programs (e-Literate analysis of IPEDS data).
Even in 2012, just two years after the for-profit peak, the for-profit sector accounted for less than 35% of fully online student enrollment, and as of Fall 2017 it was down to 21% with a clear trend. For-profits are rapidly becoming less and less relevant to the topic of online education, with no evidence to back up Protopsaltisclaims that the for-profit sector is about to make a big comeback. It is high time that responsible analysts and scholars cease conflating online ed with for-profit schools, and the authors of this report should know better. If you want to study the for-profit sector, then describe it accurately and don’t extrapolate beyond what the data supports.
“A wide range of audiences and stakeholders—including faculty and academic leaders, employers and the general public—are skeptical about the quality and value of online education, which they view as inferior to face-to-face education.”
I find it strange to put this much emphasis on perceptions from an organization that purports to provide “timely, sound, evidence-based analysis”, but perceptions are somewhat important to understand. The body of the report describes a variety of research sources, but it is inaccurate to summarize that the wide range of stakeholders “view [online education] as inferior to face-to-face education.” Especially if you look at more recent data sources.
Consider the 2018 Inside Higher Ed / Gallup survey of faculty (starting page 32), where they found that faculty with actual experience teaching online have surprising high confidence in the quality potential of online education. For those who have taught online, the percentage that agree or strongly agree that “for-credit online courses can achieve student learning outcomes that are at least equivalent to those of in-person courses in the following context”, 39% for any institution, 52% at my institution, 54% in my department or discipline, and 58% in courses that I teach. Put simply, a majority of faculty who have experience teaching online think results can be at least equivalent to in-person.
Consider the 2018 Northeastern University Survey on the Use and Value of Educational Credentials in Hiring, where they found that “Online credentials are now mainstream, with a solid majority (61%) of HR leaders believing that credentials earned online are of generally equal quality to those completed in-person, up from lower percentages in years past.”
Yes, perception issues are important. But the report’s conclusions are misleading and out of date.
“Students in online education, and in particular underprepared and disadvantaged students, underperform and on average, experience poor outcomes. Gaps in educational attainment across socioeconomic groups are even larger in online than in traditional coursework.”
This topic deserves its own report, and the GMU authors are right to point out that simply comparing online to face-to-face outcomes can obscure the important issue of underprepared and disadvantaged student experiences. On the surface, the conclusion about achievement gaps being “larger in online than in traditional coursework” is also accurate. But the more important question is not whether there is a problem, but rather how to minimize or reverse the achievement gap.
The report references several studies from the California Community College system, mostly from years ago, describing how students “were less likely to complete online courses and when they completed them, less likely to pass them”. Yet the authors did not look at the trends within this system, as easily found in the most recent Distance Education report from the system, where the gap in performance overall for online versus tradition is closing rapidly.
More importantly, the achievement gains applied to all ethnic groups.
It does appear that the performance gaps within online education are not closing by ethnicity despite the broad improvements. That is a real question to consider. Rather than viewing a simplistic view that online = bad results, we should focus on how to maintain current improvements while figuring out how to do even better in providing equal opportunity.
“Online education has failed to improve affordability, frequently costs more, and does not produce a positive return on investment.”
This conclusion is largely based on the NBER Hoxby report that was subsequently withdrawn, and for which I provided a detailed critique. I was not able to get a response from the report author. Beyond a gross mischaracterization of the source data, the Hoxby report made a fundamental flaw in its ROI analysis.
This view of online education – students choosing between non-selective face-to-face institutions or online institutions – takes a zero-sum approach, as if you have the same student population just choosing between institution types. This view ignores the large and growing number of working adults who can only attend college – often in degree-completion programs or masters level programs – because of an online option. Their real choice should be seen as online institution or not at all.
The GMU report relies on the withdrawn Hoxby report and does not even describe that it was withdrawn.
There is an excellent point made that pricing for students has largely not been lower for online education, but there are specific examples (UF Online, SNHU, WGU, to name a few) where they specifically provide much lower-priced offerings to students than comparable face-to-face programs. It would be interesting to study enrollment trends and student outcomes for lower-priced online programs compared to comparably-priced programs.
“Regular and substantive student-instructor interactivity is a key determinant of quality in online education; it leads to improved student satisfaction, learning, and outcomes.”
“Online students desire greater student-instructor interaction and the online education community is also calling for a stronger focus on such interactivity to address a widely recognized shortcoming of current online offerings.”
These last two points get to the primary purpose of the GMU report – current federal policy making efforts that include a re-evaluation of the Regular and Substantive Interaction (RSI) requirement for programs to be classified as online education and no correspondence courses.
The GMU report describes a large body of work documenting the importance of interaction to online student success, and the report accurately describes how “the online education community has also emphasized recently the importance of student-instructor interaction for ensuring quality.” This point is crucial – the vast majority of educators working in online education understand and accept the importance of interaction; there is not significant disagreement on the subject.
What the GMU report gets wrong is conflating actual quality interaction within courses with federal regulations. Much of the basis of the GMU analysis is a series of Office of Inspector General (OIG) reports calling out weak implementation of the RSI regulations. In the biggest case – a report on Western Governors University (WGU) and its competency-based model – this conflation is unwarranted, as I described in a detailed analysis of that action. There were two particular problems with the OIG findings in my view – the first is that the OIG defined their own terms due to the ambiguous nature of the RSI regulation.
The OIG used a binary role-based approach (you are an instructor or you are not) leading to conclusion that only course mentors and evaluators could be considered as instructors, however. The basis of this determination was an instructor must “provide instruction on course content” – clearly a content-dissemination view that rejects alternative pedagogies. And this interpretation that the OIG treats as unambiguous is not based on law, regulations, or commonly-accepted educational terminology. [snip]
This is why I call the audit methodology as hyper-literal. Somehow the OIG thinks they can determine – without any disagreement or ambiguity – the “ordinary meaning of those terms” based on their own interpretations.
The second problem was that the OIG did not evaluate the actual courses or even address the issue of course quality.
Also note that the determination was entirely based on course design materials – think syllabus and course outlines. The OIG did not look at interactions arising during the course of actual course work, just whether there were pre-defined webinars, meetings, and student-instructor interactions. [snip]
These views essentially reject not just WGU’s approach to CBE but also the broader movement of faculty from “sage on the stage to guide on the side”. Instructors, from the OIG view, must provide instruction on course content and interactions must be pre-planned in the course design materials, at least for online courses.
The OIG did not look at student outcomes, applied its own hyper-literal translation of an ambiguous regulation, and did not look at the course interactions themselves – just whether pre-planned course materials described future course interactions. Note, however, that despite the weakness of the OIG report, this does not mean thatWGU is off the hook. Likewise, this report’s over-reliance on the OIG reports mistakes regulation for actual interaction quality, but that does not mean that there is not an issue where many or most online courses could improve faculty-student interaction.
It is broadly understood that the RSI regulation is important but flawed. I agree with the GMU report that a simple elimination of the regulation would be a mistake. But it is overly simplistic and completely subjective for the GMU report to conclude that “unbundled faculty models that have difficulty complying should make changes to match the law instead of changing the law to match the needs of such models.” That is a policy position and not based on “timely, sound, evidence-based analysis”.
In Conclusion
This last point gets to the danger of this GMU report. It is a subjective set of policy recommendations disguised as extensively-documented evidence-based research. There is value in the questions asked, in much of the research documented in the footnotes, and in the clear policy position presented on regular and substantive interaction. But there is more harm than good from the report due to the mischaracterizations, selective data usage, and flawed analysis provided. Read it as a policy paper and not a research report.
Paul Fain from Inside Higher Ed provided a valuable, pithy summary at the end of his article on the report.
The report’s co-authors and its critics agreed that further research is needed on the rapidly evolving field of online education, particularly as more high-quality colleges and universities ramp up their online offerings.
