product-market fit

In his last post, Curtiss raised two crucial points that I suspect will become thematic for e-Literate:

In each of a number of different product categories (like LMSs and digital curricular materials), vendors have collectively spent hundreds of millions or even a billion dollars trying to develop software solutions.
And yet, for a variety of reasons, the student experience in these products remains fragmented and…well…bad.

A third point he implies in his title but does not flesh out in detail is that many of these investments by EdTech vendors do not pay off well (or at all). So the companies are spending hundreds of millions of dollars developing products that often do not serve their customers particularly well and do not result in highly sustainable and profitable companies. It’s lose-lose.

Why is that?

This post is going to explore that question in detail, using digital curricular materials as a case study. If we want EdTech products that actually do a good job of serving student and educator needs, and we want EdTech companies that are reliably profitable enough that they are less likely to do desperate and stupid things that hurt their customers or go out of business entirely, then we must foster the conditions under which EdTech companies can understand real customer needs well enough to meet them while still being able to pay their employees and owners.

Product-market fit is the key

“Product-market fit” is the deceptively simple idea that a product is well-suited to serve the needs of an adequately large and well-defined group of customers. By “adequately large and well-defined,” I mean that (a) there are enough customers who need or want the product that the company can be sustainable and profitable and (b) that the set of people who want the product are clearly defined enough that that the company can figure out who they are and how to reach them.

Think about some product you have purchased that you truly love. Why do you love it? What need does it fulfill for you? Why is it better than alternatives that you may have considered?

For example, my wife and I recently purchased a pair of Oru Kayaks—origami kayaks that fold up into 20-pound suitcases with carrying handles:

My wife and I both love being out on the water and live five minutes from a lake. We also both have bad backs. So getting a two-person canoe or a pair of kayaks on and off the roof of our car is no longer a trivial operation for us, especially if the boats are heavy. And in the current COVID environment, we don’t feel safe going swimming at the public pool. So getting exercise—particularly upper body exercise that won’t strain our backs—is harder at the moment.

Our new kayaks can fit easily in the back of a hatchback when folded up:

Each folded up 20-pound kayak has a handle, so they’re easy to get in and out of the car as well as to and from the water. We don’t have to lift them over our heads, although they are light enough that we could.

They are meticulously designed to be easy to fold and unfold. Everything—the seat back, seat cushion, footrest, and so on—has a clearly marked place to be stored when the kayak is folded up. The kayak itself is very clearly marked with the folds that must be moved in order to open or close them and the order and orientation in which everything must be put together or put away. They are at least as quick and easy to set up and take down as inflatable kayaks. They also steer easily and look cool. And, surprisingly, they are very comfortable to sit in.

My wife and I love taking them out on the lake:

Our grandkids love them too, and it gives them something safely socially distanced to do when they visit:

Grandkids taking off

This product is perfectly designed for older people who want to remain physically active and can afford to spend a little money on a well-designed product. In other words, it’s perfect for us people like us, of which there are plenty. It’s fairly easy to identify people who would want to buy this product as well as the search terms that we would use when looking for something like it, even if the searcher doesn’t know that there is any such thing as an origami kayak. And then there are the families who need five or six kayaks and value the compact storage factor. I suspect that Oru Kayak does a healthy business.

Let’s put this in perspective. There’s a larger market for people who want to kayak. It’s big enough that it supports a number of “segments.” For example, there is the market segment of people who use their kayaks on whitewater rivers and another segment of people who kayak on open oceans. We definitely do not fall into either of those, which a kayak maker or seller could easily tell by our search terms (or just by looking at me). Each segment is big enough to support a number of companies that innovate and differentiate. Kayak companies that understand what the people in a particular market segment really need and want—for example, a really light boat that can fit in the back of the car, is usable on lakes and calm rivers, is easy to use and maintain, is comfortable to sit in, and looks cool—can design products that will sell well, making the customer happy and the company profitable. Win-win.

That is product-market fit. It’s like a key in a lock, where the customer’s unmet need is the lock and a well designed product is the key that opens it.

Product-market fit in EdTech is really hard

Now let’s see if we can apply the same principle to the old-fashioned paper textbook. On the surface, it seems like it should be easy. There’s a large market. It’s easy to identify the educators who would prescribe a textbook and the students who would buy one. It’s easy to differentiate between people who are in the market for a macro-economics textbook from people who are looking for an organic chemistry textbook. People who want a chemistry book are going to want chemistry content and chemistry homework, while people who want a macro-economics textbook will want different sorts of content and assessments.

So…easy, right?

Wrong.

The manufacturer of a 20-pound origami kayak knows quite a bit about what their customers will and won’t do with it. The prospective buyers won’t take it out in a whitewater situation or attach a motor to it. They might take it out on the ocean, but if so, they will be looking for (and be willing to pay for) a kayak that is shaped and constructed for that kind of use.

The same cannot be said of paper textbooks. How will instructors use the books? Will they assign the readings? Skip the readings but assign the end-of-chapter homework? Do both? Use 100% of the chapters? Eighty percent? Sixty percent? If it’s sixty percent (for example), then which sixty percent will they use, and why?

There are some patterns in the answers to these questions—for example, math professors are probably more likely to assign end-of-chapter questions than history professors—but the variability even within subjects is massive. A variety of reasons can dramatically affect how a book is used. Here is a non-exhaustive list:

Student populations at different colleges may need different kinds of explanations of concepts and/or different practice problems.
Even within a college, different subpopulations may need different approaches, e.g., physics for non-majors or statistics for psychology majors.
The design of the course may need to fit into the overall slant of the program or department, which is trying to appeal to certain students because of its particular strengths.
The individual instructor may be an expert on a particular topic and therefore may bring a particular slant to the course.
The individual instructor may not be particularly proficient on a particular topic and therefore may choose to either lean heavily on a particular chapter in the book or skip the chapter altogether.
The individual instructor may (rightly or wrongly) believe that her particular approach to teaching the content is effective and requires a certain kind of use (or non-use) of the chosen textbook.

In response to this substantial and unpredictable variability, textbook publishers adopted two strategies. First, they published a few different titles for each subject, written by different authors with different pedagogical approaches and different features. At best, this could be seen as analogous to Oru Kayak building one origami kayak for sea kayaking and another for lakes. But textbook publishers deal with many more usage variations than kayak manufacturers and they often did a poor job of tracking which variations sold enough copies to justify their continued manufacture. It would be like Oru Kayak building 20 different kayaks—half of which were designed for one particular customer that one Oru Kayak salesperson thought was important—and tried to sell all 20 with little idea of who they might appeal to or how many of those buyers there might be.

Their second response was to cram more and more and more stuff into each product. If some professors wanted more homework problems or didn’t like the homework problems that are in the book, the publishers had an easy answer. In the next edition, the editors would triple the number of problems. If instructors varied in the topics that were covered, that wasn’t a problem either. Every successive edition would add a couple of more chapters on topics that maybe a handful of professors would assign. In other words, the editors largely gave up on trying to achieve product-market fit. Instead, they tried to put in something for every professor. Some consequences were that, with each successive edition, (a) the book got bigger and heavier to carry, (b) perhaps more importantly, it got more expensive, and (c) students in any given class were using a smaller and smaller percentage of their increasingly big, heavy, and expensive textbooks. Rather than fitting the key into the lock of product-market fit, the publishers just made books that were heavy enough to act as battering rams.

Bad product-market fit is bad for (almost) everyone

What were the consequences of this decision? For starters, each book became more costly and less profitable. (This was particularly true for the books that never sold particularly well and weren’t really profitable in the first place.) If a product costs more to produce and sell, then the maker has to raise the cost of the product. (Before you start feeling too smug about the disaster that you know is coming in the next few paragraphs, remember that this cost problem is just as true for college tuition as it is for textbooks.)

Students obviously did not love the ever-increasing cost of textbooks, particularly since subjects like linear algebra and survey-level chemistry hadn’t changed much in decades. They resented the companies, and they also resented what they perceived as a lack of care on the part of the professors for assigning overpriced products that the instructors then barely used or used in a braindead way that didn’t actually promote learning (like giving a lecture covering the same material that was in the reading or assigning back-of-the-chapter homework problems without going over them in class afterward). So these students found ways to get around having to buy textbooks. Most notably, they created markets for used textbooks and for back-of-the-chapter homework problem answers. Remarkably, several companies that served these student needs grew into so-called “unicorns,” which is when companies achieve a stock market valuation of $1 billion dollars. Chegg was the first, but there have been a few in this mold.

Unicorns in EdTech are particularly rare. The fact that several different start-ups became unicorns by helping students get around the overpriced products and bad service provided by the textbook publishers is a sign of just how bad the publishers’ product-market fit really was.

This was not a good situation for anyone but the shareholders in the unicorns.

Publishers got lucky but learned the wrong lesson

The publishers were slow to figure out digital products. At first, they treated digital assets the same way they treated everything else. They were more junk to throw in along with the kitchen sink. In addition to getting 17 chapters students would never use, students would also get CD-ROMs or web sites that they also probably wouldn’t use.

But then a few smart people inside Pearson figured out that math professors had a problem that could be addressed with a digital product. Math is often taught by carefully crafting a sequence of problems that challenge students to apply the skills they are learning in increasingly sophisticated ways. But the math professors had two challenges. First, since the answers to the back-of-the-chapter problems were now all available online and easy to find, student cheating became so easy that it was almost irresistible. Second, large lecture classes meant that, even with the help of TAs, instructors struggled to give their students timely feedback on the practice problems that students needed to be doing in order to learn.

This second challenge is worth taking a brief detour to examine. Why did survey courses become so large? Not to serve the students or to support the instructors of those survey courses. One reason that survey courses are large because they are profitable for the university. Academics hate to use that word—”profit”—but the cold, hard truth is survey courses generate more money than they cost to teach. That’s profit. Universities then use this excess cash to cross-subsidize other, less profitable activities, like small programs, seminar classes, and graduate programs. Was this the right trade-off for universities to make in order to fulfill their mission goals? I don’t know. I do know that, for the most part, it has gone unexamined. And one knock-on effect was that instructors in those large survey courses increasingly needed to outsource certain functions to the textbook publishers in order to meet their students’ needs as best they could. ((Spoiler alert: In future e-Literate posts, we will examine how many crucial practices of colleges and universities that consider themselves to be “student-centered” are, in fact, faculty-centered.))

Enter MyMathLab, which was (and still is) a blockbuster profitable product. Instructors get the support they needed to give students graded practice problems with at least some feedback while reducing the grading workload on both the instructor and the TAs. (They also got ahead of online homework cheating for a while, although the online sites have since largely caught up.) Students get instant feedback on their practice problems. It might not be as specific or as helpful as feedback from a human expert, but it is better than nothing. Publishers had finally found a good product-market fit for a digital product. MyMathLabs made everyone’s life just a little better than it had been, and the publishers therefore reaped the profits.

But they learned the wrong lesson from MyMathLab. Because the problem they had been worried about was not really product-market fit. Rather, they were worried about the fact that students had found increasingly effective ways to avoid buying new textbooks. For many decades prior, if an instructor assigned a textbook, of course students would go out and buy it. To do otherwise would violate a deep norm. It was such a reliable and long-standing pattern that it felt almost like a law of nature. As long as publishers produced textbooks that instructors continued to adopt, then (they believed) students should continue to buy them. A few people in the industry—including the early creators of Pearson’s MyLabs—understood that the problem was product-market fit. But most employees of publishers didn’t think that way. They thought the instructors were their customers and that those customers mostly were buying a large collection of raw materials for teaching. Students were just a malfunctioning part of the system. So the problem that the publishers were solving for was how to fix the malfunction by making students buy their products again.

The lesson they learned from MyMathLabs was that, if instructors assign graded homework that can only be completed inside the product, then students have to buy it. When I started my tour of duty working inside a textbook publisher in 2011, I was shocked at how explicitly veteran textbook industry folks would state this goal without even a hint of embarrassment. The business model they aspired to was hostage-taking. Students could not get their grades unless they bought the product. Again, publishers did not think of students as stakeholders whose needs they should be concerned about. It took the industry a long time to figure out how to start thinking about students as customers (or, I would argue, as humans).

This strategy also completely misunderstood the needs of the people who the industry thought were their customers, i.e., the instructors. Common pedagogical approaches in math fit well with a sophisticated auto-graded homework platform. A handful of other disciplines have common pedagogical approaches that kinda sorta fit with this model. And then there are the vast majority that don’t. The MyMathLabs family of products sold like crazy (and still does). MyLab Religion, on the other hand? I’m guessing not so much.

Product-market fit in education is extremely hard to achieve even for the very best product designers, and the textbook publishers have mostly proven themselves to be terrible at it.

Technology makes digging deep holes easier

Pearson’s strategy in the early days of MyLabs was to treat their digital product development planning like they treated their analog textbook product development planning. Basically, every disciplinary group did what it wanted without talking to others. In a textbook world, this can work because the tools needed to produce, distribute, and support a physics textbook aren’t any different than those needed for an English composition reader. Books are books. Maybe one uses more colors and more images than another, but basically, the technology is the same. The “platform” for analog textbooks is the printing press and the paper it prints on. The cost of altering that platform to support each new book is limited.

Digital is different. For example, Pearson built a few different platforms that were collectively called “MyLabs” but supported different disciplinary groups. They shared some infrastructure, like a grade book, but there was a lot of separate development for each platform and then many discipline-specific variations of the platforms. This was before cloud computing, so each title had its own distinct application. Changes that were common across the products had to be merged into the code base of each product. And each product had to be separately hosted and updated.

As you might imagine, this was expensive. Printing a religion textbook that doesn’t sell costs money. Building and hosting a MyLabs Religion app that doesn’t sell costs a lot more money. Multiply that by the number of MyLabs apps that didn’t sell because Pearson fundamentally didn’t understand product-market fit, and you have a pretty deep hole.

I’m singling out Pearson only because they essentially created the homework platform market with MyMathLabs. They were no worse than the other publishers. The people who work in these companies aren’t stupid (though some of them learned to be lazy thinkers because their careers developed when textbook publishing was an easy business). As I described earlier, product-market fit for education is really hard, even when you know that’s your challenge and are really good at it. Neither of those things was true for older-generation textbook publisher employees.

Things often get worse before getting better

In a likely apocryphal story, Henry Ford supposedly said, ““If I had asked people what they wanted, they would have said faster horses.” Once the textbook publishers figured out that they were not going to sell auto-graded homework platforms to religious studies professors, they set out to create a faster horse. People told them that they didn’t like how textbooks are so heavy and so expensive. So what did publishers give them?

Ebooks! They’re lighter and they’re cheaper than paper textbooks.

And, at least as textbook replacements, they suck. Every year, textbook publishers would survey students and instructors, asking them how they feel about eBooks. Every year the same answer came back: “We still hate them.” And every year, publishers would try to make them suck less by adding a 37th highlighting color or a fifth color for digital “sticky notes” for writing in the textbook margins. (This started well before the iPad took off, so people were typing their margin notes in the eBooks they were reading on their computers.)

And so, every year, the hole got deeper. eBook versions of textbooks that have to be read on a desktop or laptop computer will never cause students to prefer them over used analog books, no matter how many highlight colors the product supports. Religion professors will never rely heavily on graded homework no matter how fancy the platform is or how many questions are supported. These companies burned massive piles of cash.

Slowly, ever so slowly, the textbook publishers figured out how to build a Model T rather than a faster horse. Modern courseware platforms are textbook replacements that suck less than paper mostly when students are more inclined to use them for the original intended purpose of the original textbook. Which they will choose to do (often enough) if the courseware product is both actually useful and reasonably priced. For example, one big revelation for publishers was that, while instructors in many disciplines don’t want to assign auto-graded homework as core pedagogy, both instructors and students like it if the product can give feedback on how well they remembered and understood the assigned readings. Students like to know this because it gives them a sense of confidence going into class. Instructors like to know so that they can know how to help their students get unstuck without having to turn every class into a lecture that repeats the content covered in the assigned reading.

There are still many, many challenges with this model and many kinks that the publishers have been slowly working out. When Cengage MindTap came out—arguably the first modern courseware platform—the genius CEO of Cengage at the time decided that its first use would be…to function as a reader for flat eBooks. Which would be bundled as an “extra” with the overly expensive and absurdly heavy paper textbook.

Product-market fit rarely displaces a co-dependent relationship

The companies now formerly known as “textbook publishers” have evolved a great deal in recent years. But change is still slow and hard, for reasons that aren’t entirely their fault. Remember my earlier point about the fact that we wouldn’t have auto-graded homework platforms if academia hadn’t first created giant survey courses. The problem that academics asked the publishers to solve for them was grading homework in certain large survey courses. So that is the problem that publishers solved. By and large, academia didn’t ask publishers’ to solve the textbook affordability problem. Until fairly recently, faculty outside of the most strongly access-oriented institutions mostly didn’t ask about textbook prices. So publishers didn’t worry about cost until it became clear that they were in a death spiral where the ever-increasing price was driving away students so fast that annual price increases no longer kept their revenues from falling.

This problem plays itself out all over EdTech. For example, why do revenue-sharing Online Program Management companies (OPMs) exist? Academic leaders defined their problem as, “We need to increase revenue somehow so we don’t have to cut anything, but we also don’t have any money to invest in building new revenue-generating programs.” The answer the vendors came back with was a revenue-sharing OPM model. If academic leaders had instead defined the problem as, “We need to balance our changing costs with our changing revenues so that we can stay sustainable and resilient in a dynamic market,” then they would have gotten a different answer and a different product offering.

Vendors tend to solve the problem that their customers articulate to them. If the customers are in denial about the true nature of the problem that they need to solve, then vendors provide solutions which reinforce underlying problems, whether those problems are an irresponsible reliance on ineffective survey courses to hide financial problems elsewhere in the university or an obsession with building out new academic programs with borrowed money to do the same. If product-market fit is a key in a lock, then most of the OPM conversation has consisted of hand-wringing, debate, and outrage over the shape of the key. It apparently hasn’t occurred to anyone to ask why the lock works the way it does or whether it could work differently.

Returning to the publisher world, vendors have worked hard to develop evidence that their products can actually help students to learn more effectively. One would think that faculty would want to at least seriously examine these claims. But most instructors don’t choose curricular materials products based on evidence of learning impact; nor do they tend to change their teaching practices just because somebody who isn’t a peer tells them that there is evidence that doing so will help students to learn. There are complex reasons for this behavior, just as their are complex reasons for textbook publisher behavior. Nevertheless, we will not get reliably more effective educational products until the market demands them. The key has to fit the lock.

“Never waste a good crisis”

A number of critical aspects of the current COVID-19 crisis have changed the nature of the lock in product-market fit for curricular materials. First, the overwhelming majority of faculty are having to redesign their courses to deal with the current situation. Research and experience tell us that faculty generally only tend to put in the considerable effort of a complete course design when extrinsic factors—such as having to teach a course in an online or blended format for the first time—motivate them to do so. At such moments, faculty are more open to changing up their curricular materials than they normally are.

Many of these instructors likely noticed two big issues when they had to switch to remote learning without that redesign last term. First, every student is now an at-risk student. Drop rates for courses are much higher than normal. Second and relatedly, instructors who move their traditional in-person course formats directly to synchronous web conference without making substantial modifications often find that they are flying blind. The kinds of feedback that they are used to getting from students are no longer available to them. These factors may make them more open to the value of digital affordances such as learning analytics or to evidence of student impact than they have in the past. The lock is changing, which means that the key which will open it is also likely to change.

This is not a short-term change that will revert once the health crisis is over. First, some instructors who go online won’t turn back. Second, the health crisis is in the process of triggering rolling waves of financial crises among colleges and universities which will, in turn, lead to major course and program redesigns, the creation of new ones, and the elimination of old ones. The extrinsic factors that drive instructors to rethink how they teach will continue to be in play for quite some time to come.

Whether the curricular materials companies will be any more nimble at responding to these changes than they have in the past remains to be seen.

Tag: product-market fit

The Great Online Migration and Curricular Materials Product-Market Fit

Change is coming

The Great Online Migration

Major changes in the market

How to Dig a Billion-Dollar EdTech Hole