Can Pearson Solve the Rubric’s Cube?

Love ’em or hate ’em, it’s hard to dispute that Pearson has an outsized impact on education in America. This huge company—they have a stock market valuation of $18 billion—touches all levels from kindergarten through career education, providing textbooks, homework platforms, high-stakes testing, and even helping to design entire online degree programs. So when they announce a major change in their corporate strategy, it is consequential.

That is one reason why I think that most everybody who is motivated to read this blog on a regular basis will also find it worthwhile to read Pearson’s startling publication, “The Incomplete Guide to Delivering Learning Outcomes” and, more generally, peruse their new efficacy web site. One of our goals for e-Literate is to explain what the industry is doing, why, and what it might mean for education. Finding the answers to these questions is often an exercise in reading the tea leaves, as Phil ably demonstrated in his recent posts on the Udacity/SJSU pilot and the layoffs at Desire2Learn. But this time is different. In all my years of covering the ed tech industry, I have never seen a company be so explicit and detailed about their strategy as Pearson is being now with their efficacy publications. Yes, there is plenty of marketing speak here. But there is also quite a bit about what they are actually doing as a company internally—details about pilots and quality reviews and hiring processes and M&A criteria. These are the gears that make a company go. The changes that Pearson is making in these areas are the best clues we can possibly have as to what the company really means when they say that they want efficacy to be at the core of their business going forward. And they have published this information for all the world to see.

These now-public details suggest a hugely ambitious change effort within the company. Phil and I have consulted for a few textbook publishers, including Pearson, and I worked for Cengage for a year and a half. We have a pretty good idea of the magnitude of the change management challenges these companies face right now and the strategies that various publishers are bringing to bear in an effort to meet them. I can say with absolute conviction that what Pearson has announced is no half-hearted attempt or PR window dressing, and I can say with equal conviction that what they are attempting will be enormously difficult to pull off. They are not screwing around. Whatever happens going forward, Pearson is likely to be a business school case study for the ages.

As if all of this drama weren’t enough, Pearson’s strategy raises another question which should be fascinating for educators; namely, can a rubric transform a multi-billion-dollar company?

Fair warning: This post is ridiculously long. Even by my standards.

Transforming U

Before we can really dig into the questions at hand, let’s talk about people’s beliefs about Pearson and how those beliefs may color their interpretations of the company’s actions. I’ve been thinking a lot lately about motivated cognition—the depressingly robust and pervasive process by which humans strongly tend to draw conclusions from new facts or arguments that are consistent with what they already believe (and want to believe), particularly about hot-button issues like politics. (As usual, Mike Caulfield has already written a thought-provoking post on this topic and some of its implications for education.) Now that Blackboard isn’t suing anybody anymore, Pearson is probably the single most emotionally charged brand in the education industry. Anything at all having to do with their actions or intentions is likely to function as something of a Rorschach test. It can be helpful in such situations to try to analyze the questions before us in a different, less emotionally charged context first, and then apply what we know about the context afterward to see how it might change our judgments.

To that end, I propose a thought experiment. Imagine if the president of U.C. Berkeley wrote the following:

The learning challenge is unrelenting in scale (by 2025 the world’s population will have doubled twice in the space of 75 years), and in its increasing complexity. Globally, at least 60 million children remain without access to primary school education, and an estimated quarter of a billion are lacking in basic reading and writing skills. The International Labor Organization estimates 200 million people are unemployed in 2013. The labor market demand for routine tasks has fallen rapidly over the past 50 years, meaning we can no longer rely on memorizing and reproducing knowledge acquired from a specific curriculum to support us in our careers.

At U.C. Berkeley, we are building our strategy around trying to make some impact on those big needs. Education remains very much a black box in which inputs are turned into outputs in ways that are difficult to quantify or predict consistently. But that’s no excuse for us or indeed for anyone else involved in education. We need to learn more. At Berkeley, our mission is to help people make progress in their lives through learning. So we had better be sure that we can demonstrate that progress and measure our impact in a meaningful way. Our commitment is to make improving learning outcomes the central driving force of the university, to report on progress and to use the findings to propel continuous innovation and improvement.

We borrowed the term ‘efficacy’ from healthcare. Just as a pharmacist would talk about relieving a tickly cough rather than listing the ingredients of the syrup, we want to be able to state the outcome we help to produce rather than describe the input we provide. Efficacy might be complex enough in healthcare; in education it is frequently many more times so. The process of learning is a social, dynamic, and interactive one. Context — culture, community, language — is central to the learning outcome. We may never be able to ‘prescribe’ an educational process to deliver a desired learning outcome with the same precision as a doctor or pharmacist. Yet all of us involved in education must take a lesson from that most famous phrase in school reports: we must do better.

As we transform our university, our aim is to ensure that our actions, our decisions, and our investments are driven by a clear sense of how we can make a measurable difference on learning outcomes. That idea will shape the choices that we make—what we do, what we stop doing, where we invest and innovate, who we partner with, how we engage with our students, and how we recruit, develop and reward our people.

To get started, we needed a definition. Keeping it simple, we chose to define as follows: a course has efficacy if it has “a measurable impact on improving people’s lives through learning.”

Note that, with this definition, it is the learning outcome that we are pursuing. Passing a test or an exam is good, but it is not an end in itself: what we really want to see is the benefit of doing so in somebody’s life. To give an example, if a student passes an ESL course, that is good. What really matters, though is that their mastery of English helps them to make progress in their career. Or, as another example, achieving an “A” in a class is good, but what really matters is that, as a result, the student can progress to their degree or career of their choice, prepared for work, college, and citizenship.

Central to this effort, we designed our Efficacy Rubric which, instead of imposing a model, asked a set of questions which, once answered by the faculty, would set the course on a path to efficacy. The rubric is in four sections. The idea is that, for each of the four sections, an judgment is reached on a four-point scale. Those four judgments can be combined into a single overall judgement on the likely efficacy of the course. We should emphasize at the outset that the rubric does not claim to be scientific: its basis is in informed human judgment. Crucially, though, it is provocative and challenging. It asks difficult questions; it demands that people think deeply about what they do and why.

Each of the four sections of the rubric is intended to get at a specific vital angle of the course’s efficacy. The brief summary is as follows:

• Section 1, The Efficacy Goals: What are you trying to do?
• Section 2, The Evidence: Why should we believe you?
• Section 3, The Plan: How do you intend to achieve your efficacy goals?
• Section 4, Capacity: Do you, and those you depend on, have the knowledge, skills, and relationships to deliver the efficacy goals?

The idea is that if the faculty can answer the questions convincingly, then their courses are likely to either already be demonstrating efficacy, or at least to be on the path to efficacy.

This would be a pretty provocative pronouncement for the president of a public university (or any university) to make. It would raise all sorts of questions. At the highest level we might ask, Is efficacy the right model for expressing our educational responsibility to our students? What kinds of conversations would adopting this model lead to among faculty, and how would it influence the way courses are designed and taught? What trade-offs would we make by adopting this particular framework, and do we do more good than harm, on balance, by picking one framework for everyone? Then there would be practical questions. How would faculty begin to apply such a framework as a group, across all departments and courses? Would they even accept the idea that they should? Do they have the skills to do so? What support might they need?

The words above come not from the president of U.C. Berkeley but from Pearson’s report, with some elisions and modifications on my part to make them fit with the thought experiment. I do think that we may ask different questions and, in some cases, answer the same questions differently when we think about the framework in terms of its appropriateness and usefulness for education separately from its context as vendor public relations materials.

Transforming (For? Because Of? Despite?) You

Of course, Pearson is not a public university, and their decision to pursue this strategy as what has historically been a textbook company also raises some different questions. As you think about Pearson declaring that they are now focused on evaluating all their products based on efficacy, one reaction that you may be having is something along the lines of, “Wait. You mean to tell me that, for all of those educational products you’ve been selling for all these years, your product teams are only now thinking about efficacy for the first time?” Another reaction might be, “Wait. You mean to tell me that you think that you, a textbook company, should be defining the learning outcomes and determining the effectiveness of a course rather than the faculty who teach the course?” These questions represent the rock and the hard place between which Pearson finds itself. I will address the former now and the latter in the last section of this post.

It’s impossible to unpack the meaning of Pearson’s move without putting it in the context of the historical relationship between the textbook industry and the teachers who adopt their products. Despite all of the complaints about how bad textbooks are and how clueless these companies are, the relationship between textbook publishers and faculty is unusually intimate. To begin with, I can’t think of any other kind of company that hires literally thousands of sales representatives whose job it is to go visit individual faculty, show them the company’s products, answer questions, and bring feedback on the products back to the company. And speaking of those products, the overwhelming majority of them are written by faculty—many with input from an advisory committee of faculty and pre-publication reviews by other faculty. You can fairly accuse the textbook publishers of many different faults and sins, but not taking faculty input seriously isn’t one of them. Historically, they have relied heavily on that faculty input to shape the pedagogical features on the textbooks. And they have had to, because most of the editors are not teachers themselves. More often than not, they started off as textbook sales reps. If they taught at all, it was typically ten or twenty years ago, and just for a few a few years—long enough for them to figure out that teaching and the academic life weren’t for them. This doesn’t mean that they don’t care about pedagogy or don’t know anything about it, but it does mean that most of what they know comes from talking with their authors and customers.

And by “customers,” I mean faculty, despite the fact that it is the students who actually buy the product. Pearson’s choice to build their learning outcomes effort around a term that comes from the pharmaceutical industry is an historically apt one for the textbook industry. In higher education in the United States, faculty prescribe textbooks and students purchase them. As a result, textbook publishers have generally designed their products to please faculty rather than students. One consequence of this is that they had no need to distinguish product features that offer faculty convenience from those that actually impact student learning. When faculty/customers said to the textbook publishers, “I want my book to come with slides, lecture notes, and a self-grading homework platform so that I don’t have to put as much work into that annoying survey course the department head is making me teach,” then that’s what they provided. Whether that collection of materials had positive impact, negative impact, or no impact on student outcomes was not a question that the textbook publisher had any particular reason to ask. For the most part, the publishers relied on their authors and customers to make good decisions for the students. As long as the they provided the raw materials that the faculty said they needed, the companies’ work was done.

That relationship has been slowly breaking down over the past couple of decades and has now reached a crisis point, for a variety of reasons. The first and most obvious is that students are no longer meekly taking—or at least, and more importantly for the textbook publishers, buying—the “meds” prescribed by their teachers. The used textbook market has been growing since at least 1980s when I was a student, meaning that the number of copies sold of each textbook edition has been going down. For many years, the textbook companies compensated for their losses by artificially creating reasons for new, incompatible editions every few years and by raising prices every year to make up in dollars per book sold what they were losing in number of books sold. This strategy worked for a surprisingly long time, but the publishers have finally hit the price ceiling in the market—aided somewhat by new ways for students to find used books, increasing student willingness to skip buying the books altogether, and increasing faculty willingness to include free or low-cost alternative curricular materials. The textbook industry is in trouble and no longer has the luxury of just trying to squeeze another year out of the old model. Revenues and profits are down. Even Pearson, which is in the best shape of the cohort, have had bad financial results and expect the problems to continue. Most of the major publishers have gone through very significant reorgs, downsizings, and in some cases recapitalization exercises in the past few years. Cengage declared bankruptcy. McGraw Hill sold off their education business to private equity. And so on.

By and large, the textbook publishers have adopted three new strategies to deal with their existential crisis. First, they are diversifying into services. They see that a lot of schools are having trouble, for example, launching online learning programs, so they provide outsourced services to those schools. Second, they are bundling homework platforms and curricular content into one digital platform. If students have to get graded in the homework platform in order to pass the class, then they have to buy the product. And if those homework platforms rely on digital access keys that expire at the end of the semester, then students can’t resell them into the used textbook market. And third, they are finally giving some thought to how they can design products that students actually want to use and buy. They are doing this mostly by trying to learn lessons from other industries about how to design products that make their customers happy. Cengage, for example, has recently undergone a major restructuring to emphasize product management teams in the style of software companies rather than traditional editorial teams.

There is a fourth strategy that is only beginning to emerge. Textbook publishers have noticed that products that have the word “analytics” or “adaptive learning” associated with them sell well. There is still a lot of whiz-bangery to the industry’s thinking, but it is starting to dawn on them that their products might sell better if they can prove that those products actually…you know…work. If they do work. If they provably help people to learn.

Pearson’s efficacy strategy is all about the lessons that they are learning from the success of their MyLabs products. They are trying to change the buying criteria for curricular materials. What if, instead of choosing a textbook based on the reputation of author who wrote it, faculty were to choose the materials they prescribe to their students based on the evidence that those materials actually impact learning? What if students thought of those materials not just as that $150, 10-pound piece of garbage that their professor is making them buy and of which they will only read 10%, but as something that they have reason to believe will help them? What if university administrations and state governments began demanding evidence that students are getting value from the education that they are paying for? (Oh wait; that’s already happening.)

Pearson believes that this world is coming, they want to accelerate its arrival, and they believe that they can position themselves as the market leader in it. As I will outline in the next section, we have plenty of evidence to suggest that they are very, very serious about it. Whether they can actually pull it off is a vastly harder question.

Transforming P(earson)

Many academics who have never worked in a large corporation harbor a stereotype that whatever the CEO says goes (in stark contrast to the academic environment). And there are some companies for which that is mostly true. When I worked at Oracle, I was amazed at how, even in an organization of 120,000 people, when Larry Ellison wanted something to happen, it happened. Even more amazingly, if there was something that somebody in that company wanted to happen, and that somebody wasn’t Larry Ellison or one of his lieutenants, it rarely happened. But in my experience, Oracle is the exception that proves the rule. Most companies have their fiefdoms, their internecine squabbles, their cliques and their cultural inertia that a CEO cannot simply overrule. Let’s start with the sales force. In many companies that have a sales force (including textbook companies), sales reps get either most or all of their compensation not from salary but from commission. Which means that, in an important sense, they don’t really work for the company. They are like franchisees. They are bound by certain rules of their agreement with the company that provides the products that they sell, but they are fundamentally in business for themselves. In such a scenario, which products do salespeople sell? There are two answers to this question: (1) whichever ones they can and (2) whichever ones earn them the biggest commission. Suppose you’re the CEO of a textbook company and you want your sales force to emphasize a new product line that you think has a bright future or that takes your company in a new direction. But your sales rep finds it easier to sell the old product. Which product is she going to sell? Why, the one that’s easier. You can tip the scales a little by providing a bigger commission for the new product, but at the end of the day, your rep is going to weigh that commission against how hard the sale is and how many sales of each kind of product she thinks she can make. Even if you, as CEO, are willing to let the company’s revenues take a short-term hit by emphasizing the product that has better long-term prospects and losing some sales in the process, you can’t force your rep to go along with that decision because she doesn’t work for you. Not really.

So most typical companies that have substantial commission-driven sales forces have at least some significant limits on the CEO’s ability to turn the ship, so to speak. But in this regard, textbook companies are not typical companies. They are much, much worse. All the big textbook publishers got big because they are what business folks call “rollups“. Some physics and engineering textbook publisher somewhere had a really great couple of years and decided to buy a math textbook publisher. That company grew and eventually bought a biology publisher. And so on. But the main business purpose of these mergers and acquisitions was to save costs. Why run two distribution warehouses when you can run one? Why have two printing operations, or two marketing or HR departments? In terms of the core work of producing content, the physics editors have no more reason to talk to the biology editors now that they are under one roof than they did when they were separate. Each small publisher still ran mostly on its own. And that strategy worked just fine until publishers had to do start doing big, whole-company projects like building digital platforms. Then it became a nightmare. But because each individual business unit has its own strong cultures, processes, and internal loyalties, getting them all to line up behind one strategy and set of priorities—particularly in cases where the leader of one business unit had to sacrifice in the interest of helping another—was incredibly hard. Again, in a sense each business unit was a franchise, compensated through its own success and working in its own ways. The history of large textbook companies in the past decade is a history of new leadership trying to centralize power, failing, and then being replaced by new leadership that starts all over again.

So if you’re the CEO of major textbook publisher and you want to unite the entire 45,000-employee company around a plan to transform the way the company does business, what do you do? Surprisingly, Pearson’s CEO John Fallon’s answer was, “I’ll create a rubric.”

I’m not going to analyze Pearson’s rubric in detail here because (a) this post is already ridiculously long even without that analysis, and (b) there are plenty of others in the educational community who are incredibly well qualified to offer a critique, and I’m hoping that somebody will. (Pearson has an online tool for applying the rubric if you want to give it whirl.) I’ll say this much about it: It’s nothing special. It’s not bad, but it’s not genius either. There are plenty of flaws and limitations you could find if you worked at it and applied it broadly enough. There is no magic in it.

But here’s the thing: There is never any magic in a rubric. The magic, when there is any, happens from the norming conversations that the rubric engenders. It happens when one colleague says to another, “What do you mean by ‘quality of evidence’?” Or “I scored that course a 2 on effectiveness. Why did you think it was a 4?” To the degree that the Effectiveness Framework proves to have any magic for Pearson, it will be in the norming conversations that it engenders across the company. Like our hypothetical Berkeley president, Fallon is working with diverse groups within an institution that has a culture of independence and Balkanization. Some of this is for good reason; conversations about effectiveness in chemistry education should look very different from conversations about effectiveness in fine arts education. Some of the fractiousness is about lack of a common culture and language necessary to discuss what otherwise are common challenges. And some of it is just human territoriality and self-interest. The first two challenges might be addressed by having a deep and wide ongoing norming conversation about a rubric that is general enough to cover a wide range disciplines and products but focused enough to provoke important discussions. The goal is for that conversation to become the basis for a new culture. The third challenge might be addressed by reinforcing that culture through your HR and other business practices.

The actions spelled out in Pearson’s document demonstrate that the company is making a substantial investment in activities that should be conversation- and culture-building:

• They recruited volunteer business leaders within the company who were interested in piloting the framework.
• The created a process by which the product teams under those business leaders conducted self-assessments of their products using the framework. Typically six to eight members of each team participated in the reviews.
• Each product team would norm their self-evaluations with the central efficacy review team.
• After the first few reviews, Pearson began running training sessions on the review process for volunteers—in fact, prospective attendees had to apply—in at least five locations around the globe.
• To date, the company has conducted 100 efficacy reviews across 15 countries and trained 600 reviewers across 25 countries.
• They have developed the capacity to conduct 150 new efficacy reviews per year.
• They have recruited a volunteer Efficacy Steering Committee of people who “lead efficacy for entire regions or business units on top of their day jobs.”
• They have developed a workshop to practice applying the efficacy framework which they have delivered to 5,500 participants.
• They are developing an e-learning module for training on the framework.

It’s hard to argue that internal activity of this scope is just PR. This is internal-focused effort, not just decorations for press releases. How much of it is genuine culture-building? It’s hard to know from outside, but the company took pains in the document to talk about the importance of fostering “debate” and building enthusiasm that inspires employees to participate rather than forcing them.

In addition to the actions themselves, the document gives us some clues as to how Pearson executives characterize what all these efforts add up to:

To convene a company around a single idea, you need to do three essential things: realign the existing portfolio around the idea, set the routines in place to embed that idea in the future, and build a community to make the change irreversible.

Reviewing the existing portfolio will help steer the current business towards delivery better outcomes. But in order to have perpetual change, we also need to shift an idea from the periphery to the centre of the organisation. You need to ensure that it’s a recurring thought, day in, day out, and a part of every employee’s routine.

In fact, you need to deconstruct the essential routines that form the core business processes of a company, and reconstruct them around the idea you’re seeking to embed. It’s the only way to ensure that future generations of employees and leaders across the company hold the values on which you depend.

We will need to address each of these to deliver ‘institutionalisation’ of efficacy at Pearson, taking all the elements of the company that keep the core business engine running and ask ourselves: “What if we could redesign these processes from scratch to ensure they resulted in delivering learning outcomes?”

If we take them at their word, they are trying to rebuild the company from the ground up. Again, the scope of the effort is consistent with the rhetoric. In addition to the cultural work, they are making significant changes to their business processes:

• They have created a global educational research function in the company, as well as a research center for each of three major areas—Schools, Higher Education, and Professional Learning.
• They are developing a repository of internal and external educational research that will be available to “business leaders” (presumably this means within Pearson).
• They have developed a partnership with Nesta to define standards of evidence in educational efficacy research.
• They have built a formal efficacy evaluation into their business acquisition process which applies to every acquisition above $3 million.
• They apply an efficacy review to internal product investments that are over $1 million and to selected smaller projects that have strategic importance.
• They have developed efficacy review tools that have to be applied on both bids and post-delivery reviews for bids on large-scale projects.
• They have added statements about commitment to efficacy in their job advertisements, made efficacy a topic in interviews of prospective employees, and added efficacy training to employee orientation.
• They are adding efficacy criteria to employee performance reviews and, in some cases, linking efficacy results to compensation.
• They are including efficacy updates in their monthly progress reports to their board of directors.

So we have plenty of reason to believe that Pearson is quite serious about becoming a company whose mission is to deliver educational efficacy, whatever that may mean to them. But that leads us to several more questions. First, does Pearson’s notion of efficacy truly align with the academic community’s ideas of what a good education is supposed to accomplish? And second, will Pearson be successful as a company if they deliver “efficacious” educational products? To my mind, these are the same question. Pearson can succeed in the market with this strategy to the degree that they build products which accomplish educational goals that educational stakeholders agree is important, as proven by measures that educational stakeholders agree are valid and significant.

And this brings us to a huge gap in Pearson’s thinking about efficacy to-date.

The Other Half of the Job

Let’s think some more about the analogy to efficacy in health care. Suppose Pfizer declared that they were going to define the standards by which efficacy in medicine would be measured. They would conduct internal research, cross-reference it with external research, come up with a rating system for the research, and define what it means for medicines to be effective. They would then apply those standards to their own medicines. And, after all is said and done, they would share their system with physicians and university researchers in the hopes that the medical community might be reassured about the quality of Pfizer’s products and maybe even contribute some ideas to the framework around the edges. How confident would we be that what Pfizer delivers would consistently be in the objective best interest of improving health? This is not entirely hypothetical; much of the drug research that happens today is sponsored by drug companies. Unsurprisingly, this state of affairs is viewed by many as deeply problematic, to say the least. It certainly doesn’t help the brand value of Pfizer. But at least much of that medical research is conducted by physicians and academic researchers and is subject to the scientific peer review process. Pearson is creating their framework largely on their own, selectively inviting in external participants here and there.

I get why they had to do this. The company is bleeding money, it will take some time to stop the flow of blood, and they couldn’t wait to build consensus before they tied the tourniquet. But it is not going to get them where they want to go. While there are obvious concerns about ethics and about whether such a company-driven approach is fundamentally compatible with progress on complex questions such as defining what an education is good for and how we know when we have achieved these ends, I want to focus on the business aspects of the problem. I want to focus on why continuing down this path is bad for Pearson. Or rather, why driving hard toward becoming facilitators rather than owners of efficacy research is good for Pearson.

I could give a number of examples, but one should hopefully suffice. In preparing to write this post, I asked Annie Cellini, Pearson’s Senior Vice President of Marketing and Strategy, whether Pearson intends to share the completed rubrics for their products with customers and prospects. This was her reply:

Though we don’t plan to share product efficacy scoring as part of our sales and marketing materials per se, where a product has a strong research and evidence base, we will communicate that to customers. It’s also worth saying that the most important output of an efficacy review isn’t a rubric score. We believe that much of a review’s value comes from the conversations that it prompts teams to have, which focus on the path forward, and on how to improve the product or service from a learner perspective. A poor score does not mean the product doesn’t work well. It often means that teams are not collecting the type of data needed in order to get a sufficiently robust view of the product’s efficacy, or they may not have a sufficiently practical plan to continuously enhance the product based on data. Their improvement plan will encourage them to start gathering new information, to start working in new ways, and to make sure that their customers are aligned with the outcomes they plan to achieve and understand their role in the product’s path to efficacy.

This is a perfectly sensible and responsible reply if you believe that the main value of the Efficacy Framework to customers is in the data that results from the work a product team does after an efficacy review. But remember, the magic of the rubric is in the norming conversations. Annie’s reply suggests that Pearson understands this in terms of the Pearson-internal processes but not yet in terms of their relationships with their customers. If Pearson were to say to faculty, “Here’s what we think we know about the efficacy of this product, here’s what we don’t know yet, and here is how we are thinking about the question,” they might get a number of responses. Maybe they would get, “Oh, well here’s how I know that it’s effective with my class.” Or “The reason that you don’t have a good answer on effectiveness yet is that your rubric doesn’t provide a way to capture the educational value that your product delivers for my students.” Or “I don’t use this product because it has direct educational effectiveness. It frees me up from some grunt work so that I can conduct activities with the class that have educational impact.” Most of all, if you’re John Fallon, you really want faculty to say to their sales reps, “Huh. I never thought about the product in quite those terms, and it makes me think a little differently about how I might use it going forward. What can you tell me about the effectiveness of this other product that I’m thinking about using, at least as Pearson sees it?” And you really want your sales reps to run back to the product teams, hair on fire, saying “Quick! Tell me everything you know about the effectiveness of this product!”

Pearson won’t get that conversation by just publishing end results of their internal analysis when they have them, which means that they have a high risk of failing to align their products with the needs and desires of their market if they think about the relationship between their framework and their customers in that way. I don’t think Pearson fully gets that yet. While the authors of The Incomplete Guide frequently invoke terms like “community” and “leaders” in the document, they generally seem to mean the community and leaders within Pearson. The company’s efforts to reach out to the academic community for feedback and participation are generally framed as an extension of their efforts rather than the very heart of them. And yet, Pearson’s brightest possible future is not as a company that designs educationally effective products, but as one that facilitates conversation and research about efficacy within the broader academic community (and in so doing is able to design products that their customers agree are effective for important educational goals as determined by meaningful measures).

There are a number of reasons why this part of the transformation will be at least as difficult as the part that Pearson is undertaking now. First, it is far from clear that the company has the trust of the academic community that would be necessary for them to take such a role. That would have to be built, in some cases from the ground (or even the basement) up. Pearson does have real strengths that are known within certain segments of the academic community—in data science, for example—but this does not transfer to a general reputation. Second (and relatedly), unlike the medical research community, the educational research community is still nascent and fragmented. Finding non-paternalistic but effective ways to bring that community together and facilitate useful conversations will be difficult to say the least. These two challenges are outside the company’s sphere of control, which means that Pearson will have to develop new ways to think about how to build their relationships with the broader educational community. Mike Caulfield wrote a post a while back about Eric von Hippel’s work on customer innovation, which is one potential font of inspiration for the company. But make no mistake; this will be a tough nut to crack.

Internally, changing the way they think about answering the questions that the framework asks them will entail as much subtle, difficult, and pervasive re-engineering of the corporate reflexes and business processes as the work being undertaken now. As I described earlier, all textbook companies that have been around for a while are wired for a particular relationship with faculty that is at the heart of how they design, produce, and sell their products. Their editors have gone through decades of tuning the way they think and work to this process, and so have their customers. When Pearson layers a discussion of efficacy onto these business processes, a tension is created between the old and new ways of doing things. Suddenly, authors and customers don’t necessarily get what they want from their products just because they asked for them. There are potentially conflicting criteria. The framework itself provides nothing to help resolve this tension. At best, it potentially scaffolds a norming conversation. But a product management methodology that can combine knowledge about efficacy, user desires, and usability requires more tools than that. And that problem is even worse in some ways now that product teams have multiple specialized roles. The editor, author, adopting teacher, instructional designer, cognitive science researcher, psychometrician, data scientist, and UX engineer may all work together to develop a unified vision for a product, but more often than not they are like the blind man and the elephant. Agreeing in principle on what attributes an effective product might have is not at all the same as being able to design a product to be effective, where “effective” is shared notion between the company and the customers.

Pearson will need to create a new methodology and weave it into the fabric of the company. There are a number of sources from which they can draw. The Incomplete Guide mentions Lean Startup techniques, which are as good a place to start as any. But there is no methodology I know of that will work off-the-rack for education, and there certainly is no talent pool that has been trained in any such methodology. I have worked with multiple educational technology product teams in multiple companies on just this problem, and it is very, very hard. In fact, it may be the single hardest problem that the educational technology industry faces today, as well as one of the harder problems that the larger educational community faces.

We should not underestimate the scope of the effort that Pearson is making, but neither should they underestimate the scope of the challenge that is yet to come.

I have often said that corporations are amoral in the same sense that lawnmowers are amoral. They do what they are designed to do. Pearson has published a remarkably detailed blueprint of how they intend to rebuild their machine from the ground up. And in doing so, they have revealed just how hard a task they have in front of them.