e-Literate

Present is Prologue

Tag: Efficacy

  • Can Pearson Sell Efficacy?

    Can Pearson Sell Efficacy?

    Almost five years ago, when Pearson announced that the company would reorganize itself around efficacy, I was impressed by the degree to which the company was going “all in” on a very much unproven strategy. It was incredibly bold. I wrote,

    In all my years of covering the ed tech industry, I have never seen a company be so explicit and detailed about their strategy as Pearson is being now with their efficacy publications. Yes, there is plenty of marketing speak here. But there is also quite a bit about what they are actually doing as a company internally—details about pilots and quality reviews and hiring processes and M&A criteria. These are the gears that make a company go. The changes that Pearson is making in these areas are the best clues we can possibly have as to what the company really means when they say that they want efficacy to be at the core of their business going forward. And they have published this information for all the world to see.

    These now-public details suggest a hugely ambitious change effort within the company. Phil and I have consulted for a few textbook publishers, including Pearson, and I worked for Cengage for a year and a half. We have a pretty good idea of the magnitude of the change management challenges these companies face right now and the strategies that various publishers are bringing to bear in an effort to meet them. I can say with absolute conviction that what Pearson has announced is no half-hearted attempt or PR window dressing, and I can say with equal conviction that what they are attempting will be enormously difficult to pull off. They are not screwing around. Whatever happens going forward, Pearson is likely to be a business school case study for the ages.

    Pearson bet the farm on efficacy. As far as I can tell, they are still betting the farm on efficacy. If anything, they have doubled down in the five years since I wrote those words.

    There are a number of reasons why this bet is remarkable, not least of which is that we still don’t know if a curricular materials company can be successful in the long run by promoting efficacy as a primary value proposition. In addition to all the hard work that Pearson needs to do to credibly claim that their products support some sensible definition of efficacy, they also have to tackle the equally hard challenge of convincing faculty that the company’s vision of efficacy, as delivered in their products, is a good reason to pick their product instead of the (many) alternatives. They have an enormous customer communications challenge. That was one of the main points of my original 7,000-word post on the company’s strategy.

    Which is why I find it mystifying to see an article in Forbes which seems almost designed to distract from or even directly undermine the efficacy message that the company has been carefully honing over the last half a decade. The article, “How 174 Year Old Pearson Is Developing The Netflix Of Education” is a longish interview with Albert Hitchcock, Pearson’s Chief Operating Officer and Chief Technology Officer. The jarring title refers back to Mr. Hitchcock’s previous use of the analogy in an interview entitled ‘Pearson aims to become the ‘Netflix’ of education.‘ In responding to that first piece two years ago, I wrote,

    Given his profile on LinkedIn, Mr. Hitchcock appears to be new to education except for whatever memories he has of his own days as a student. Let me offer a couple of suggestions on how to get along in education for him and the many vendor employees who are in a similar situation:

    1. Never, ever, say that you want your company to be the Uber of education, the Airbnb of education, the Pokemon GO of education, or the [insert name of tech darling] of education unless you really enjoy being a recluse (or you are secretly a double agent for your employer’s direct competitor).
    2. If you absolutely must say something like the above, then do not say you are the Netflix of education. Honestly, Netflix isn’t even great at being the Netflix of movies. The last time they recommended a movie that I actually wanted to watch was…uh…never.

    There is a recurring cultural fantasy that “solving” the education “problem” consists of creating a customized playlist of little content bits. So really, more like the Spotify of education, if you want to play that game. This idea enrages educators because it trivializes what they do. Nobody who has taught believes that proper sequencing of content chunks is the hard part. (For a fully fleshed out prior example—or a “worked example” in teaching parlance—of how the sorts of comments Mr. Hitchcock have made typically play out in educational corporate branding over time, see my post-mortem on Udacity’s pivot away from higher education.)

    In the more recent Forbes piece, the interviewer picks up on the Netflix analogy and asks Mr. Hitchcock about it. Not only does Mr. Hitchcock choose not to disavow the analogy; he expands on it by adding Spotify and Amazon. I’ll parse his exact language later in this post in the interest of fairness, but my point from the previous post stands. Regardless of the intention behind the analogies, they are toxic and should be avoided at all costs. This is doubly true for Pearson because, as I will explain in the next section of this post, they also run directly counter to the more credible, more refined articulation of the efficacy strategy that the company is promoting in 2018.

    The fact that the analogies did make it into an interview with a mainstream publication like Forbes raises some larger questions. Is Pearson less committed to efficacy than I have believed them to be? Is there poor alignment in the company around what efficacy means? Are they just really bad at messaging? Did Mr. Hitchcock’s intent somehow get represented unfairly by a few poorly chosen words that were then taken out of context by an editor?

    To find out the answers to these questions, I spoke with Pearson’s President of Global Product Tim Bozik, SVP of Efficacy and Research Kate Edwards, and CEO John Fallon. Here’s the short version of what I have concluded, based on those interviews and other evidence:

    1. I am mostly still convinced that Pearson is fully committed to efficacy as a primary value proposition for all of their products and services going forward.
    2. I do believe that the Forbes interviewer was bad, as were some of the editorial choices made by the publication. (Particularly the headline.)
    3.  Even granting the benefit of the doubt on the previous point, Mr. Hitchcock made a number of very bad choices that no executive in his position should make and that cannot be explained away by blaming the editor or the interviewer.
    4. Mr. Bozik and Ms. Edwards seemed fully aligned on what Pearson means by efficacy, how to talk about it, and how central it is to the company. Within the rules that they were bound to follow in an on-the-record interview through formal PR channels, they did the best job they could to articulate the most attractive and compelling version of Pearson’s efficacy strategy while diplomatically establishing some distance from Mr. Hitchcock’s questionable analogies.
    5. Mr. Fallon chose a different path. He went to some lengths to justify or explain away the analogies. In doing so, he re-opened questions that Ms. Edwards and Mr. Bozik had all but closed for me regarding whether Pearson has the sensitivity and message discipline they will need to earn their customers’ trust regarding their commitment to efficacy.

    For the long version, read on.

    But before you do, you should be aware of our conflicts of interest, which I’m going to describe in more than the usual detail. Pearson is a current sponsor of the Empirical Educator Project. In addition, they have periodically engaged us in consulting projects over the years (although we do not have any current engagements with them.) Some of these projects have involved their efficacy work, either directly or indirectly. We received prior permission from the company to blog publicly about the first such engagement. Most recently, we were hired to review the company’s public  efficacy reports, provide the kind of feedback that we might publish in a blog post, and offer suggestions for improving the work. You can judge for yourself whether these engagements make us more or less trustworthy in our assessments.

    Netflix and Spotify analogies are fundamentally incompatible with Pearson’s view of efficacy

    In order to understand why these analogies are particularly bad for Pearson’s efficacy effort, it’s important to understand how the company’s position on efficacy has evolved. In that original post five years ago, I wrote,

    Let’s think some more about the analogy to efficacy in health care. Suppose Pfizer declared that they were going to define the standards by which efficacy in medicine would be measured. They would conduct internal research, cross-reference it with external research, come up with a rating system for the research, and define what it means for medicines to be effective. They would then apply those standards to their own medicines. And, after all is said and done, they would share their system with physicians and university researchers in the hopes that the medical community might be reassured about the quality of Pfizer’s products and maybe even contribute some ideas to the framework around the edges. How confident would we be that what Pfizer delivers would consistently be in the objective best interest of improving health? This is not entirely hypothetical; much of the drug research that happens today is sponsored by drug companies. Unsurprisingly, this state of affairs is viewed by many as deeply problematic, to say the least. It certainly doesn’t help the brand value of Pfizer. But at least much of that medical research is conducted by physicians and academic researchers and is subject to the scientific peer review process. Pearson is creating their framework largely on their own, selectively inviting in external participants here and there.

    I get why they had to do this. The company is bleeding money, it will take some time to stop the flow of blood, and they couldn’t wait to build consensus before they tied the tourniquet. But it is not going to get them where they want to go. While there are obvious concerns about ethics and about whether such a company-driven approach is fundamentally compatible with progress on complex questions such as defining what an education is good for and how we know when we have achieved these ends, I want to focus on the business aspects of the problem. I want to focus on why continuing down this path is bad for Pearson. Or rather, why driving hard toward becoming facilitators rather than owners of efficacy research is good for Pearson.

    I could give a number of examples, but one should hopefully suffice. In preparing to write this post, I asked Annie Cellini, Pearson’s Senior Vice President of Marketing and Strategy, whether Pearson intends to share the completed rubrics for their products with customers and prospects. This was her reply:

    Though we don’t plan to share product efficacy scoring as part of our sales and marketing materials per se, where a product has a strong research and evidence base, we will communicate that to customers. It’s also worth saying that the most important output of an efficacy review isn’t a rubric score. We believe that much of a review’s value comes from the conversations that it prompts teams to have, which focus on the path forward, and on how to improve the product or service from a learner perspective. A poor score does not mean the product doesn’t work well. It often means that teams are not collecting the type of data needed in order to get a sufficiently robust view of the product’s efficacy, or they may not have a sufficiently practical plan to continuously enhance the product based on data. Their improvement plan will encourage them to start gathering new information, to start working in new ways, and to make sure that their customers are aligned with the outcomes they plan to achieve and understand their role in the product’s path to efficacy.

    This is a perfectly sensible and responsible reply if you believe that the main value of the Efficacy Framework to customers is in the data that results from the work a product team does after an efficacy review. But remember, the magic of the rubric is in the norming conversations. Annie’s reply suggests that Pearson understands this in terms of the Pearson-internal processes but not yet in terms of their relationships with their customers. If Pearson were to say to faculty, “Here’s what we think we know about the efficacy of this product, here’s what we don’t know yet, and here is how we are thinking about the question,” they might get a number of responses. Maybe they would get, “Oh, well here’s how I know that it’s effective with my class.” Or “The reason that you don’t have a good answer on effectiveness yet is that your rubric doesn’t provide a way to capture the educational value that your product delivers for my students.” Or “I don’t use this product because it has direct educational effectiveness. It frees me up from some grunt work so that I can conduct activities with the class that have educational impact.” Most of all, if you’re John Fallon, you really want faculty to say to their sales reps, “Huh. I never thought about the product in quite those terms, and it makes me think a little differently about how I might use it going forward. What can you tell me about the effectiveness of this other product that I’m thinking about using, at least as Pearson sees it?” And you really want your sales reps to run back to the product teams, hair on fire, saying “Quick! Tell me everything you know about the effectiveness of this product!”

    Pearson won’t get that conversation by just publishing end results of their internal analysis when they have them, which means that they have a high risk of failing to align their products with the needs and desires of their market if they think about the relationship between their framework and their customers in that way. I don’t think Pearson fully gets that yet. While the authors of The Incomplete Guide frequently invoke terms like “community” and “leaders” in the document, they generally seem to mean the community and leaders within Pearson. The company’s efforts to reach out to the academic community for feedback and participation are generally framed as an extension of their efforts rather than the very heart of them. And yet, Pearson’s brightest possible future is not as a company that designs educationally effective products, but as one that facilitates conversation and research about efficacy within the broader academic community (and in so doing is able to design products that their customers agree are effective for important educational goals as determined by meaningful measures).

    The Netflix, Spotify and, to a lesser extent, Amazon analogies all speak directly to the question of whether Pearson intends to define efficacy for educators or with educators. Netflix famously just deleted—not just removed, but deleted—all user reviews and switched from a five-star review system to a simple “thumbs-up/thumbs-down.” The subhead on the Vanity Fair article I linked to in the previous sentence is “A step closer to a wholly ‘because you watched’ world.” The implication is that Netflix trusts the algorithm more than the humans to evaluate quality. Equally famously, Apple CEO Tim Cook has highlighted the company’s decision to use human curators of music, in contrast to Spotify’s decision to rely completely on its algorithms:

    We worry about the humanity being drained out of music, about it becoming a bits-and-bytes kind of world instead of the art and craft.

    Whenever Pearson makes any mention of Netflix or Spotify—or Amazon, which is known for its recommendation engine as well—the company risks appearing to come down on the side of the algorithm over the human. Educators worry about the humanity being drained out of teaching, about it becoming a bits-and-bytes kind of world instead of the art and craft. In 2018, is that where Pearson has come down on their definition of efficacy?

    Not as far as I can tell.

    Pearson, in fact, recently made a high-profile hire away from Intel of artificial intelligence expert Milena Marinova. Here’s how she described her work in an interview with The Bookseller:

    Right now, I am working on developing human-centric AI – this means making the learning experience better for students and teachers; enabling lifelong learning through more accessible and affordable products; and building better products and solutions using new technology.

    What does “human-centric AI” mean in this context? Mr. Bozik commented on this directly in our interview. He described it as an effort to help teachers and students get a better view into their own learning. He said the company is placing a strong emphasis on early intervention and formative assessment, and providing “feedback and insights.” He also said,

    Pearson strives for respect for both the ambition and the humility of [its efficacy strategy]. The ambition is that efficacy means outcomes. Full stop. It’s the potential to help people live better lives. That is our purpose. The humility part is that it’s hard. We don’t underestimate the difficulty of it. It starts with having an empathy for teachers and learners, rather than just teaching and learning.

    For Ms. Edwards’ perspective, while I could quote her from my interview, I think you’ll get a better sense of her from her lightning talk at the Empirical Educator Project summit in February:

    Here’s the part of her talk that jumps out as relevant to the question at hand:

    How do we maximize the uniquely human attributes that educators—faculty—bring to the table, and combine them with all the productivity enhancements that come from things like data science, AI, and the improvements we’re seeing as the result of technology. All with the emphasis on helping students achieve outcomes that really matter to them.

    Ms. Edwards then goes on to talk about the resources that Pearson either has already contributed or will contribute under a Creative Commons license. These include a set of rubrics for evaluating curricular materials (or course designs) based on a set of academically accepted and empirically verified learning science principles and a set of tools for instructors that want to conduct action research. It’s these contributions, these public actions, that speak the most persuasively to Phil and me when we evaluate a company’s intent. (For more on Ms. Edwards’ views and her characterization of Pearson’s efficacy work, you can read her recent posts on LinkedIn.)

    Taken together with Pearson’s other work, such as their first publicly released efficacy reports on their products, the message conveyed by the actual work that I have seen from the company is clear: They aspire to define efficacy with educators and students. As I wrote earlier in this post, I have had opportunities to view this work from inside and out over the past five years. I honestly believe that the people working on the efficacy and product teams—Ms. Edwards’ and Mr. Bozik’s teams, respectively—buy into that goal and are actively working toward it.

    Pearson’s intent doesn’t matter if nobody believes them

    The problem is that intent of the people working at the company, level of commitment to that intent by the CEO and the small circle of people that make decisions within a company, and customer perception of intent are three different things. When I published my post the first time Mr. Hitchcock was quoted talking about Netflix, some of the people who worked on product design and efficacy at Pearson were quite angry with me. They felt that I had misrepresented Pearson’s position, because—and this is key—they also felt that Mr. Hitchcock’s comments were not representative of Pearson’s position. I replied that I wasn’t sure there was any way to determine what Pearson’s position is. Mr. Hitchcock is quite senior. He reports directly to the CEO. I could ask other executives for their own responses, but they wouldn’t be any more authoritative than Mr. Hitchcock’s. Unless I could speak directly to Mr. Fallon—which I didn’t believe would be possible at the time—I had no way of definitively determining which beliefs the elusive entity known as “Pearson” holds.

    What I did know is how academia would receive the phrase “the Netflix of Education.” I knew this because Mr. Hitchcock was not the first person to use it, as was acknowledged by a Pearson employee in a 2016 EdSurge story entitled “Why We Don’t Need a ‘Netflix for Education.’” Its toxicity was widely known inside Pearson, at least within the product and efficacy groups.

    Last month, five years into the massive effort to align the company around efficacy, the analogy shows up again, expanded upon, by the same senior executive. And this time, it’s not published in a specialty IT outlet. It’s in Forbes. Interviews at that level do not happen at big companies without being vetted. If you’re a senior executive at a company like Pearson, you go through formal media training, and you are often accompanied on interviews by a PR professional. These tend to be usually serious, orchestrated affairs. There are rules. How was it possible that this interview was approved, went through the proper channels, and went off the rails? Do Pearson executives see it as being as far off-message as I do? Is there conflict or confusion about the efficacy strategy at the highest levels of the organization that isn’t obvious to the casual observer?

    Let’s take a look at what Mr. Hitchcock actually said and then turn to the executive on-the-record reactions to it.

    Mr. Hitchcock’s comments were bad and cannot be explained away by bad editing

    Here’s the part of the article where Netflix came up:

    High: You described the vision of delivering Pearson’s education, content, and services through a single platform as creating the “Netflix of Education.” Could you talk about this long-term vision?

    Hitchcock: The intention of the message was to have the viewpoint that we needed to move to a platform type of model where we have all our products, services, and capabilities that we deliver to our customers in a single ecosystem. A great deal has been written around this model at Pearson, and that is especially relevant as our company grew through acquisitions. Ours is a diverse business that is over 170 years old and has had many different types of companies under the umbrella of Pearson for many years. Recently, education has become the primary focus, but that complexity that was acquired over those decades was inherent within the company. The way we serve our customers had been across many different brands and many different types of digital products. We sell millions of books and material on the digital side. We had to figure out how to transform that to be a consistent, high-quality branding experience and one that is comparable to the best customer experiences out there.

    Silicon Valley companies create the benchmark for the digital experience by being platform businesses. Our vision is to leverage the opportunity to transform along similar lines in terms of having a single platform globally that could deliver all our educational content and courseware. Furthermore, this would allow us to move into a more personalized experience that delivers high-quality education outcomes. It would be game-changing for not only Pearson, but for the entire industry if we could create that single platform, similar to Netflix, Spotify, and Amazon. [Emphasis added.] This platform would be highly scalable, global in nature, high-quality, and a platform that could deliver all our experiences around the world to millions of learners. We are currently working to deliver high-quality courses to students that are proven to help them learn and progress their lives through education and ultimately to their professional lives.

    In the interest of fairness, I’ll enumerate the extenuating circumstances:

    • The interviewer, not Mr. Hitchcock, brought up Netflix. And a Forbes editor, not Mr. Hitchcock, chose to feature Netflix in the headline.
    • Mr. Hitchcock seems to be making a point about the need to support a seamless, end-to-end, born-digital customer experience for students. Further, he is absolutely correct that Pearson, like all of the larger textbook publishers, grew through acquisition and have had a patchwork of many different systems that were never designed to work together. These are entirely valid and important issues for Pearson’s CTO and COO to discuss.
    • He does not take the last, most toxic step (this time) of describing Pearson as the “Netflix of education.” In fact, he seems to make a modest effort to narrow the scope of the analogy in response to the question: “The intention of the message was to have the viewpoint….”

    Is it possible that Mr. Hitchcock meant something innocuous and unobjectionable by the analogy? Yes, it is. Is it possible that he honestly tried, in his own way, to communicate the narrowness of his intent with the analogy? Absolutely. Is this a relatively minor and understandable slip that would have been fine if it had just been worded a little more clearly?

    Absolutely not.

    I reiterate: There are no circumstances under which any EdTech executive should ever make an analogy between their company or offering and Netflix or Spotify. Period. Full Stop. It is not a mistake that anyone in Mr. Hitchcock’s position should ever make. In fact, if the analogy is brought up by an interviewer, as it was in this case, the first words out of the executive’s response should be “To be clear, Pearson does not aspire to be the Netflix of education.” That goes doubly for Mr. Hitchcock because he actually said that once before and should have taken the opportunity to correct the record, and trebly because the analogy can easily be interpreted as being in direct contradiction to the company’s bet-the-farm strategy, around which it has taken them five years to arrive at their current calibration in message and in action.

    Nor can this easily be chocked up to bad editing. In fact, I question whether there was any significant editing whatsoever. This isn’t one of those tightly written articles in the reporter’s own voice with a one- or two-sentence quote sprinkled in here or there. This is an interview, with short questions followed by long answers that do not appear to be heavily edited for clarity or brevity.

    Even worse, there is a pattern to Mr. Hitchcock’s answers which leave the distinct impression that he, the CTO and COO, is directly responsible for or taking credit for the company’s strategy on teaching and learning. He deftly pairs “I believe” statements with “Pearson has done” statements. It’s very easy to read these statements as Mr. Hitchcock making a causal connection between his beliefs and Pearson’s strategy decisions, which is an impression that is only reinforced by the sycophantic fawning of the interviewer.

    Here’s an example:

    High: How is the process of experimentation around new technologies done? Specifically, as new technologies emerge, how do you encourage your team to begin to experiment and to decipher more of the specifics of how that might be leveraged?

    Hitchcock: […] Overall, I believe it is a combination of aspects. I believe it is about us looking at other ideas that other companies are implementing and seeing where we can bring these concepts together to deliver with the lasting initiatives that we are trying to put in place at Pearson. Then, as we create the single platform, it is about how we start thinking about future R&D and looking forward towards a five-year horizon. It is about seeing the ideas that are coming that we can then bolt into the platform to take us to the next level. More specifically, with AI, we recently hired a senior leader from Intel to help us to bring the entire AI area to focus in terms of what it is going to do for our business. We are building AI and machine learning capabilities and technology in other areas of our company to look at how we transform all aspects of our business using AI and machine learning.

    As I mentioned earlier, it is true that Pearson has hired a senior AI leader from intel. She reports to Mr. Bozik, not Mr. Hitchcock. As President of Global Product, Mr. Bozik’s part of the company makes product design decisions. Mr. Hitchcock’s group is in charge of the software development required to build the product. Both men report directly to Mr. Fallon. Ms. Edwards reports to Mr. Bozik. I don’t know what role Mr. Hitchcock may or may not have played in the new hire, but she definitely does not work for him. That is not at all clear from the interview.

    Here’s another example:

    High: With new technology and innovation, you underscored how the pace of change is faster than ever. Additionally, fostering learning agility for individuals, enterprises, and the people who work within companies has become an important topic especially given the fact that a skill that one has today may be rendered obsolete quickly. How do you work to stay agile and ensure that your organization is doing the same thing?

    Hitchcock: We recently released a report with the Oxford Martin School on the future skills and learning that will be required in the workplace. One of the things we focused on is the aspect of employability and equipping students with the skills that are necessary for future employment and progression within employment. One of the areas I believe is a great opportunity for Pearson is to increasingly serve education, not just through school and formal education, but with employers to transform their workforce. When I connect to our suppliers in the technology industry or my fellow CIOs, one thing that is evident is that nearly every company is faced with the challenge of re-skilling their workforce. This must be done to cope with the demands of the digital revolution that we are going through and the impact of AI, robotics, and other emerging technologies. I believe Pearson is exceptionally well positioned to assist companies in transforming the skills they need to acquire and develop.

    I started out as an engineer, and I became incorporated and then chartered through the Institute of Engineering and Technology. An engineer is no different than a surgeon or an accountant in the sense that the profession demands ongoing development certifications. I believe people want to continue the learning part. Because of this, the opportunity is there for us to become the delivery of lifelong learning for both individuals, education institutions, and government to achieve that. In terms of how we do that internally, at Pearson, there is a big focus on internal employee development.

    We started the Technology Academy shortly after I joined to try and take the technology workforce to the next level in terms of their abilities to equip them with the skills needed for our own journey around the digital change. That is something we will continue to work on and sponsor. We work very closely with institutions and universities to do that as well as our own people. It is a fascinating journey and an incredibly important one for society over the next few years. As society changes, we need to evolve to support the future digital requirements.

    One could be forgiven for assuming, based on this answer, that Pearson’s focus on workplace skills and continuing education was Mr. Hitchcock’s idea, which he arrived at on his own based on his personal biographical experience. But again, Mr. Bozik and Ms. Edwards have far more authority to set this kind of product strategy than Mr. Hitchcock does.

    The net effect of Mr. Hitchcock’s answers is a real problem for Pearson’s brand. What is a typical academic’s nightmare vision of Pearson’s worst possible future? A company whose ideas of education come from its CTO; a man whose previous job was being CIO for a phone company, who thinks that Spotify and Netflix are positive examples of the company that Pearson wants to become and isn’t afraid to say so, repeatedly, in public, despite previous criticism for having done so.

    Also? No mention of efficacy in the article. Anywhere. Not once. Not a word. Not a whisper. In an wide-ranging, open-ended interview about Pearson’s future.

    Pearson’s senior executives tried their best to clean up the mess

    As I mentioned earlier, it’s hard to figure out Pearson’s official position on…well…anything. But Pearson’s official position, or what the organization collectively “thinks,” is a critical question in this case. Given that the interview of somebody who directly reports to the CEO, and who repeated (and arguably expanded upon) an analogy for which the company had taken flak, the question is whether the root of the problem is a failure of corporate communication or a failure of alignment of senior management. This is a high-stakes question. And it’s one that can only be answered definitively by Pearson’s most senior executives.

    Given that, I wanted to get as close to a definitive answer as possible. So I did something that we don’t typically do at e-Literate. I went through proper PR channels. There are a number of reasons we don’t like gathering information this way. One is that the people we are talking to are chaperoned and constrained to hew as closely as possible to the company line. When we approach people informally, they are freer to give us information off the record that provides further validation or nuance to their on-the-record comments. We also generally try to cultivate an approach to our analysis that doesn’t depend on executives continuing to grant us access because we believe that doing so minimizes one kind of pressure on our objectivity.

    But for this particular interview, the constraints of the formal channel were helpful. I could already make pretty good inferences about what Ms. Edwards and Mr. Bozik likely think as individuals. I have gotten to know Ms. Edwards through the Empirical Educator Project and, while I haven’t had much direct exposure to Mr. Bozik, I know quite a bit about the work that has come out of his shop. Further, neither Ms. Edwards nor Mr. Bozik were responsible for—or could be individually responsible for—Mr. Hitchcock’s comments. As his peers, they were not in a position to make statements that were more definitive than his.

    By going through the formal PR channel, I could speak not just to them but through them to the aggregate entity known as “Pearson.” I wanted Mr. Bozik and Ms. Edwards to have the kind of pre-interview huddle orchestrated by PR that occurs when these media requests are made, and I wanted them to be constrained by whatever had been decided in that huddle. By doing so, they could provide answers that are official in a way that they wouldn’t be through less formal channels. Their job in this kind of an interview is to give their best, most honest version of the official company line.

    (By the way, I didn’t ask specifically for these two executives. Part of the experiment was to see who Pearson decided to send. They did not decide to send Mr. Hitchcock. I don’t know if that was purely due to scheduling conflicts or if there were other considerations.)

    Let’s be clear: 90% of my interview with Mr. Bozik and Ms. Edwards was kabuki theater. It was highly ritualized and scripted, with moments of drama that were entirely predictable. That was by design and in no way their fault. They were constrained by the script. I even gave them my basic line of questioning in advance so that the “company” could formulate “its” answers in advance. My questions consisted of four parts:

    1. Describe Pearson’s current view of what “efficacy” means, preferably covering specific examples of efficacy work
    2. Clarify Mr. Hitchcock’s role in the company specifically as it pertains to decision-making around the efficacy initiative and its integration with product design
    3. Review Mr. Hitchcock’s comments in Forbes, in the context of the first two parts of our discussion
    4. Describe Pearson’s official position on whether and in what sense it aspires to be the Netflix, Spotify, or Amazon of education

    The answers to the first two sets of questions were entirely predictable. I wanted to make sure that the official characterizations in the interview were consistent with what I thought I already knew. They were. Mr. Bozik and Ms. Edwards did their job. They faithfully described what I had come to understand as the company’s official positions on those questions. The third and fourth sets of questions were the test. In the third, Mr. Bozik did his best to provide the most charitable honest reading of Mr. Hitchcock’s comments that he could. Again, he did his job.

    When I asked him the fourth question, using more or less the same words as above, he gave the the only correct answer: “Let’s be clear: Pearson does not aspire to be the Netflix or Spotify of education.” Those were the very first words out of his mouth. Yet again, he did his job.

    At that point, I went off-script. I asked him how, given that statement, Mr. Hitchcock could use that analogy for a second time. In the entire ninety-minute interview, this was the only moment when Mr. Bozik seemed to struggle for words. As he should have. Because there is no good answer to that question.

    I had put him in an impossible situation. He was bound by the rules of the kabuki play we were in to give answers that support the company line (and his colleague). He didn’t have the option to say, “Off the record, it was a dumb thing to say.” He had only two options he could take without breaking the rules of engagement: rationalize that which I believe he knew was wrong, or flail around for the best honest answer he could give.

    Mr. Bozik chose to flail. I think better of him for it.

    But their CEO blew it

    At the end of the interview, I found out that Pearson was going to grant my request to speak directly to the company’s CEO. I honestly didn’t expect that request to be met. Pearson is at least several times as large as the next largest ed tech company that we cover. By some measures, they are an order of magnitude larger. So I was surprised and gratified that Mr. Fallon was willing to make time for the conversation.

    And it was an important step for the specific purpose at hand. Mr. Fallon is the only person at Pearson who is empowered to fully repudiate statements by the man who reports directly to him. (This isn’t true at all companies, but it tends to be true at companies as large as Pearson.) Once again, I telegraphed my line of questioning in advance, which was essentially an abbreviated version of the same approach I had used with the previous interview.

    To my surprise, Mr. Fallon jumped ahead to address Mr. Hitchcock’s comments directly. And to more or less defend them. He argued that a key constituency for Pearson is what he called the “Spotify generation,” or “Generation Z,” to which he attributed the following characteristics:

    • They would rather rent or subscribe than own
    • They have a higher expectation of the use of video
    • They probably expect to be able to engage with courses through shorter and more frequent modules
    • They have higher expectations of user experience

    The first characteristic is arguable. Within the world of curricular materials, students are strongly price-sensitive. The primary mechanism that the industry has offered students to get a lower price is rental/subscription. Students have tended to rent or subscribe to curricular materials. Does this mean that they prefer rental or subscription, or just that they are picking the least expensive option? The publishers have a financial interest in promoting rental and subscription, but the evidence of a Spotify-style preference among students seems limited.

    The other three characteristics are probably true and definitely not new. All three are trends that the sector has known about and anticipated for literally decades. Before there was the Spotify or Netflix of education, there was the iTunes playlist of education. And before that, there were re-usable learning objects. All of these points are sufficiently simple, accessible, and well understood already that an analogy to Spotify or Netflix adds nothing useful.

    Mr. Fallon was intent on getting me to admit that there is a reasonable interpretation of the analogy. Which I did. But his defense missed the point entirely. Sure, in principle, with some thought, one can find a non-offensive explanation for an analogy to Spotify or Netflix. But why would you bother to make such an analogy in the first place? And for heaven’s sake, why would you do so in public when you know that your customers are likely to find it offensive by default?

    When I pressed, Mr. Fallon said, “You have never heard or read me say that Pearson wants to be the Netflix or Spotify of education.”

    That is a carefully parsed sentence.

    In fairness, Mr. Fallon then went on to demonstrate to me that he knows exactly what the company’s official and nuanced 2018 version of efficacy is. He can cite it chapter and verse. He can refer to prior public statements that align with that vision, such as his recommendation of World Without Mind, a book about the dangers of romanticizing the power of artificial intelligence. He can speak at length and in detail about the challenges of the lack of agreed-upon frameworks for student data privacy in educational research.

    Which is not surprising. Mr. Fallon is a smart guy. Furthermore, he was the one who bet Pearson’s future on efficacy. It’s probably not an exaggeration to say that he bet his career on it as well. It would be surprising if he didn’t know this stuff inside and out. I’m not sure why he didn’t just say the analogies were bad and do not represent Pearson’s position. Maybe he was just trying to convince me that this was a non-story and not worth writing about. Had he simply led with the points about the challenges of doing efficacy right and refrained from vigorously defending the indefensible, he might have succeeded. Instead, after all of the work involved in making the interviews happen, and all of Pearson’s executive time and PR time spent dealing with me, we’re right back where we started.

    I don’t know Pearson’s official position on the analogy to Netflix or Spotify because Mr. Fallon—the only person at the company who is fully empowered to definitively lay the question to rest—chose to parse words and defend a bad interview rather than just saying, “Yeah, we don’t think about our goals that way and shouldn’t say things like that.” In a few sentences, Mr. Fallon could have closed the door on the one aspect of the Forbes article that I feel compelled to write about. But he didn’t.

    To be clear, the aspect in question isn’t that Mr. Hitchcock said something dumb in public. And it’s not that Mr. Fallon was unwilling to admit on the record that Mr. Hitchcock said something dumb. It’s that Pearson, as represented by the behavior of its CEO, does not appear to fully recognize, or at least fully acknowledge, how delicate and existential a messaging challenge they’ve created by betting their future on efficacy.

    Pearson still communicates like a roll-up, and it can’t afford to do so anymore

    So here’s the real question: Why? Why has Pearson made these unforced errors and then, having made them, compounded them?

    Once again, I can’t know the answer for certain. There often are some organizational politics behind stories like this one. That sort of thing is largely outside of our purview at e-Literate. But the company’s behavior may also be symptomatic of a more systemic problem. As I said earlier, Mr. Hitchcock was absolutely correct in saying that Pearson had a huge technology mess resulting from the fact that it is a “roll-up”—a big company that was created by buying up many small companies. He is also correct in saying that Pearson cannot compete unless they fix that mess by creating one seamless platform. (This is exactly where it is tempting to make analogies to giant internet companies and exactly where EdTech executives should resist that temptation.)

    Pearson has had an analogous problem with their organizational culture (as have all the major textbook publishers). It used to be that every editor had his or her own fiefdom, run relatively independently from the others. This doesn’t work when you all have to deliver content via a common and complex technology platform and when you have to adhere to common standards of learning design. The company has been working on changing that culture for a while, has made some progress, and probably isn’t finished yet.

    But there’s a third leg to this stool which appears to be the one that Pearson is falling over: message discipline (which is icky marketing-speak for “stick to talking about the things you think are important to communicate and don’t go off on tangents about things that are less important”). Even just a few years ago, it would have been impossible to describe the company’s official positions or priorities on much because they were a conglomerate. What priorities do textbook editors share with the publishers of the Financial Times and the people who run Penguin Books? There has always been a certain amount of laissez faire chaos in Pearson’s communications, for understandable reasons. But the company has been selling off all the parts of the business that don’t fit under the common theme of educational impact. I don’t think it would be an exaggeration to say that Pearson aspires to sell only one thing, and that thing is efficacy.

    And yet, they don’t communicate that way. They still communicate like a company with lots of products and lots of messages for lots of different audiences. They have not made this part of their transformation and, based on what what I have seen during the course of writing this story, I’m not confident that they even know that they need to make this change. I have no reason to believe this is the fault of Pearson’s marketing and communications people. It’s much more likely that the problem is the prioritization that is being given to them, or not being given to them, from the top.

    Nobody yet knows if a large education vendor can succeed by making efficacy its main product. But I can say one thing with confidence: They won’t succeed if they don’t sell it. And right now, they are not selling it. They say some things about efficacy, but they say some things about a lot of things, like Netflix, and Spotify, and Microsoft Hololens, and jobs of the future, and probably lots of other stuff that Gmail mercifully filters out for me. If they want their monumental bet to have a real chance of paying off, then they need to be talking about efficacy. Only efficacy. Always efficacy. Everything else needs to be subordinated to a very specific, meticulously crafted, always-being-tuned message about efficacy. Anything that outright clashes or distracts from that message should be mercilessly cut. Any missteps should be immediately and definitively corrected. Saying something stupid in public is a recoverable sin for a company. Failing to relentlessly focus on communicating the complex and somewhat controversial value proposition, upon which you have bet the entire future of your company, to an academic audience that is not terribly inclined to listen to or trust your messages in the first place? That may not be a recoverable mistake.

  • Good Enough vs. Better Enough: The Macmillan Example

    Good Enough vs. Better Enough: The Macmillan Example

    In my recent post on Cengage Unlimited, I made a brief mention of the battle shaping up in the curricular materials world between “good enough” and “better enough.” I argued that Cengage is coming down on the “good enough” side by emphasizing all-you-can-eat pricing.

    The distinction I’m trying to make between two strategies is a little tricky. I’m not arguing that Cengage, for example, thinks that their products aren’t great or that they think all anybody needs is the cheapest PDF possible. And on the other hand, “better enough” no longer means better editing or better production values, which is the way that textbook publishers used to position themselves against OER (and still do sometimes, although that reflex is beginning to fade). Rather, it’s about improving student outcomes.

    To borrow a phrase from David Wiley, the fight boils down to standard deviations per dollar. ((“Standard deviation” is just statistics geek speak for a measure of difference—in this case, improvement—from the norm.)) This formulation boils the battle down to a fraction. In the numerator, we have impact. In the denominator, we have cost. David likes to say that it’s easier to change the denominator, i.e., reduce cost, than it is to change the numerator, i.e., improve student outcomes. One of the reasons this is true is that putting a different product in a class usually doesn’t have a big impact unless the instructor’s teaching practices also change to take better advantage of the product’s features. Or, if you prefer a formulation that emphasizes the teaching over the tools (which I do), digital courseware tends to have the most impact in classrooms where it supports the chosen pedagogical approach of the instructor.

    For the incumbents, neither the numerator nor the denominator is particularly easy to change. In my last two posts, I wrote about the major investments—and risks—that Cengage took on to deliver their products at a better price point and still make their business model work (they hope).

    But that may be a cake walk for the publishers compared with the challenge of changing the numerator. In my original post about Pearson’s efficacy strategy, I explored these challenges at length. I have chosen to quote a hefty excerpt here because none of these problems have gone away:

    Let’s think some more about the analogy to efficacy in health care. Suppose Pfizer declared that they were going to define the standards by which efficacy in medicine would be measured. They would conduct internal research, cross-reference it with external research, come up with a rating system for the research, and define what it means for medicines to be effective. They would then apply those standards to their own medicines. And, after all is said and done, they would share their system with physicians and university researchers in the hopes that the medical community might be reassured about the quality of Pfizer’s products and maybe even contribute some ideas to the framework around the edges. How confident would we be that what Pfizer delivers would consistently be in the objective best interest of improving health?…

    If Pearson were to say to faculty, “Here’s what we think we know about the efficacy of this product, here’s what we don’t know yet, and here is how we are thinking about the question,” they might get a number of responses. Maybe they would get, “Oh, well here’s how I know that it’s effective with my class.” Or “The reason that you don’t have a good answer on effectiveness yet is that your rubric doesn’t provide a way to capture the educational value that your product delivers for my students.” Or “I don’t use this product because it has direct educational effectiveness. It frees me up from some grunt work so that I can conduct activities with the class that have educational impact.” Most of all, if you’re [Pearson CEO] John Fallon, you really want faculty to say to their sales reps, “Huh. I never thought about the product in quite those terms, and it makes me think a little differently about how I might use it going forward. What can you tell me about the effectiveness of this other product that I’m thinking about using, at least as Pearson sees it?” And you really want your sales reps to run back to the product teams, hair on fire, saying “Quick! Tell me everything you know about the effectiveness of this product!”

    Pearson won’t get that conversation by just publishing end results of their internal analysis when they have them, which means that they have a high risk of failing to align their products with the needs and desires of their market if they think about the relationship between their framework and their customers in that way….

    There are a number of reasons why this part of the transformation will be at least as difficult as the part that Pearson is undertaking now. First, it is far from clear that the company has the trust of the academic community that would be necessary for them to take such a role. That would have to be built, in some cases from the ground (or even the basement) up. Pearson does have real strengths that are known within certain segments of the academic community—in data science, for example—but this does not transfer to a general reputation. Second (and relatedly), unlike the medical research community, the educational research community is still nascent and fragmented. Finding non-paternalistic but effective ways to bring that community together and facilitate useful conversations will be difficult to say the least. These two challenges are outside the company’s sphere of control, which means that Pearson will have to develop new ways to think about how to build their relationships with the broader educational community.

    Internally, changing the way they think about answering the questions that the framework asks them will entail as much subtle, difficult, and pervasive re-engineering of the corporate reflexes and business processes as the work being undertaken now….  [A]ll textbook companies that have been around for a while are wired for a particular relationship with faculty that is at the heart of how they design, produce, and sell their products. Their editors have gone through decades of tuning the way they think and work to this process, and so have their customers. When Pearson layers a discussion of efficacy onto these business processes, a tension is created between the old and new ways of doing things. Suddenly, authors and customers don’t necessarily get what they want from their products just because they asked for them. There are potentially conflicting criteria. The framework itself provides nothing to help resolve this tension. At best, it potentially scaffolds a norming conversation. But a product management methodology that can combine knowledge about efficacy, user desires, and usability requires more tools than that. And that problem is even worse in some ways now that product teams have multiple specialized roles. The editor, author, adopting teacher, instructional designer, cognitive science researcher, psychometrician, data scientist, and UX engineer may all work together to develop a unified vision for a product, but more often than not they are like the blind man and the elephant. Agreeing in principle on what attributes an effective product might have is not at all the same as being able to design a product to be effective, where “effective” is shared notion between the company and the customers. ((Believe it or not, that is a short excerpt as measured as a percentage of the total word count of the post.))

    Publishers that want to improve the numerator will have to completely rewire the ways that they work, both internally and externally. They need to rethink their product design process from the ground up while simultaneously completely resetting their relationships with their customers.

    That post was published on December 31st, 2013. As we enter 2018, we are beginning to see examples of what such efforts might look like. For today’s example, I’m going to draw on recent work by Macmillan.

    Resetting the Conversation

    Before I get into the details, a little more disclosure than usual is called for here. I am a paid member of Macmillan’s Learning Impact Research Advisory Council (IRAC). As such, I was paid to provide input on the paper I’m about to write about as well as the underlying research processes that the paper describes. I was not paid to write this post about the paper. Or rather, I was paid to write something about it, but I was asked to write one page—one page!—of private feedback on the paper. I asked if I could write my feedback as a public blog post of unspecified length. The folks at Macmillan agreed.

    The paper is called Unpacking the Black Box of Efficacy: A framework for evaluating the effectiveness and researching the impact of digital learning tools. Registration is required.

    First piece of feedback for Macmillan: If you really want to foster a new dialog with academics, don’t start it by requiring them to give you their email addresses just to read your paper.

    But the approach outlined in the paper is another matter. Recall that in the Pearson post quoted above, I advised the company to approach customers with something like the following proposition:

    Here’s what we think we know about the efficacy of this product, here’s what we don’t know yet, and here is how we are thinking about the question.

    That is essentially what Macmillan’s paper attempts to do. It starts with an inventory, in plain English, some common educational research methods, how they work, and what their strengths and weaknesses are. The section on randomized controlled trials (RCTs) alone is worth the price of admission, given how often it is simplistically held up as the “gold standard” in research. Any thoughtful educator reading the description of the process will immediately think, “Hey, that’s…problematic in education.”

    Even better, Macmillan was able to accomplish that with one page of text and one picture. They will need to be this incisive on a consistent basis if they are going to reach their intended audience.

    Next, the paper describes their product development lifecycle. Again, there is a good balance here of clarity and brevity. The first stage of that lifecycle is called “Co-design & Learning Research.” While publishers have pretty much always started their product design process with input from customers, I wouldn’t call the historic process “co-design.” Rather, it was typically an author/editor collaboration with some limited and focused customer input. More recently, publishers have developed all kinds of hybrid processes. But Macmillan at least claims to be starting with a clean sheet of paper. They are certainly not the only publisher to do this, but from a communication perspective, framing educational product design as a combination of co-design with customers and structured but comprehensible research is a good move.

    Speaking of which, the third section maps the various research methods described in the first section to the product design process in the second. There’s even a development timeline. The net effect is that educators (and students) have a clear and concise document explaining how Macmillan products are developed, how their potential learning impact is tested, and just how much it’s fair to say that the company knows about that impact at any stage in the development lifecycle.

    While I am by no means claiming credit, this paper reads as if it could have been written as a direct response to my critique of Pearson’s first iteration of efficacy.

    So yeah. I like it.

    Good Enough for What?

    You didn’t think I’d let them off that easily, did you?

    Remember waaay back, all the way at the beginning of the post, when I made the point that learning outcomes are hard to improve with curricular materials partly because their impact depends on what humans in the classroom do with them? That problem still looms, and Macmillan’s paper barely touches it.

    When I talk to students at length about the curricular materials that their instructors assign, their top complaint isn’t price. Don’t get me wrong; they hate the prices. But what they really hate is being told to buy a $200 book that the instructor barely mentions, let alone integrates into the class on a programmatic basis.

    This is what “better enough” is competing against. I always thought it was funny that textbook publishers refer to everything outside the book as “ancillaries,” because many instructors tend to see the categories as reversed. The book is ancillary. It’s not central to the learning that happens in the classroom. Before Macmillan, or any other publisher, can sell products based on the value proposition of “efficacy” or “learning impact” or “learning outcomes”, instructors must first come to believe that these three propositions are true:

    1. Instructors are responsible for learning how to improve their students’ learning outcomes by improving their teaching craft.
    2. Improving their teaching craft includes learning to employ research-validated practices.
    3. Macmillan’s products support and enable research-validated practices effectively enough that they can make the credible case for having more than “ancillary” value.

    This paper makes a good start—as good a start as any short paper can make—on the third proposition. The first proposition isn’t fair to lay at the publishers’ feet; it’s more driven by the incentives and culture of academia. It’s a problem, but not one that Macmillan or its peers can do much about directly. The second proposition is where the vendors, including but not limited to Macmillan, need to figure out how to do more.

    To be fair, the paper nibbles around the edges of this problem. The educators and product developers need to develop shared goals. That’s what a co-design process is for. Educators and product developers also need to develop a shared sense of proof that the goals are being met better by one method than another. A lot of the paper develops the basis for a conversation around this.

    But left implicit is the argument that education should be empirical and that empiricism needs to be formalized at least some of the time. There should be theories of learning impact and rules for what counts as evidence that supports or disproves those theories. This needs to apply not just to curricular materials design but for what happens in the classroom.

    It’s probably too much to expect this paper, as focused as it is, to open up this Pandora’s Box. This paper is, in part, a trust-building exercise, and Macmillan needs to build trust before they can fully own up to the fact that incorporating curricular materials that meaningfully improve learning outcomes usually entails a course redesign. But that’s where both Macmillan and the industry need to get to if they want to be able to sell more heavily researched and designed products at a higher price point.

    “Good enough” means “good enough for the way I use curricular materials in my classroom.” “Better enough” means “better enough that I’m convinced I should change the way I teach.” Macmillan has written a really good paper on the standards of proof they propose to live up to and how they propose to live up to them. But they also have to convince their customers to agree to live up to those same standards in their own teaching.

  • Pearson Releases a Significant Learning Design Aid

    Last week, Pearson announced the release of some resources they have created around science-based learning design. (Full disclosure: Pearson is a client of our consulting company, although we have not consulted with them on the subject of this post.) The resources in the release include the following:

    • A 102-page rubric the company has started using for evaulating curricular products based on their effective use of learning design principles—released under a Creative Commons license
    • A white paper describing how they are beginning to apply these principles to their own product designs
    • A few links to descriptions of early examples of these principles as they have been applied in released products.
    • A blog post providing some more background and perspective by David Porcaro, Pearson’s Director of Learning Design, whose team is behind this effort.

    In the short term, you should think of this publication as the company’s effort to “show their work,” in the classroom sense of that phrase. They want you to know more about the process they go through to incorporate research-backed learning design principles into their products. They hope that doing so will increase your confidence in the value of those products. In the medium term, they also aspire to make the framework they are developing for themselves also a useful tool for the academic community, particularly as the company refines and expands on this early release of the materials.

    In my view, the work itself is a significant contribution. It also is a positive indicator about Pearson’s future direction as a participant in and influencer of that community, although how strong an indicator is a much harder question to evaluate. And it gives us another clue about the co-evolution of educational institutions and ed tech vendors that we are likely to see over the next years and decades. In this post, I’m going to evaluate each of these aspects in turn.

    (more…)

  • Pearson, Efficacy, and Research

    A while back, I mentioned that MindWires, the consulting company that Phil and I run, had been hired by Pearson in response to a post I wrote a while back expressing concerns about the possibility of the company trying to define “efficacy” in education for educators (or to them) rather than with them. The heart of the engagement was us facilitating conversations with different groups of educators about how they think about learning outcomes—how they define them, how they know whether students are achieving them, how the institution does or doesn’t support achieving them, and so on. As a rule, we don’t blog about our consulting work here on e-Literate. But since we think these conversations have broader implications for education, we asked for and received permission to blog about what we learn under the following conditions:

    • The blogging is not part of the paid engagement. We are not obliged to blog about anything in particular or, for that matter, to blog at all.
    • Pearson has no editorial input or prior review of anything we write.
    • If we write about specific schools or academics who participated in the discussions, we will seek their permission before blogging about them.

    I honestly wasn’t sure what, if anything, would come out of these conversations that would be worth blogging about. But we got some interesting feedback. It seems to me that the aspect I’d like to cover in this post has implications not only for Pearson, and not only for ed tech vendors in general, but for open education and maybe for the future of education in general. It certainly is relevant to my recent post about why the LMS is the way it is and the follow-up post about fostering better campus conversations. It’s about the role of research in educational product design. It’s also about the relationship of faculty to the scholarship of teaching.

    (more…)

  • Pearson’s Efficacy Listening Tour

    Back around New Year, Michael wrote a post examining Pearson’s efficacy initiative and calling on the company to engage in active discussions with various communities within higher education about defining “efficacy” with educators rather than for educators. It turns out that post got a fair bit of attention within the company. It was circulated in a company-wide email from CEO John Fallon, and the blog post and all the comments were required reading for portions of the company leadership. After a series of discussions with the company, we, through our consulting company, have been hired by Pearson to facilitate a few of these conversations. We also asked for and received permission to blog about them. Since this is an exception to our rule that we don’t blog about our paid engagements, we want to tell you a little more about the engagement, our rationale for blogging about it, and the ground rules.

    (more…)

  • Head in the Oven, Feet in the Freezer

    Some days, the internet gods are kind. On April 9th, I wrote,

    We want talking about educational efficacy to be like talking about the efficacy of Advil for treating arthritis. But it’s closer to talking about the efficacy of various chemotherapy drugs for treating a particular cancer. And we’re really really bad at talking about that kind of efficacy. I think we have our work cut out for us if we really want to be able to talk intelligently and intelligibly about the effectiveness of any particular educational intervention.

    On the very same day, the estimable Larry Cuban blogged,

    So it is hardly surprising, then, that many others, including myself, have been skeptical of the popular idea that evidence-based policymaking and evidence-based instruction can drive teaching practice. Those doubts have grown larger when one notes what has occurred in clinical medicine with its frequent U-turns in evidence-based “best practices.” Consider, for example, how new studies have often reversed prior “evidence-based” medical procedures. *Hormone therapy for post-menopausal women to reduce heart attacks wasfound to be more harmful than no intervention at all. *Getting a PSA test to determine whether the prostate gland showed signs of cancer for men over the age of 50 was “best practice” until 2012 when advisory panels of doctors recommended that no one under 55 should be tested and those older  might be tested if they had family histories of prostate cancer. And then there are new studies that recommend women to have annual mammograms, not at age  50 as recommended for decades, but at age 40. Or research syntheses (sometimes called “meta-analyses”) that showed anti-depressant pills worked no better than placebos. These large studies done with randomized clinical trials–the current gold standard for producing evidence-based medical practice–have, over time, produced reversals in practice. Such turnarounds, when popularized in the press (although media attention does not mean that practitioners actually change what they do with patients) often diminished faith in medical research leaving most of us–and I include myself–stuck as to which healthy practices we should continue and which we should drop. Should I, for example, eat butter or margarine to prevent a heart attack? In the 1980s, the answer was: Don’t eat butter, cheese, beef, and similar high-saturated fat products. Yet a recent meta-analysis of those and subsequent studies reached an opposite conclusion. Figuring out what to do is hard because I, as a researcher, teacher, and person who wants to maintain good health has to sort out what studies say and  how those studies were done from what the media report, and then how all of that applies to me. Should I take a PSA test? Should I switch from margarine to butter?

    He put it much better than I did. While the gains in overall modern medicine have been amazing, anybody who has had even a moderately complex health issue (like back pain, for example) has had the frustrating experience of having a billion tests, being passed from specialist to specialist, and getting no clear answers. ((But I’m not bitter.)) More on this point later. Larry’s next post—actually a guest post by Francis Schrag—is an imaginary argument between an evidence-based education proponent and a skeptic. I won’t quote it here, but it is well worth reading in full. My own position is somewhere between the proponent and the skeptic, though leaning more in the direction of the proponent. I don’t think we can measure everything that’s important about education, and it’s very clear that pretending that we can has caused serious damage to our educational system. But that doesn’t mean I think we should abandon all attempts to formulate a science of education. For me, it’s all about literacy. I want to give teachers and students skills to interpret the evidence for themselves and then empower them to use their own judgment. To that end, let’s look at the other half of Larry’s April 9 post, the title of which is “What’s The Evidence on School Devices and Software Improving Student Learning?” (more…)

  • Efficacy, Adaptive Learning, and the Flipped Classroom, Part II

    In my last post, I described positive but mixed results of an effort by MSU’s psychology department to flip and blend their classroom:

    • On the 30-item comprehensive exam, students in the redesigned sections performed significantly better (84% improvement) compared to the traditional comparison group (54% improvement).
    • Students in the redesigned course demonstrated significantly more improvement from pre to post on the 50-item comprehensive exam (62% improvement) compared to the traditional sections (37% improvement).
    • Attendance improved substantially in the redesigned section. (Fall 2011 traditional mean percent attendance = 75% versus fall 2012 redesign mean percent attendance = 83%)
    • They did not get a statistically significant improvement in the number of failures and withdrawals, which was one of the main goals of the redesign, although they note that “it does appear that the distribution of A’s, B’s, and C’s shifted such that in the redesign, there were more A’s and B’s and fewer C’s compared to the traditional course.”
    • In terms of cost reduction, while they fell short of their 17.8% goal, they did achieve a 10% drop in the cost of the course….

    It’s also worth noting that MSU expected to increase enrollment by 72 students annually but actually saw a decline of enrollment by 126 students, which impacted their ability to deliver decreased costs to the institution.

    Those numbers were based on the NCAT report that was written up after the first semester of the redesigned course. But that wasn’t the whole story. It turns out that, after several semesters of offering the course, MSU was able to improve their DFW numbers after all:

    MSU DFWThat’s a fairly substantial reduction. In addition, their enrollment numbers have returned to roughly what they were pre-redesign (although they haven’t yet achieved the enrollment increases they originally hoped for).

    When I asked Danae Hudson, one of the leads on the project, why she thought it took time to see these results, here’s what she had to say:

    I do think there is a period of time (about a full year) where students (and other faculty) are getting used to a redesigned course. In that first year, there are a few things going on 1) students/and other faculty are hearing about “a fancy new course” – this makes some people skeptical, especially if that message is coming from administration; 2) students realize that there are now a much higher set of expectations and requirements, and have all of their friends saying “I didn’t have to do any of that!” — this makes them bitter; 3) during that first year, you are still working out some technological glitches and fine tuning the course. We have always been very open with our students about the process of redesign and letting them know we value their feedback. There is a risk to that approach though, in that it gives students a license to really complain, with the assumption that the faculty team “doesn’t know what they are doing”. So, we dealt with that, and I would probably do it again, because I do really value the input from students.

    I feel that we have now reached a point (2 years in) where most students at MSU don’t remember the course taught any other way and now the conversations are more about “what a cool course it is etc”.

    Finally, one other thought regarding the slight drop in enrollment we had. While I certainly think a “new blended course” may have scared some students away that first year, the other thing that happened was there were some scheduling issues that I didn’t initially think about. For example, in the Fall of 2012 we had 5 sections and in an attempt to make them very consistent and minimize missed classes due to holidays, we scheduled all sections on either a Tuesday or a Wednesday. I didn’t think about how that lack of flexibility could impact enrollment (which I think it did). So now, we are careful to offer sections (Monday through Thursday) and in morning and afternoon.

    To sum up, she thinks there were three main factors: (1) it took time to get the design right and the technology working optimally; (2) there was a shift in cultural expectations on campus that took several semesters; and (3) there was some noise in the data due to scheduling glitches.

    There are a number of lessons one could draw from this story, but from the perspective of educational efficacy, I think it underlines how little the headlines (or advertisements) we get really tell us, particularly about components of a larger educational intervention. We could have read, “Pearson’s MyPsychLabs Course Substantially Increased Students Knowledge, Study Shows.” That would have been true, but we have little idea how much improvement there would have been had the course not been fairly radically redesigned at the same time. We also could have read, “Pearson’s MyPsychLabs Course Did Not Improve Pass and Completion Rates, Study Shows.” That would have been true, but it would have told us nothing about the substantial gains over the semesters following the study. We want talking about educational efficacy to be like talking about the efficacy of Advil for treating arthritis. But it’s closer to talking about the efficacy of various chemotherapy drugs for treating a particular cancer. And we’re really really bad at talking about that kind of efficacy. I think we have our work cut out for us if we really want to be able to talk intelligently and intelligibly about the effectiveness of any particular educational intervention.