e-Literate

Present is Prologue

Author: Michael Feldstein

  • Short eLiterate Course on Analytics and Adaptive Learning

    A while back, we released a playlist of animated explainer videos that could be used to introduce faculty to personalized learning, which we argue is a family of pedagogical approaches rather than a set of technologies: ((The videos featured in this post were funded by grants from the Bill & Melinda Gates Foundation))

    We now have a new set of six explainer videos that represent a short introductory course to the concepts behind adaptive learning and learning analytics, which are tools that are useful for personalized learning pedagogical approaches:

    We still lead with the pedagogy, as always. For example, we have one explainer that’s focused on continuous formative assessment. Here’s a taste of it:

    And here’s a bit about memory decay from our episode on memory-based adaptive learning:

    We also break down the technologies, connecting them to the pedagogical concepts. For example, we explain that there are actually multiple kinds of adaptive learning products that work in different ways:

    We give individual explanations of memory-based adaptive learning…

    …and skill-based adaptive learning.

    And, as we always try to do, we give educators tips for how to think about these tools in ways that help them evaluate the usefulness of various technologies and products for their particular teaching contexts:

    We hope you find these useful. Let us know what you think.

     

     

  • Empirical Educator Project: A Quick Update

    As you may recall, we recently announced that eLiterate is launching a new initiative called the Empirical Educator Project (EEP). From the announcement post:

    We believe one major barrier to progress is social. Work is being done, but it is fragmented. New approaches are being pioneered, but they aren’t shared, tested in different educational contexts, or framed in a way that makes them easy for other individuals and organizations to adopt them. An innovation often doesn’t make it outside the classroom or institution where it was created. When it does, it usually travels to peer groups—other members of the same department, other colleagues in the same discipline, or other universities of the same type.

    Colleges that have educational challenges they need to solve often lack researchers to help them, while research universities often have researchers who are looking for good problems to solve. Despite the obvious collaboration opportunity that this resource imbalance presents, classroom educators aren’t aware of relevant educational researchers, and researchers rarely make their work product accessible to and adoptable by classroom educators. And if those researchers happen to work for vendors rather than universities, the problem is even worse. Vendors have no good way to convince the average academic that their research is valid, even when they have real and significant contributions to make. Academics who want to engage with vendors on research projects have no clear avenues to do so.

    At e-Literate, we are in the rare and lucky position to be able to see into and across all these various silos. Given that we are seeing both increasing motivation and increasing activity in pockets across the higher education sector, we want to focus on increasing propagation and collaborative refinement of this work by lowering the barriers and increasing the opportunities and incentives for collaborative work. But part we believe that creating incentives for action is not the biggest challenge. Many people and organizations already have enough incentive that they are taking actions now, on their own, with whatever resources they can muster. In many cases, we will just need to create enough additional incentive to make collaboration worth the effort over and above whatever energy they are already putting into their projects.

    We had our summit at Stanford about 2 weeks ago with 50 participants from a pretty wide range of institutions and roles. (You can see the attendee list on the project web page.) We’re still processing everything that we’re learning from that gathering and the follow-up, but here are two big lessons that I’m taking away from it personally:

    1. Our hypothesis that gaps in our networks are preventing us from seeing the progress that is being made and collaborating to accelerate that progress was strongly supported by the early outcomes of that meeting. Participants seem thrilled to hear about work they didn’t know about from people they’d never met at institutions they never encounter. The number of collaborative project ideas coming out of the summit was far, far higher than we expected. We will see how many of them come to fruition, and how many of those produce real impact, but we are off to a running start.
    2. The work of Carnegie Mellon University anthropologist Lauren Herckis, which we have highlighted here twice, really resonated with the group. It turns out that telling faculty they should adopt effective teaching practices isn’t an effective teaching practice. With the caveat that she has conducted one (extensive) study at one institution, her work suggests that we tend to dramatically underestimate the degree to which most faculty care about teaching while dramatically overestimating the degree to which they will be persuaded by academic research that they should change their teaching practices. Faculty have all kinds of reasons why they resist pedagogical change, some of which are very logical and some of which are more intuitive or emotional. If we don’t engage with these cultural blockers head-on, then no amount of research, promotion, technology investment, or grant money will move the needle on student success.

    Put these two observations together, and it looks like EEP will be most helpful if it promotes collaboration around projects that move campus culture in the direction of embracing an open-minded and experimental approach to educating, in which increasing student success is viewed as a collective endeavor in which we can advance the state of the art using the same intellectual tools of inquiry, debate, and peer review that academics apply all the time in their disciplinary work.

    We will have much more to share about both the summit and our plans for EEP going forward within a couple of weeks. In the meantime, we have encouraged the participants to share their own views on the project as they see fit through blogs, articles, or other venues.

    Stay tuned.

  • Announcing the Empirical Educator Project

    Phil and I are excited to share the news that e-Literate is launching a new initiative called the Empirical Educator Project (EEP). This is something we’ve been working for about half a year now with an increasing number of collaborators. We are far enough along now that we feel we have something to share.

    The short version is that we are starting a year-long sponsor-funded experiment intended to test the proposition that we can accelerate improvements in educational access and outcomes by fostering collaborations between organizations that have either similar or complementary needs and goals but don’t know about each other or have trouble figuring out how to work together. If the experiment is successful, then we will continue and expand the effort.

    The details matter. I’d like to outline the project for you here and address some of the common questions we’ve gotten as we’ve described the project to various friends in the education sector.

    What’s the Big Problem to be Solved?

    Higher education is starting to move from talking about improving student outcomes to trying to improve student outcomes. We’d like to focus some of that new momentum in ways that get us to the part where we actually start improving student outcomes a little faster

    There has always been a philosophical commitment to the “education” part of higher education in America. But compared to the commitment to disciplinary scholarship, there has not been anything close to the same level of collective focus and institutional support for excellence. In recent years, a number of different factors have created new motivations to start addressing this imbalance:

    • The rising awareness among Americans about growing economic inequality has caused more people to ask how well our current educational system is serving those that are being left behind and to experiment with various alternative educational strategies.
    • At the same time, good jobs and the expertise that they require are changing so quickly that even people who graduate with what is traditionally considered a “good” degree lack skills that employers need, creating a problem for both employers and job seekers.
    • The cost of a college education, which has been rising quickly for some time, has motivated an increasing focus on the economic value of a degree by both students and policymakers, which has raised the profile of the first two problems.
    • Elite institutions, which have been able to resist the economic forces for the most part, are starting to realize that today’s students expect more engagement and better use of technology than they get from even the best traditional lecture classes.
    • Meanwhile, vendors of educational products ranging from textbooks to learning management systems are starting to see their products commoditize. Nothing less than direct improvement of student outcomes will justify the prices they want to charge. Companies in new product categories that are still trying to break through—like learning analytics—are also dependent on their ability to convince prospective customers that they can directly impact student learning.

    As a result of these and other forces, we have seen an increase in activity around improving educational access and outcomes. New faces are showing up at conferences, and many of these people bring a sense of urgency with them. We see more symposia, more pilot projects, and more grants focused on “efficacy” or “outcomes.” Vendors are hiring researchers, publishing peer-reviewed papers, and risking the appearance of presumptuousness by joining in academic conversations about effective teaching practices.

    And yet, for all this increased activity, we don’t have much to show for it yet in terms of results. This is going to be a tough nut to crack. Colleges and universities do not have the institutional knowledge, practices, and support structures in place to drive this transformation. Faculty do not have the training, incentives, and support they need. Vendors have neither the credibility they need or the means for establishing it in order to make contributions.

    We believe this nascent movement toward a more systemic effort at educational excellence will take at least a generation to come to fruition—and that’s if all goes well. As the saying goes, to change everything, you need everyone.

    EEP is our attempt to make a meaningful contribution.

    What’s the Big Idea?

    We believe one major barrier to progress is social. Work is being done, but it is fragmented. New approaches are being pioneered, but they aren’t shared, tested in different educational contexts, or framed in a way that makes them easy for other individuals and organizations to adopt them. An innovation often doesn’t make it outside the classroom or institution where it was created. When it does, it usually travels to peer groups—other members of the same department, other colleagues in the same discipline, or other universities of the same type.

    Colleges that have educational challenges they need to solve often lack researchers to help them, while research universities often have researchers who are looking for good problems to solve. Despite the obvious collaboration opportunity that this resource imbalance presents, classroom educators aren’t aware of relevant educational researchers, and researchers rarely make their work product accessible to and adoptable by classroom educators. And if those researchers happen to work for vendors rather than universities, the problem is even worse. Vendors have no good way to convince the average academic that their research is valid, even when they have real and significant contributions to make. Academics who want to engage with vendors on research projects have no clear avenues to do so.

    At e-Literate, we are in the rare and lucky position to be able to see into and across all these various silos. Given that we are seeing both increasing motivation and increasing activity in pockets across the higher education sector, we want to focus on increasing propagation and collaborative refinement of this work by lowering the barriers and increasing the opportunities and incentives for collaborative work. But part we believe that creating incentives for action is not the biggest challenge. Many people and organizations already have enough incentive that they are taking actions now, on their own, with whatever resources they can muster. In many cases, we will just need to create enough additional incentive to make collaboration worth the effort over and above whatever energy they are already putting into their projects.

    We believe the bigger challenge will be in the matchmaking.

    Why “Empirical Educator Project”?

    The first task in fostering collaboration across a diverse group is framing the problem broadly enough to be relevant to everyone yet specific enough to clarify which sorts of problems are in or out of scope for collective action. In this case, we want anyone who is directly responsible for helping students succeed to think of themselves as educators in some sense, and for all educators to think of themselves as being empirical in some sense. The details will vary from educator to educator and context to context. Our project title is intended to define these unifying goals while allowing for flexibility within that shared purpose.

    We chose “empirical” rather than “evidence-based,” “efficacy,” “learning science-based,” or any other more popular term because people and organizations differ in both their developmental arcs and the specific kinds of empiricism that make the most sense for their educational contexts. By “empirical,” we simply mean creating some sort of a feedback loop, which can be boiled down to a basic four-step method. First, start with an idea of how you can improve results from some existing baseline. Second, try your idea. Third, examine your results and compare them to the baseline. And finally, adjust your beliefs and actions accordingly.

    This method can be applied across a wide range of educational processes. For example, (1) design a lesson that you believe will improve the students’ performance relative to how they do with the current lesson, (2) try the lesson, (3) compare the students’ performance to your baseline, and (4) decide what, if anything, you will do differently next time. Or (1) design a student orientation strategy that you think will improve first-year retention over previous years, (2) try the strategy, (3) check the retention numbers, and (4) decide what to do next year. Or (1) design a new tenure and promotion system that you think will incentivize more effective teaching by some measure you’ve identified, (2) try the new incentive system, (3) see if your effectiveness measurement changes, and (4) adjust accordingly.

    Feedback loops can be more or less formal, frequent, and elaborate. Some classes have no feedback loops at all. It’s not uncommon for the only assessment in a law school course to be the final exam. Students have no tools for testing and improving their study strategies throughout the course. In cases where there are no feedback loops, a goal might be to add some. In other cases, goals might be to improve the frequency or quality of the feedback.

    There can be many different kinds of feedback loops, as scholars know from their research work. A debate about experimental design, or about the best method for analyzing literature, is a debate about the best way to construct a feedback loop. It is expected that scholarly methods will and should vary dramatically from discipline to discipline, or even between different research problems within a discipline. Likewise, pedagogical approaches and methods ranging from personalized learning to competency-based education to social constructivism all have their own methods of constructing educational feedback loops. We take no position on which is the best kind. We simply want to foster a culture in which educators think of themselves as empirical in the sense that they continually develop and refine feedback loops for themselves and their students.

    That brings us to “educator.” We believe that anybody who has some sort of direct responsibility for students’ education should think of themselves as educators. They don’t have to think of themselves as solely or even primarily educators; they can also think of themselves as scholars, researchers, deans, academic technology administrators, instructional designers, product managers, Mets fans, Episcopalians, and so on. But somebody who teaches a class is, among other things, an educator. This may seem obvious, but we have found that it isn’t to all academics. Somebody who sets tenure and promotion policies for teaching faculty is also an educator, as is someone who designs curricular materials or educational technology platforms. To the degree that all these people think of themselves as educators, it will be easier for them to find common ground. We expect the people who participate in the Empirical Educator Project to do so as educators.

    Finally, “project.” We intend to practice what we preach by creating our own feedback loop. EEP is a (roughly) one-year experiment. If you’ve read this far, then you know our basic hypothesis. On February 28th, we will be bringing together about 50 people with different roles from a wide range of organizations to peer review that hypothesis, workshop our experimental design, act as an initial cohort to identify a handful collaborative projects that can serve as models, and help us figure out how to open up the experiment to broader community participation.

    In the year that follows, we will foster collaborative projects among pairs or small groups of EEP participants. The scope is wide open, as long as the projects create or improve educational feedback loops at the participating colleges and universities. In fact, we are specifically looking to find out which sorts of projects participants are sufficiently motivated to take on without external incentives like big grants.

    At the end of the year, we will evaluate whether or not the level of project activity and quality of outcomes support the EEP hypothesis. If our feedback loop tells us we are onto something, we will adjust the design and expand the project. If it doesn’t, we will publish the results of the experiment and think about designing a different one.

    How, Exactly, Will This Work?

    We will finalize our detailed plan based on the peer review from the summit at the end of this month. But the our core work after the summit will be matchmaking. We will help organizations that are doing similar projects find each other and compare lessons learned. We will help researchers and educators in need of research support find each other. We will help vendors and schools figure out how to collaborate with each other.

    Some of this will be directly facilitated by us and our staff, occasionally supported by small grants out of the pool of sponsorship money. A lot of it will happen in an online community space. We have already set up a private conversation area for the summit group. Sometime shortly after the summit—it might take us a little while to figure out the best way to do it—we will open up that space as broadly as is practical to the academic community, possibly in several iterations. If you are reading this and you work at a college or university, we want to make it possible for you to participate.

    Vendors and other non-university actors (like foundations, for example) can participate if they are accepted as sponsors.

    You Keep Mentioning Sponsors…

    Yes. We have sponsors. We don’t need a lot of money to make this pilot work, but we need some. More importantly, we need vendors who have skin in the game.

    For the first round of sponsors, we reached out only to a subset of the people that we personally know and trust who are also in positions to foster project participation and support from their companies. We limited the number of sponsors we invited in advance of the summit, in part because we have limited space. Mitchell Stevens and Candace Thille kindly offered to lend us the use of the Lytics lab at Stanford, which is a great space but not a huge one. We had to ensure that we have the right mix of people, that the sponsors don’t outnumber the academics, and so on.

    We also made a conscious decision not to privilege sponsors based on their size and ability to pay larger fees than their smaller competitors. There are no gold, silver, and bronze sponsors. We created three standard fee levels that we assign to sponsors based on their valuation. Billion-dollar companies paying the highest fee and small start-ups paying the lowest fee get exactly the same benefits, which are primarily opportunities to participate in the conversation and support the project work.

    Since we will be writing about the projects, and since we will be working to help EEP projects get some form of formal or informal peer review whenever possible, sponsors that make real contributions will get credit that would be hard for them to earn otherwise. In addition to supporting projects started by academics, sponsors can propose projects, as long as (a) the projects investigate educational improvements that are not product- or vendor-specific, (b) the sponsoring company agrees to some sort of community or informal peer review of the work, (c) the results are contributed to the educational community under a Creative Commons license or some equivalent contribution mechanism, and (d) there is at least one college or university helping to drive the project.

    Here are the current EEP sponsors, in alphabetical order:

    • 2U
    • Blackboard
    • D2L
    • Hobsons
    • iDesign
    • Instructure
    • Lumen Learning
    • McGraw-Hill Education
    • RealizeIT
    • Pearson
    • Unicon

    While we are not actively soliciting additional sponsors, we do anticipate that others will be interested in participating in the project work. (We have already been approached by a couple.) We will vet them and include them using the same criteria we have used so far, including that (a) we only want sponsors who have something to contribute beyond money, and (b) sponsors of all shapes and sizes will be treated fairly and equally. We feel very good about the spirit that our current sponsors are bringing to the work so far and want to maintain a collaborative environment free of sales and marketing pitches.

    What About the Academics?

    Since we haven’t yet asked the participants’ permission to share their names, I can’t share them here (although I invite participants to announce themselves in the comments if they feel so inclined). While we have encouraged pairs of people to come from institutions in order to increase the likelihood of project work getting initiated, academics participating in the summit are not officially representing their universities. (That includes our host.) So we are being very careful not to put them in awkward positions with their home institutions.

    We expect to get permission to share more about who they are after the summit. For now, I will say that we worked hard to get representation from a wide range of institutional types and to include researchers, practitioners, and leaders.

    How is This Different from X? And Does This Compete with Y?

    There are many projects with somewhat overlapping goals in different corners of academia. That’s healthy. We believe that EEP is unique in its focus on barriers to collaboration, emphasis on participant projects with sound experimental design, and diverse representation across the higher education sector.

    We do not intend to compete with any existing efforts. On the contrary, our primary goal is to develop connective tissue between and among them. While we made a decision not to invite any representatives of existing network organizations to the summit because of the space limitations, we were delighted to publicly announce EEP to our friends at the recent EDUCAUSE ELI conference. We look forward to collaborating more with them, as well our friends in other network organizations, as we move forward.

    Where Does the Money Go?

    The majority of the money covers the cost of the summit, including travel expenses for academics, as well as our staff time for the work involved. What’s left will go toward facilitating the match-making, any additional related staff support time, and likely some small project grants (like paying a graduate assistant, for example). If we get more money, we will reinvest it into project-related activities.

    How Does EEP Fit With e-Literate and MindWires?

    We’ve heard two versions of this question. One addresses underlying confusion about the relationship between e-Literate, a 14-year-old public service blog, and MindWires, a consulting company that has sprouted up around the blog in recent years, run by the same two guys but seemingly only loosely related. The other asks about the long-term sustainability of EEP under our care. It turns out the answers to both are related.

    Almost all of our consulting business comes from readers of e-Literate, and our clients often hire us to help them apply our analysis to their local contexts. But because our paid work usually takes the form of traditional consulting, legal and ethical barriers prevent us from writing about it. As a result, we are not able to share many of the valuable lessons we learn. On top of that, potential conflicts of interest that we are ethically bound to disclose in our writing can appear worse than they really are because we are unable to disclose the specific nature of the engagements.

    We haven’t decided how we will fund EEP going forward because any plans for Year 2 depend upon the results from Year 1. But we’re not worried about figuring that out because EEP can potentially solve these problems for us, which would make the endeavor worthwhile for us to sustain even if it just breaks even. To the degree that our clients begin to hire us to support and extend EEP-related projects, the support and publicity they will get from working under the EEP umbrella should incentivize them to share valuable lessons learned. Sharing more information about the work we do for our clients will help us clarify, and in some cases reduce or eliminate, potential conflicts of interest in our writing.

    And if EEP can change market incentives such that companies and universities alike will get rewarded for making genuine contributions to a community and culture of educational improvement, then we’re confident we can find ethical ways to get paid to facilitate that change. Our clients may recognize the benefits of engaging us to solve some of their harder problems through the matchmaking and peer support that we can facilitate. But we’re a long way from that. For this year, we are focused on helping EEP participants to develop good collaborative projects.

    When and How Can I Join the Fun?

    If you work at an academic institution, just sit tight for a little while. We intend to open up community participation within a few weeks of the summit. We will publish the specifics as soon as we have worked them out.

    If you work at a potential sponsoring organization, please have the appropriate person contact my colleague Jeanette Wiseman for an initial conversation.

  • e-Literate will make a Big Announcement at ELI This Week

    We usually analyze other people’s announcements, but for once, we will have one of our own. e-Literate has had a project in stealth mode for about the last six months, and we will be unveiling it to the world at EDUCAUSE ELI this week. I won’t drop any spoilers, other than to say that it’s called the Empirical Educator Project. (That’s not really a spoiler, since it’s in the title of my talk.) I’ll be making the big announcement on Tuesday at 10:45 AM CT in the Grand Ballroom.

    Be there or be square.

    (Don’t worry; we like squares. And really, people of all geometric shapes. If you aren’t going to ELI, don’t worry. We’ll publish a post about it here after the conference.)

  • Cengage Unlimited Draws the Battle Lines in the Curricular Materials War

    Cengage Unlimited Draws the Battle Lines in the Curricular Materials War

    As Phil wrote about recently, Cengage has announced “Cengage Unlimited,” which is being described in various outlets as the “Netflix” or “Spotify” of curricular materials. It’s an all-you-can-eat digital subscription service to Cengage’s complete catalog. Spotify is probably the more apt comparison, both because the Netflix analogy is contaminated and because the music industry is a more apt analogy for the economic pressure this puts on content creators.

    Make no mistake; this is a potential inflection point in the curricular materials market. There is a war raging between curricular materials that are “good enough,” meaning that the lower price has a bigger impact on student outcomes than any differences in the quality of more expensive alternatives, versus “better enough,” meaning both instructors and students believe the product makes a sufficient difference in student outcomes that the more expensive product is worth the premium. Cengage is betting the farm on “good enough” beating out “better enough” and, win or lose, their bet could cause tectonic shifts in how curricular materials are developed, purchased, and used. It will have implications for inclusive access, adaptive courseware, textbook companies, textbook authors, and the landscape of options available to students and teachers.

    (more…)

  • Good Enough vs. Better Enough: The Macmillan Example

    Good Enough vs. Better Enough: The Macmillan Example

    In my recent post on Cengage Unlimited, I made a brief mention of the battle shaping up in the curricular materials world between “good enough” and “better enough.” I argued that Cengage is coming down on the “good enough” side by emphasizing all-you-can-eat pricing.

    The distinction I’m trying to make between two strategies is a little tricky. I’m not arguing that Cengage, for example, thinks that their products aren’t great or that they think all anybody needs is the cheapest PDF possible. And on the other hand, “better enough” no longer means better editing or better production values, which is the way that textbook publishers used to position themselves against OER (and still do sometimes, although that reflex is beginning to fade). Rather, it’s about improving student outcomes.

    To borrow a phrase from David Wiley, the fight boils down to standard deviations per dollar. ((“Standard deviation” is just statistics geek speak for a measure of difference—in this case, improvement—from the norm.)) This formulation boils the battle down to a fraction. In the numerator, we have impact. In the denominator, we have cost. David likes to say that it’s easier to change the denominator, i.e., reduce cost, than it is to change the numerator, i.e., improve student outcomes. One of the reasons this is true is that putting a different product in a class usually doesn’t have a big impact unless the instructor’s teaching practices also change to take better advantage of the product’s features. Or, if you prefer a formulation that emphasizes the teaching over the tools (which I do), digital courseware tends to have the most impact in classrooms where it supports the chosen pedagogical approach of the instructor.

    For the incumbents, neither the numerator nor the denominator is particularly easy to change. In my last two posts, I wrote about the major investments—and risks—that Cengage took on to deliver their products at a better price point and still make their business model work (they hope).

    But that may be a cake walk for the publishers compared with the challenge of changing the numerator. In my original post about Pearson’s efficacy strategy, I explored these challenges at length. I have chosen to quote a hefty excerpt here because none of these problems have gone away:

    Let’s think some more about the analogy to efficacy in health care. Suppose Pfizer declared that they were going to define the standards by which efficacy in medicine would be measured. They would conduct internal research, cross-reference it with external research, come up with a rating system for the research, and define what it means for medicines to be effective. They would then apply those standards to their own medicines. And, after all is said and done, they would share their system with physicians and university researchers in the hopes that the medical community might be reassured about the quality of Pfizer’s products and maybe even contribute some ideas to the framework around the edges. How confident would we be that what Pfizer delivers would consistently be in the objective best interest of improving health?…

    If Pearson were to say to faculty, “Here’s what we think we know about the efficacy of this product, here’s what we don’t know yet, and here is how we are thinking about the question,” they might get a number of responses. Maybe they would get, “Oh, well here’s how I know that it’s effective with my class.” Or “The reason that you don’t have a good answer on effectiveness yet is that your rubric doesn’t provide a way to capture the educational value that your product delivers for my students.” Or “I don’t use this product because it has direct educational effectiveness. It frees me up from some grunt work so that I can conduct activities with the class that have educational impact.” Most of all, if you’re [Pearson CEO] John Fallon, you really want faculty to say to their sales reps, “Huh. I never thought about the product in quite those terms, and it makes me think a little differently about how I might use it going forward. What can you tell me about the effectiveness of this other product that I’m thinking about using, at least as Pearson sees it?” And you really want your sales reps to run back to the product teams, hair on fire, saying “Quick! Tell me everything you know about the effectiveness of this product!”

    Pearson won’t get that conversation by just publishing end results of their internal analysis when they have them, which means that they have a high risk of failing to align their products with the needs and desires of their market if they think about the relationship between their framework and their customers in that way….

    There are a number of reasons why this part of the transformation will be at least as difficult as the part that Pearson is undertaking now. First, it is far from clear that the company has the trust of the academic community that would be necessary for them to take such a role. That would have to be built, in some cases from the ground (or even the basement) up. Pearson does have real strengths that are known within certain segments of the academic community—in data science, for example—but this does not transfer to a general reputation. Second (and relatedly), unlike the medical research community, the educational research community is still nascent and fragmented. Finding non-paternalistic but effective ways to bring that community together and facilitate useful conversations will be difficult to say the least. These two challenges are outside the company’s sphere of control, which means that Pearson will have to develop new ways to think about how to build their relationships with the broader educational community.

    Internally, changing the way they think about answering the questions that the framework asks them will entail as much subtle, difficult, and pervasive re-engineering of the corporate reflexes and business processes as the work being undertaken now….  [A]ll textbook companies that have been around for a while are wired for a particular relationship with faculty that is at the heart of how they design, produce, and sell their products. Their editors have gone through decades of tuning the way they think and work to this process, and so have their customers. When Pearson layers a discussion of efficacy onto these business processes, a tension is created between the old and new ways of doing things. Suddenly, authors and customers don’t necessarily get what they want from their products just because they asked for them. There are potentially conflicting criteria. The framework itself provides nothing to help resolve this tension. At best, it potentially scaffolds a norming conversation. But a product management methodology that can combine knowledge about efficacy, user desires, and usability requires more tools than that. And that problem is even worse in some ways now that product teams have multiple specialized roles. The editor, author, adopting teacher, instructional designer, cognitive science researcher, psychometrician, data scientist, and UX engineer may all work together to develop a unified vision for a product, but more often than not they are like the blind man and the elephant. Agreeing in principle on what attributes an effective product might have is not at all the same as being able to design a product to be effective, where “effective” is shared notion between the company and the customers. ((Believe it or not, that is a short excerpt as measured as a percentage of the total word count of the post.))

    Publishers that want to improve the numerator will have to completely rewire the ways that they work, both internally and externally. They need to rethink their product design process from the ground up while simultaneously completely resetting their relationships with their customers.

    That post was published on December 31st, 2013. As we enter 2018, we are beginning to see examples of what such efforts might look like. For today’s example, I’m going to draw on recent work by Macmillan.

    Resetting the Conversation

    Before I get into the details, a little more disclosure than usual is called for here. I am a paid member of Macmillan’s Learning Impact Research Advisory Council (IRAC). As such, I was paid to provide input on the paper I’m about to write about as well as the underlying research processes that the paper describes. I was not paid to write this post about the paper. Or rather, I was paid to write something about it, but I was asked to write one page—one page!—of private feedback on the paper. I asked if I could write my feedback as a public blog post of unspecified length. The folks at Macmillan agreed.

    The paper is called Unpacking the Black Box of Efficacy: A framework for evaluating the effectiveness and researching the impact of digital learning tools. Registration is required.

    First piece of feedback for Macmillan: If you really want to foster a new dialog with academics, don’t start it by requiring them to give you their email addresses just to read your paper.

    But the approach outlined in the paper is another matter. Recall that in the Pearson post quoted above, I advised the company to approach customers with something like the following proposition:

    Here’s what we think we know about the efficacy of this product, here’s what we don’t know yet, and here is how we are thinking about the question.

    That is essentially what Macmillan’s paper attempts to do. It starts with an inventory, in plain English, some common educational research methods, how they work, and what their strengths and weaknesses are. The section on randomized controlled trials (RCTs) alone is worth the price of admission, given how often it is simplistically held up as the “gold standard” in research. Any thoughtful educator reading the description of the process will immediately think, “Hey, that’s…problematic in education.”

    Even better, Macmillan was able to accomplish that with one page of text and one picture. They will need to be this incisive on a consistent basis if they are going to reach their intended audience.

    Next, the paper describes their product development lifecycle. Again, there is a good balance here of clarity and brevity. The first stage of that lifecycle is called “Co-design & Learning Research.” While publishers have pretty much always started their product design process with input from customers, I wouldn’t call the historic process “co-design.” Rather, it was typically an author/editor collaboration with some limited and focused customer input. More recently, publishers have developed all kinds of hybrid processes. But Macmillan at least claims to be starting with a clean sheet of paper. They are certainly not the only publisher to do this, but from a communication perspective, framing educational product design as a combination of co-design with customers and structured but comprehensible research is a good move.

    Speaking of which, the third section maps the various research methods described in the first section to the product design process in the second. There’s even a development timeline. The net effect is that educators (and students) have a clear and concise document explaining how Macmillan products are developed, how their potential learning impact is tested, and just how much it’s fair to say that the company knows about that impact at any stage in the development lifecycle.

    While I am by no means claiming credit, this paper reads as if it could have been written as a direct response to my critique of Pearson’s first iteration of efficacy.

    So yeah. I like it.

    Good Enough for What?

    You didn’t think I’d let them off that easily, did you?

    Remember waaay back, all the way at the beginning of the post, when I made the point that learning outcomes are hard to improve with curricular materials partly because their impact depends on what humans in the classroom do with them? That problem still looms, and Macmillan’s paper barely touches it.

    When I talk to students at length about the curricular materials that their instructors assign, their top complaint isn’t price. Don’t get me wrong; they hate the prices. But what they really hate is being told to buy a $200 book that the instructor barely mentions, let alone integrates into the class on a programmatic basis.

    This is what “better enough” is competing against. I always thought it was funny that textbook publishers refer to everything outside the book as “ancillaries,” because many instructors tend to see the categories as reversed. The book is ancillary. It’s not central to the learning that happens in the classroom. Before Macmillan, or any other publisher, can sell products based on the value proposition of “efficacy” or “learning impact” or “learning outcomes”, instructors must first come to believe that these three propositions are true:

    1. Instructors are responsible for learning how to improve their students’ learning outcomes by improving their teaching craft.
    2. Improving their teaching craft includes learning to employ research-validated practices.
    3. Macmillan’s products support and enable research-validated practices effectively enough that they can make the credible case for having more than “ancillary” value.

    This paper makes a good start—as good a start as any short paper can make—on the third proposition. The first proposition isn’t fair to lay at the publishers’ feet; it’s more driven by the incentives and culture of academia. It’s a problem, but not one that Macmillan or its peers can do much about directly. The second proposition is where the vendors, including but not limited to Macmillan, need to figure out how to do more.

    To be fair, the paper nibbles around the edges of this problem. The educators and product developers need to develop shared goals. That’s what a co-design process is for. Educators and product developers also need to develop a shared sense of proof that the goals are being met better by one method than another. A lot of the paper develops the basis for a conversation around this.

    But left implicit is the argument that education should be empirical and that empiricism needs to be formalized at least some of the time. There should be theories of learning impact and rules for what counts as evidence that supports or disproves those theories. This needs to apply not just to curricular materials design but for what happens in the classroom.

    It’s probably too much to expect this paper, as focused as it is, to open up this Pandora’s Box. This paper is, in part, a trust-building exercise, and Macmillan needs to build trust before they can fully own up to the fact that incorporating curricular materials that meaningfully improve learning outcomes usually entails a course redesign. But that’s where both Macmillan and the industry need to get to if they want to be able to sell more heavily researched and designed products at a higher price point.

    “Good enough” means “good enough for the way I use curricular materials in my classroom.” “Better enough” means “better enough that I’m convinced I should change the way I teach.” Macmillan has written a really good paper on the standards of proof they propose to live up to and how they propose to live up to them. But they also have to convince their customers to agree to live up to those same standards in their own teaching.

  • Cengage Unlimited Textbook Author Update

    Cengage Unlimited Textbook Author Update

    The Textbook & Academic Authors Association has a blog post up exploring the implications of author royalties that I raised in in my last post on the subject. Here are some of the highlights:

    • What we’ve heard from Cengage’s competitors is true; most of the authors were not given significant notice prior to the announcement. This was done for reasons of competitive secrecy; Cengage didn’t want the word to leak out before they were ready.
    • Because Cengage Unlimited doesn’t officially launch until August 2018, the company believes it will have time to work out the details with the author.
    • Cengage has stated that it intends to honor existing contracts with its authors but also claims that those contracts grant it the flexibility it needs for Cengage Unlimited.
    • There is precedent for coming up with pricing models. Publishers already bundle products together and have contractual provisions in place to handle pricing for such bundles. So legally, they have a precedent. But how this would apply to an “everything” bundle is far from clear.
    • A key to figuring out author royalties on these bundles will be usage, which means new technology tracking and some uncharted territory in terms of metrics. For example, Cengage says that the student just putting a product in their digital locker is not sufficient to count as usage for royalty purposes. But then what is? And does the publisher have the ability to track this usage reliably? And how intrusive would this tracking be?
    • The lawyers quoted in the article definitely give the impression that there is a lot of gray area—and variability—in terms of what author contracts do and don’t permit. So there will be Intense negotiations and quite likely some lawsuits (although author lawsuits against textbook publishers is apparently relatively common).
    • The argument given by Cengage to its authors is that the authors have as much motivation as publishers to get back to a world in which all students buy their curricular materials, even if it’s at a lower price.

    I see a couple of main take-aways from this. First, we should not underestimate the complexity of the move that Cengage is making from legal, technical, and financial perspectives. And they definitely don’t have it all figured out yet.

    Second, the question of royalties and what counts as usage is going to raise all sorts of analytics questions, some of which many overlap with student privacy concerns. Who has a right to know how much students are using books in which ways? And what steps are taken to keep those data properly anonymized for people who need the aggregate but not individual data (such as authors or their lawyers)?

    Third, all this complexity is definitely going to incentivize Cengage to use as much content as possible that they either own outright or can adopt under a Creative Commons license.

    And finally, the argument that Cengage is making to its authors brings to mind yet another analogy to a disruptive service in the tech industry. We’ve already seen analogies to Spotify, Netflix, and Amazon Prime for Cengage Unlimited, but it may be like iTunes in one important respect. The idea is that it reduces incentives for piracy by giving students easy access to the product at a reasonable price is exactly the same argument that Apple made to the music industry.