e-Literate

Present is Prologue

Category: Ed Tech

The “Ed Tech” category includes posts about educational technology products themselves, including LMSs and other learning platforms, adaptive learning and other digital curricular materials products, learning analytics, and educational apps of all types. It also includes technical aspects of ed tech products, especially interoperability.

  • A Courseware Platform for Expressing Pedagogical Intent

    A Courseware Platform for Expressing Pedagogical Intent

    In a recent post, I announced a collaboration between Carnegie Mellon University’s (CMU) Open Learning Initiative (OLI) group and Arizona State University’s Center for Education Through eXploration (ETX) to build a next-generation, open-source, open-standards project, which has been facilitated by the Empirical Educator Project (EEP). As I noted in that post, because each group has intra- and extra-university diplomacy and policy responsibilities, I have pre-approved the previous post and this one with ETX and OLI, respectively. With these two posts in place, I will explore some implications in subsequent posts without worrying about coloring outside the lines.

    The first post in this series described the nature of the collaboration. This one will describe the nature of the project itself.

    A recap of how we got here

    Here’s the sequence of events that led to the current project:

    First, about a year and a half ago, CMU announced at the EEP summit that they were releasing an open-source collection of over $100 million’s worth of learning engineering software they had developed in a collection they call the OpenSimon Initiative. OpenSimon initially included parts of the OLI’s platform—also called OLI—which is both the original template for most current-generation courseware platforms and a platform designed from the start to support research into applied learning science. (The reasons why they could not initially release all of OLI are unimportant for this story. The resolution of the problem, which I’ll get to momentarily, is more salient.) Norman Bier, who is both the Executive Director of the Simon Initiative and the Director of OLI, spearheaded OpenSimon’s release and has led OLI’s participation in the collaborations to come.

    Separately, in the spring of 2019, ETX Director Professor Ariel Anbar decided to fund a courseware interoperability standards effort as part of an Open Educational Resources (OER) development grant the US Department of Education had awarded to the Center (a project which has since been branded OpenSkill). Since ETX planned to develop their OER courses on the proprietary Smart Sparrow platform, Ariel believed that those courses would need to be portable to other platforms to satisfy the spirit of OER. So he hired EEP to help convene a group and facilitate the work. One of the participants that I recommended was OLI. We had a kick-off convening in September of 2019.

    Shortly after that convening, Pearson announced that it had acquired the SmartSparrow platform and would be discontinuing support of the product in its current form. (As I noted in my previous post, I have nothing to say about Pearson’s actions other than that ETX stakeholders have told me that the company has worked with the university to make the transition as smooth as possible.) Unfortunately, since we do not yet have courseware interoperability standards, Pearson’s ability to make the transition smooth is limited. There is no easy way to get the functioning courseware out of Smart Sparrow and import it intact into some other one.

    Rather than letting this substantial transition derail the interoperability work—ETX has over $15 million’s worth of grant–funded courseware that they need to migrate off of Smart Sparrow by a fixed deadline—Ariel decided his predicament proved that interoperability is too important to ignore. So he began looking for partners who would be willing to invest in this goal.

    During roughly this same time period, OLI began developing the next generation of their platform, code-named “Torus,” which could be released as open-source. (The code repository for the project is available on Github today.) After some discussion and exploration, ETX and OLI agreed to be primary partners on this project, developing a specification that would enable ETX’s Smart Sparrow-based experiences to run intact on Torus. They also agreed to design the architecture to use existing interoperability standards whenever possible, propose enhancements to those standards when it is helpful, and to invent non-standard approaches as needed. Torus will be a reference implementation for proposed courseware interoperability standards. It will demonstrate the viability of the proposed standards by showing that it can import and run ETX’s existing Smart Sparrow intact and by enabling ETX to continue authoring Smart Sparrow-style courseware in the future. ETX has engaged EEP to facilitate this collaboration.

    A rare opportunity

    In this project, we have two different applied academic centers that both build courseware for use at scale. They focus on substantially different (though complementary) research-based course design approaches. Each has chosen a platform because of its suitability for their respective approaches. Both perform efficacy research on the courseware they develop. By bringing them together in this way, we have a rare opportunity to intentionally drive the evolution of the entire product category, starting with a consideration of research-supported pedagogy and driving it deep into the fundamental structure of how courseware platforms are built by influencing their interoperability standards. Technology is the means, not the end. The real goal is to develop a generalized approach for courseware platform architecture that can implement a range of evidence-based design approaches and drive continuous improvement of digital curricular materials that demonstrably support student success and improve equity.

    For this reason, I will not be writing about the technical details in this post. Instead, I will focus on foundational pedagogical and functional concepts that drive the design decisions. Readers who are specifically interested in technical interoperability or architectural questions will have to read between the lines. I promise to cover these topics more explicitly in later posts.

    Because we have been building toward this public collaboration for quite some time, I have been laying a breadcrumb trail regarding this project since long before I could talk about it explicitly. If you want to really study this project, I recommend reading the other posts that I reference. The links to previous e-Literate posts embedded in this one are more important than usual.

    First principles

    Let’s start by laying out some basic pedagogical principles that both organizations endorse and that infuse all aspects of plans for this project, from the interoperability standards to the software architecture to the user experience.

    First, “to learn” is an active verb. Students cannot “get learned” or “be learned.” They must do the learning themselves.

    Learning results from what the student does and thinks, and only from what the student does and thinks. The teacher can advance learning only by influencing the student to learn.

    Herb Simon

    “Influencing the student to learn” can be interpreted as provoking students to think and do things that you hope will stimulate them to learn. People learn when they try something or think about something and come to a conclusion or insight based on that process of exploration. Part of the craft of teaching is knowing what to give students to think about or try.

    The actual learning happens inside a student’s head, which means that it is not directly observable. So another part of teaching is giving students tasks to perform that provide observable clues as to what they may or may not have learned. We call this teaching task “assessment.” Importantly, self-taught learners assess themselves. For example, if I am learning how to make scrambled eggs, I will try to scramble and egg and observe how it comes out. I won’t know if I learned how to make scrambled eggs until I try to scramble one. Likewise, I won’t know if I’ve failed to learn how to scramble an egg. If I assess myself, then I can keep trying things and observing the outcomes until I learn whatever it was that I missed the first time (like, for example, how to tell if the pan is the correct temperature for scrambling an egg). Self-taught learners are, in part, self-assessing learners. They construct tests for themselves to figure out what they haven’t gotten right yet. Educators assist learners with that process.

    Regardless of whether I have a cooking coach or if I am teaching myself, I may very well decide what to try next based on the results of my assessment. If my eggs come out poorly, then I will think about what I might have done wrong and try something else. If they came out well, then I might try incorporating them into a breakfast burrito. Or I might move on to learning how to make bacon.

    In other words, learning is a feedback loop, where the input is the learning experience and the output is the assessment result. Learning may happen in between the input and the output, inside the black box that is the learner’s head. It may also happen when the learner sees the output, which might become the input for another learning feedback loop. A failed attempt to scramble an egg may cause me to do or think something that enables me to learn how to successfully scramble an egg.

    It works something like this:

    1. Input: I read a recipe for scrambled eggs.
    2. [Weird things happen in the dark place between my ears.]
    3. Output: I try to scramble an egg.
    4. Inference: I judge whether I am ready to move onto making a breakfast burrito or if I have not yet achieved egg-scrambling competency.

    (It’s not exactly this clear cut. For example, I learn things about scrambling eggs in the process of scrambling an egg.)

    As part of that last step, I’m evaluating whether I’ve learned what I set out to learn. Hence, the feedback loop. Based on that inference I make from the assessment results, I decide whether to repeat the learning feedback loop, adjust it, or move on to a new one. But again, a critical point here is that Step 2—the actual learning—is invisible. The brain is a black box. We only know what goes in and what comes out.

    A widely practiced discipline in learning design, known as “backward design,” is built on these fundamental insights. At its simplest, backward design has three steps:

    1. Specify the desired learning goal or competency (e.g., scrambling an egg)
    2. Design the assessment that will test progress toward the competency (e.g., trying to scramble an egg)
    3. Design the experience that will stimulate the learner to do or think something that will hopefully enable them to learn (e.g., reading a recipe)

    Steps two and three in backward design correspond to the output and input of the learning process. While step one was implicit in my previous example—I started off wanting to learn how to scramble an egg—it must be explicit when designing a curriculum using this method.

    The backward design learning feedback loop. (Credit: Carnegie Mellon University)

    In fact, the curriculum can be thought of as the scope and sequence of these specified learning goals. The scope refers to the full range of competencies that I need to master to reach a larger goal. Is my ultimate aim to learn how to make a bacon-and-egg breakfast or a breakfast burrito? The sequence is the order. I can’t learn how to make a breakfast burrito until I first learn how to scramble an egg. In other words, we can think of the process of learning design—and teaching, and learning—as constructing a series of nested learning feedback loops. Many educators (and learners) construct these loops instinctively and unconsciously, which means that they sometimes miss opportunities to make their designs more effective. But whether consciously or not, all learners, all educators, and all designers of instruction construct sequences of feedback loops. Because without learning feedback loops, there is no learning.

    Computers are (mostly) dumber than humans

    So humans can’t learn, and therefore can’t teach, without feedback loops. They can’t tell if learning is happening, never mind provoking thought that leads to learning. Guess what? Computers can’t either. And no amount of data or fancy AI algorithms will change that, because both learning and teaching happen via feedback loops. I’ve seen (and written about) a lot of magical thinking in EdTech that talks about data as if it’s fairy dust: one speck is pretty much the same as any other, but the more you have of it, the more magic you get.

    I wrote a blog post last December called Pedagogical Intent and Designing for Inquiry which was all about this misconception and how it harms our efforts to build effective EdTech. “Pedagogical intent” should be interpreted as the reasoning behind the design of the learning feedback loop. For example, the software might detect patterns in students’ answers that suggest a particular skill is not being taught well (or at all). Or it might identify a prerequisite skill or bit of knowledge a student missed earlier that is causing her to struggle now.

    To find these insights, the computer needs more information about the activities than just the clicks. It needs information about pedagogical intent. Why is the educator assigning the student a particular activity to perform? What was a particular assessment answer choice intended to suggest about the student’s internal mental processes? Without knowing the answers to these questions, the computers are as blind as we are. Here’s an example from that post:

     ACT recently released a paper on detecting non-cognitive education-relevant factors like grit and curiosity through LMS activity data logs. This is a really interesting study that I hope to write more about in a separate post in the near future, but for now, I want to focus on how labor-intensive it was to conduct. First author John Whitmer, formerly of Blackboard, is one of the people in the learning analytics community who I turn to first when I need an expert to help me understand the nuances. He’s top-drawer, and he’s particularly good at squeezing blood from a stone in terms of drawing credible and useful insights from LMS data.

    Here’s what he and his colleagues had to do in order to draw blood from this particular stone…

    First, they had to look at the syllabi. With human eyeballs. Then they had to interview the instructors. You know, humans having conversations with other humans. Then the humans—who had interviewed the other humans in order to annotate the syllabi that they looked at with their human eyeballs—labeled the items being accessed in the LMS with metadata that encoded the pedagogical intent of the instructors. Only after they did all that human work of understanding and encoding pedagogical intent could they usefully apply machine learning algorithms to identify patterns of intent-relevant behavior by the students.

    LMSs are often promoted as being “pedagogically neutral.” (And no, I don’t believe that Moodle is any different.) Another way of putting this is that they do not encode pedagogical intent. This means it is devilishly hard to get pedagogically meaningful learning analytics data out of them without additional encoding work of one kind or another.

    Pedagogical Intent and Designing for Inquiry

    So far, machine learning has not proven to be generally effective at deducing pedagogical intent based on low-level data like student clicks or time they spend watching a video. It’s getting pretty good at deducing pedagogical intent from data when it is given inputs that are already well-structured to supply hints of that intent. LMS activity logs are not well-structured in this way. Ironically, analog textbook chapters are because they are designed to give humans clues about these things using only text organization, pictures, font styles, and the like. But that doesn’t help us to reliably deduce pedagogical intent in the realm of the digital. There ought to be a better way.

    Luckily, there is one.

    Learning feedback loops in courseware design

    It turns out that virtually every commercial courseware product on the market today, and virtually every course created with substantial input from a professionally trained learning designer, has been built using backward design. Furthermore, most interesting and useful learning analytics in commercial courseware products today get virtually all of their utility from this backward design and very little, if any, from some super-genius artificial intelligence algorithm. Machine learning is most often useful when it can “read” the backward learning design to work its magic. Today’s learning analytics and adaptive learning systems are best thought of as algorithms that attempt to squeeze as much insight and value as possible from the pedagogical intent that has been encoded into the courseware platform in a machine-readable form. They may use other data too. But as the ACT study above illustrates, those other data have limited value when they are disconnected from the context of the learning feedback loop designs and the pedagogical intent behind them.

    Last September, I wrote a three-part post series on exactly this topic called “Content as Infrastructure” (the title of which was borrowed from David Wiley). You can probably skip the first post since it covers a lot of the same ground you’ve just read. The second post shows examples of different products that take advantage of backward design to create better educational products, while the third one digs into the various types of insights that can be gained from computers when they have access to semantic information about pedagogical intent. If you want to dig a little deeper, you can go back to my post from 2012 about how CMU’s learning engineering techniques use learning feedback loops to identify hidden prerequisites that the educators may not have realized they need to address (like the recipe writer who doesn’t realize that not all aspiring egg scramblers know how to tell if the pan is the right temperature). Or you can read my post from 2013 about how ETX’s Habitable Worlds course creates experiences that provoke student learning.

    Generalizing feedback loops

    As promised, I will keep this post non-technical. But given the pedagogical principles above, we can begin to think through the basic requirements for a universal technology design.

    First, we need standard ways to describe the smallest and simplest learning feedback loops, like a single multiple-choice question. The student reads the question, chooses an answer, and receives feedback. Simple. And yet, not. A well-written multiple-choice question will have wrong answers—or “distractors,” as they are often (problematically) called—that are intended to diagnose specific common misunderstandings. Students might be given multiple tries on the question, thus traversing the feedback loop multiple times. They might get feedback based on their answer which is intended to help them learn or reinforce their learning. They might be given the option to ask for a hint. Or multiple hints. And each successive hint may be designed to get at a different possible blocker to the student’s learning the correct answer. A single well-design multiple-choice question implemented in a system that gives options for multiple tries, feedback, and hints can be absolutely loaded with pedagogical intent. We want a system that captures as much of that as possible and as automatically as possible. The more that the author’s pedagogical intent can be captured as part of the natural authoring process—rather than requiring the author to add that information afterward—the better.

    Second, we need a system that enables learning feedback loops to be connected to each other in different ways, and to create loops within loops. On a fundamental level, if I want to make a bacon-and-egg breakfast, I have to learn how to scramble an egg. If I want to learn how to scramble an egg, I have to learn how to tell if the pan is the right temperature. If I want to learn how to become a short-order cook in a diner, then I need to learn all these skills plus many more.

    We also need the feedback loops to be connectable in many ways. A common approach in OLI courses is to give students a string of activities that address the same “learning objective”—a label we attach to a learning goal of a certain granularity that is meaningful in the context of the larger course—and watch to see if the students’ error rates on those problems go down. This requires the ability to tie those problems together in a sequence. A common approach in ETX courses is to create “guided experiential learning” exercises, in which students learn by exploring. This requires the ability to branch students to different learning feedback loops based on their answer to the previous one (even if the branch is just a short detour that leads back to the main path). Both approaches are valid and not mutually exclusive. They can be used in the same course. Sometimes they can even be incorporated into the same exercise.

    When we talk about interoperability—which was where this project started—we can mean at least two very different and equally important goals. The first is to be able to export a learning feedback loop design of any grain size—a multiple-choice question, a quiz, a loop that addresses a skill, or a learning objective, or a competency, or a lesson, a course, a certificate program, etc. (You might be surprised at the amount of thought that goes into designing a single multiple-choice question to create a truly diagnostic learning feedback loop.)

    The generic word we use for a compound learning loop of any size is an “ensemble.” I’ve seen a lot of debate and hand-wringing over the years about what a “learning object” is, how big or small one can be, and so on. The problem with “object” is that it’s a fancy name for “thing.” Of course very different kinds of learning things should be very different sizes. In this project, we talk about learning feedback loops, which also can be very different sizes. But they are all the same basic type of thing, in the sense that they have learning inputs and produce learner outputs which are diagnostic of particular learning goals. Any learning thing—whether it is a picture or a course unit or something else—which does not have these characteristics is not a learning feedback loop.

    We should be able to move the content, the execution logic of the learning loops, and the encoding of the pedagogical intent, intact, from one system to another. We also should be able to analyze student activities in the context of the pedagogical intent of the learning loop the same way for any courseware in any system. We should be able to know what the learning goal is, what the student’s assessment answer suggests about progress toward that goal, and so on. The interoperability project aspires to achieve both of these goals, although they will take some time to refine and to take them through an official standards-making process.

    And we want to create a system that encourages educators to author learning feedback loops that are well-crafted and continuously improve that content based on feedback from the system on how well it is helping the students. Most educators—particularly ones who have no formal training in education, like the vast majority of college professors—do not naturally think about what they are doing as crafting series of learning feedback loops. Even when they understand the basic idea, they may not have all the skills that could help them (just as understanding the basic concept of scrambling an egg does not automatically convey the skill of making yummy scrambled eggs). For example, a learning designer recently told me that some of the professors she works with don’t know that they should try to write multiple-choice distractors designed to diagnose specific misconceptions. In an ideal world, a courseware platform would have an authoring interface, analytics, and nudges that all encourage courseware designers—whether they are first-time amateurs or skilled professionals—to constantly be working to test and refine their learning feedback loops.

    This thing is real

    These are obviously big ambitions that will take a long time to realize. Some of them will take at least few years to get right or to get through formal standards-making processes. That said, this is more than just another grand idea that will probably never happen.

    It is happening right now.

    Here are some milestones that the collaboration has achieved so far:

    • Torus exists. The source code is available. Some simple courses are in production on a stable version of it.
    • OLI had several teams build sample lessons on Torus during a virtual summer school program.
    • ETX, with the help of Unicon’s architects and developers, has exported the lessons from Smart Sparrow, including the logic of the learning loop design.
    • Unicon has built several iterations of a harness that can run a wide range of ETX Smart Sparrow with increasing fidelity. To be clear, they are not rebuilding these lessons. They are running the exported lessons, with the full learning experience, intact.
    • The collaboration team has demonstrated the ability to launch the ETX lessons from OLI.
    • IMS Global, one of the major technical interoperability standards bodies for education, has been kept apprised of the work and has been actively supportive of it.
    • Unicon’s latest iteration of the Smart Sparrow test harness runs the full representative sample of ETX lessons with production-quality fidelity. (A few minor tweaks remain to be completed.)

    Here are some planned milestones:

    • The collaboration team is in the process of generalizing the runtime commands for the ETX learning loops into a specification that can be implemented in Torus and submitted as a formal interoperability specification candidate.
    • In parallel, ETX and Unicon are working on a developer’s aid that will enable ETX experts to begin creating new ETX-style lessons in the first quarter of 2021. While it will not be suitable for general use, it will help us continue to refine the specification and think about broader authoring requirements.
    • By the fall of 2021, ETX expects to migrate all of its Sparrow courses onto Torus and OLI expects to put a substantial subset of its courses into production on Torus.
    • Work on enriching the authoring interface will proceed in parallel, with the goal of releasing an authoring solution that fully supports multiple styles of course design by January of 2022.
    • OLI will complete its migration of all its courses to Torus by the fall of 2022. Given the current size and growth of the two projects, I expect the combined annual enrollments on the platform to be well north of 150,000 – 200,000 at a minimum at that time.

    The progress and productivity of this collaboration has been amazing. I’ve never seen anything like it in my entire career. And the architectural innovations are quite clever. I can’t wait to start writing about them.

    But the main take-away for now should be that this project is real, it is already producing tangible results, and it will be out in the world in a meaningful way within a year.

    Some of you have already asked me if you can observe or even participate in this project. We are thinking about ways to do that and will have some answers for you in the new year. There will likely be several options, depending on the nature and seriousness of your interest. Opportunities for observation will likely come before we can open up for participation, particularly given the aggressive deadlines that both ETX and OLI have to meet. But this is a public good project with a core commitment to openness. If you are interested in getting involved, please contact me. I’m gathering information about interest so that we can begin planning a community strategy.

    Stay tuned for more

    This post barely scratches the surface of the ambition. In future posts, I will be digging deeper into the details. In the meantime, I suggest again that you (re)read the second and third installments of my Content as Infrastructure post series to get a little more grounding in how much work the basic structures I’ve described here can accomplish and my post Pedagogical Intent and Designing for Inquiry regarding why capturing pedagogical intent is so important for gaining insights on teaching and learning out of these systems.

  • Good vs. Great Product Teams

    Allowing teams to do their work is the hard part.

    Michael and I do a fair bit of work on topics like product-market fit and validating clients’ product and market strategies. We recently completed work with a company that I would characterize as a good exemplar of a “great” product team. In fact, during one of the calls with the client, Michael pinged me on Slack, “they appear to be amazingly competent”. It’s what inspired me to write this post.

    We see a lot of different organizations in both non-profit and commercial setting and always learn a ton about organization design, leadership and the impact of product teams on the enterprise’s relative success. Of course, I have my own learning experiences leading a large, multi-tens-of-millions dollars product organization as well. There are several distinct themes that separate the bad or merely good from the great, which I will try to parse out with a bit of EdTech context:

    1. Customer centricity vs. business centricity
    2. Trust vs. command and control
    3. Leadership vs. management

    Great product teams serve the customer

    Being “customer first” may seem trite or cliché to some, but it is worth being very clear about why this matters and why it is hard for some organizations to be great at it. First, many organizations do business strategy planning and the product roadmaps fall out of what the various business functions believe to be important – often claiming to be customer-centric. Here’s what that might look like in different settings:

    • The sales and marketing teams listen to the hype cycle being driven by hot startups and/or innovators in the industry. In EdTech the classic example is “adaptive learning”; a category rife with widely divergent approaches. That hype cycle buzz is amplified by the media and the talk on the floor of the trade shows. That hype cycle in turn creates real and perceived roadblocks with customers in the sales process. Sales and marketing in turn demand that “product” add those features in the roadmap immediately, only to find that by the time “product” delivers, the hype has moved into the trough of disillusionment.
    • The technology team emerges from business planning worried about a range of real-world threats to the tech stack, cybersecurity, tech debt and growing overhead costs associated with the delivery environments. FERPA, GDPR, and especially web accessibility offer consistent headaches for EdTech teams. The technology team believes they—not “product”—are accountable for these business risks and in turn demand that well more than half of the available roadmap is devoted to “non-functional requirements” to offset the real and perceived risks. Often this leads to risk-shifting, where one risk is solved only to push the cost and accountability (e.g., reduced product innovation, forced tech migration costs) elsewhere in the organization and on the balance sheet.
    • The “product” team is really operating as an agile scrum team, which could be great, but they have become religious about a specific implementation of agile. Agile methods deconstruct customer value into smaller “chunks” for execution, based on the premise that teams can iterate subsequent releases to get customer value “just right”. The challenge, particularly in EdTech, is that release cycles do not typically allow for incremental customer validation because shipping features that change educator or learner experiences during a term is strictly verboten. The overly precious focus on the process model undermines innovation, and the “product” person becomes a de facto backlog manager, delivering output but not necessarily value.
    • The leadership team believes they need a big project to talk about with the board and shareholders. They establish a “big idea” initiative and pull key talent into a “tiger team” or something similar, set them up with their own budget and tell them to go build the next generation something or other platform. I have seen this a number of times in EdTech: an LMS trying to build a content marketplace, a book seller building an ebook platform, an institution building its own LMS, ebook and marketplace. The project begins to consume all the internal executive spotlights, water cooler chatter, investor attention and generates negative energy across the rest of the organization, causing talent to flee and customers to question what they can expect.

    Great product teams serve the customers in ways that are aligned to and meet the needs of the business. They are specifically a cross-functional mix of product, design, analysts, engineering and architecture, intentionally supported by other key functions and empowered to solve problems. Those problems are framed as being directly customer-centric and are lined out against customer value creation; customer utility and usability; the feasibility of organization’s ability to implement (on-time/on-budget); and the viability of the solution across dimensions of the business. The product team is then held accountable to metrics related to the solving of those problems. And everyone in the business understands the importance of supporting that product team, helping them make informed decisions, avoiding delivery pitfalls, and messaging clearly to internal and external audiences about customer value and related business results.

    In the case of our exemplar client, while we do not have organization charts or much visibility into how they operate, it became clear that the product team holds true to the role of creating customer value. The other teams spoke to us of how they are contributing to a shared set of outcomes, how they are providing critical information for the product team to consider about feasibility, time to market, user interaction design. And that the product team owns the customer value creation in line with the needs of the business.

    Great product teams assemble great talent and earn trust

    A lot of companies hire product managers and tell them to, well, manage products. As discussed above, this might happen in different flavors, with different results, past performance isn’t indicator of future performance etc. etc.

    Great product teams (and companies) realize very early on that the qualitative nature of their products holds a very specific correlation to the qualitative nature of their talent and how they are empowered. These teams place considerable emphasis on talent acquisition and management. The talent acquisition is about character equally as much as past track record, technical skills, or the pedigree of their education. Short cuts and trade-offs are not allowed, character matters a lot. Demonstrated integrity and personal motivations for wanting to be on the team are critical, but humility, or lack thereof, is a deciding factor.

    And as great product teams are assembled they are set up deliberately to earn trust across the organization.  There are myriad ways to do this, and if you’re following the thread, this trust is essential, it is why great product teams focus on talent acquisition and grooming to demonstrate their competence and ultimately to own the customer value creation in line with the needs of the business.

    In the case of our exemplar client, we have no direct insight to their talent acquisition approaches, but we were struck by a very consistent attitude across the functional teams: they are all in it together. Often we hear functional teams at odds, and air grievances with us hoping we can “help” (which only sometimes we can, see below). Competence, confidence and ability to execute are all enhanced when the team(s) are rowing in the same direction.

    Great product teams understand where leadership and management matter

    All too frequently, companies believe the role of executive leadership is to provide a command-and-control model of “management”. It’s pretty straightforward, especially with experienced, and “intelligent” (frequently not humble) executives. They think they know what the customer wants, they see trends in the market to guide the arc of the business, and they start directing product teams what to build, how to build it, and when they want it. They might even require the team to work with an external “partner” that brings a technological innovation or apparent engineering competence not found in the existing team. From there the work falls to managing the “partner” and crafting multi-page decks for the regular executive update meeting, where all focus is on delivery, and very little on customer value creation. The product teams all fall into a rhythm of meeting their individual roadmap’s obligations, with little or no cohesion or coherence when it’s rolled up.

    Management should not be confused with Leadership. Management of a great product team should be about what I discussed in the previous section: how you approach talent acquisition, talent development and coaching, and the goals and objective outcomes by which talent will be measured. (yes, the financial business metrics flow from this, your org is your product)

    Leadership, on the other hand, begins with the transition point from well-managed team development to the grooming of trusted teams that are empowered to solve the problems that stand in the way of them creating customer value in line with the needs of the business. Leadership is the arbiter of how trust is developed within great product teams.  And there is a straight line between a carefully crafted description of the company’s vision (or the product vision in bigger companies) and how product strategy and priorities are executed. Leadership sets the tone, provides the purpose and the critical points of alignment, and establishes the operating culture; that word – trust – looms large in the responsibility set of leadership.

    And great leadership, in great product companies, then becomes the number one cheerleader—gets out of the way, empowers, and evangelizes. Even better, a hallmark of great leadership is ensuring credit goes where and when credit is due.

    In the case of our exemplar client, again without benefit of being “on the shop floor”, we got a keen sense of teamwork and trust. Leadership providing the framework for what’s important, the value of the mission, and the goal for creating customer value and how it aligns to the vitally important strategic objectives for the business. Yes, amazingly competent.

  • The Netflix of Education, ad nauseum

    To get the model right, we need to inspect the value chain more closely.

    It’s been a COVID- and US national politics-blurred couple of months since I last posted here. Much has transpired in those months and it’s clear that nothing in education and EdTech will be the same as it was, and yet we still have so much to learn about how the sector will evolve. If you’ve been following Michael’s posts, you know we have our fingers in some things and remain excited about the possibilities for large-scale, positive impact with the next wave of technology-enabled teaching and learning.

    I have to say, this “Netflix of Education” thing has legs. The typical half-life of “X of Education” comparisons is roughly 42 days. Yet after a spate of comparisons by McGraw-Hill, Pearson, D2L and Udemy a few years ago, we have at least two notable players claiming the specific comparison again in recent months.

    I get that Netflix is a media company, some call it a TV station, or a video on demand platform, and therefore media…content…education, you know, they are kinda related?

    But what is driving serious companies and executives to use the phrase in often remarkably different contexts? What’s driving the interest in aligning a business value proposition to that of Netflix? I thought about a couple of different ways to examine the question and decided on an easy framework of “they either mean it literally, or they mean it figuratively”.

    Let’s look at both cases.

    Literally

    If taken literally, it should mean the company is proposing some kind of platform for a direct to student streaming service that is offered by subscription. Netflix is inherently a consumer service, so I will include that in the literal definition This would allow learners to access a wide variety of “learning” on internet connected devices. And ostensibly the learners would receive “personalized” lists of additional learning to consume, you know, to increase | improve | extend their learning.

    Certainly, Cengage has referred to their Unlimited subscription service as a “Netflix” model. For 4 months at a time, students can get access to any of Cengage’s eTextbooks for $69.99, or upgrade to get all the online homework platforms as well for $119.99. But the adoption is tied directly to courses the students are taking. I believe the rule of thumb is if a student is taking at least two courses that use Cengage materials, they benefit economically. The math for the bundled access seems to work well for some students, as Cengage reports continued, albeit slowing, growth of Unlimited adoption. But it is not an obvious no-brainer, as half of their digital activations are not under the Unlimited umbrella.

    Like Netflix, this is uber transactional. I don’t know whether some students are trying to self-substitute Cengage products for their other non-Cengage-using courses, or if others are simply hoovering up loads of knowledge, but I do know people who flip from one streaming service to the next to get what they want, when they want it. This is far from a lifelong relationship.

    Looking at Pearson’s shifting language over the last few years, the Netflix reference was originally about “one platform to rule them all” inside a company with more platforms than sharks have teeth or Netflix has TV titles. Michael covered this at the time and was called into the CEO’s office for a word, so I won’t retread that, but the magical platform has yet to appear at any real scale across different “learner ages and stages”. It definitely has not provoked visible innovation, new product launches, or helped reposition Pearson in the market four years hence.

    Recent language from the new leadership at Pearson gives a nod to Netflix but wisely eschews a full embrace. Nonetheless, they are very open about the direct-to-consumer relationship opportunity they see and are organizing the business very directly in that manner, referring to it as the “direct-to-learner” strategy to have a direct relationship with millions of learners globally.

    The challenge here, well-trodden but worth repeating, is education is not intrinsically a consumer market. Higher Education today is pre-dominantly heavily mediated by accreditation, institutional cultures and individual teacher/faculty choice. The same holds for K12 with less teacher choice. Students choose their course. Students do not choose their course material. The course platforms do not, and should not, offer “…other students who used Product X, also liked Product Y” cues.

    Furthermore, what is true about education, and not media, is that high quality, rigorous, and certifiable education essentially requires a human known as an educator. The learner in this equation is not simply streaming content but interacting with it and getting cues from the educator. Simply providing “consumers” with a playlist of educational content will not be the winning strategy. Education is something that is done by the learner, actively facilitated by the educator, rather than “Netflix and Chill”. (Dude, did you see that chapter of Campbell’s Biology last night? Those B-heads, Woah!)

    And yet furthermore, a direct-to-learner strategy begins to position Pearson as a competitor to its own customers today. In other words, a direct-to-learner, learn-on-demand AWATAD (anywhere, any-time, any-device) strategy inserts Pearson directly between institutions and students. This is true even if Pearson’s strategy is more oriented around supplemental, or strictly skills-oriented learning, because all of the institutions are busy trying to pivot to similar offerings. Indeed, for Pearson to make that move, you also have to believe the broader sector realizes tectonic level breaks in long-established norms like accreditation. Otherwise, the quantity and quality of demand is simply too unpredictable, as the MOOCs know all too well.

    There’s another likely outcome of Pearson moving to a direct-to-learner model: to achieve this strategy, Pearson needs to “spin-off” the Higher Education business unit. Doing so takes some of the customer-cum-competitor stressor off the table. Plus, it fits well with the same thesis that drove the separation of the K12 publishing unit; that the old-line content publishing franchise doesn’t reflect the simpler, digital-first business model and experience brand dynamic of the “new” Pearson. Selling the unit might be a tall order given some of the declining metrics like market share that might make it difficult to operate stand-alone. And the post-secondary textbook publishing space doesn’t seem to offer many options for classic roll-up private equity plays as the recent CL/MHE saga bore out. But it would lighten up the mother ship to pursue a dramatically different strategy!

    Figuratively

    So maybe “the Netflix of education” means something less literal. What could it be? Let’s start with culture.

    Seven Aspects of Netflix’ Culture:

    • Values are what we value – there are 9 behaviors and skills that are valued: judgement, communication, impact, curiosity, innovation, courage, passion, honesty, selflessness
    • High Performance – rigorous talent management
    • Freedom & Responsibility – empowered teams
    • Context, not Control – no top-down autocracy
    • Highly Aligned, Loosely Coupled – clear on strategy and key objectives, degrees of freedom to get things done
    • Pay Top of Market – and sever generously and unapologetically
    • Promotions & Development – feed the talent virtuous cycle

    Nope. Based on my own experiences and observations, the aspects of Netflix’ culture don’t completely fit with the publishers. Granted, it’s easier to foster a culture based on context, not control, keeping alignment with loose coupling, and empowering teams when you are not excising billions of dollars of cost out of the business. So that can’t be what they mean.

    I will offer that the most applicable figurative interest for any business to compare itself to Netflix is its earnings multiple, currently 4x Pearson’s.

    A way forward for the Netflix of Education?

    Ok, enough of that, I do want to press ahead on the question. Is there actually a way for in which a “Netflix of education” could work and create value for the sector? I think so, but it requires a far less literal analogy and must express a deep understanding of the complexities and nuances of the value chain and affordances necessary to deliver education.

    A drive by of Ben Thompson’s terrific Stratechery.com site gives a good framework for how we might think about this in the context of aggregation theory. The idea is straightforward:

    The value chain for any given consumer market is divided into three parts: suppliers, distributors, and consumers/users. The best way to make outsize profits in any of these markets is to either gain a horizontal monopoly in one of the three parts or to integrate two of the parts such that you have a competitive advantage in delivering a vertical solution. In the pre-Internet era the latter depended on controlling distribution.

    But the internet has disrupted distribution of digital goods by making it essentially free and neutralizing the integrated supplier-distributor advantage. The internet also drives transaction costs close to zero, making it far easier for a distributor to integrate forward with end users/consumers at scale. Thompson goes on:

    Instead, suppliers can be commoditized leaving consumers/users as a first order priority. By extension, this means that the most important factor determining success is the user experience: the best distributors/aggregators/market-makers win by providing the best experience, which earns them the most consumers/users, which attracts the most suppliers, which enhances the user experience in a virtuous cycle.

    So, for the Netflix of Education to work, the first issue is proper identification of the end user. I posit the most effective model suggests that is the educator. The educator decides what is taught, what material is used when, the manner with which that material is to be interacted with the learners. And the educator governs the assessment of the learner, which routinely involves feedback and guidance. All of this holds largely true even with adjuncts. To get the model right, we need to recognize educators for what they are: consumers, content creators and curators, and distributors. Thus, the Netflix for Education would need to embrace the content creators broadly and allow for large-scale syndication and sharing of their work and practice.

    A key reason Netflix works on the technical level is there are standards for delivered video content and they in turn manage bit rates and device specific formatting to ensure the AWATAD user experience. No such set of standards exists for education content and assessment. Vendors/providers are notoriously restrictive with their copyright IP and black-box with their delivery and assessment technology. And content stranding is a common fear for early adopters of adaptive platforms. So, the Netflix of Education would need technical standard models that support content transparency and portability beyond what is currently available today.

    And Netflix’ rise was also due to the focus on a great user experience as Thompson suggests. Since virtually all of the publisher products support only a fraction (typically well less than half) of a course on syllabus- or portion-of-grade basis, the teaching faculty are truly the final-mile content creators and curators. They drive the pedagogical interaction model with the learners. They determine what matters in the “preferences” of the learner. The Netflix of Education would need to afford them much more control over that learner journey personalization and experience, regardless of modality.

    Seamlessly delivering interactive, instructor-mediated learning experiences at scale is a far more complicated problem than seamlessly delivering non-interactive video. The fragmented ecosystem we know today lacks the depths of standards and alignment of business models to create the frictionless world we might imagine when we invoke Netflix. That doesn’t mean it is impossible, it simply means there’s a lot of work to do. It doesn’t mean it isn’t worth trying to do, it just means we need to consider where the value accrues.

    More. Soon.

    Disclosures: Pearson is a current sponsor of the Empirical Educator Project. Curtiss is a prior Pearson and Cengage employee. Michael is a prior Cengage employee. Neither Curtiss nor Michael have been employed by Netflix, but both are subscribers. Curtiss’ oldest daughter occasionally refers to a (non-Campbell) Pearson biology textbook while attending college. All of Curtiss’ daughters are highly active and likely lifelong consumers of his Netflix subscription. Michael has chickens.

  • A Next-Generation Open Source Courseware Platform Collaboration

    I started the Empirical Educator Project (EEP) two and half years ago based on the idea that higher education could greatly accelerate progress in technology-enabled education if only they could get better at sharing their projects and collaborating on addressing their common or complementary needs. Since then, I’ve been able to write about increased sharing facilitated through the EEP network but not much about collaborations.

    Today I have some massive news to share on that front. Carnegie Mellon University’s Open Learning Initiative (OLI) and Arizona State University’s Center for Education Through eXploration (ETX) are collaborating on a next-generation, open-source, open-standards platform for designing and delivering courseware.

    That’s a lot of words, yet it doesn’t really convey the heart of the work. There is much to unpack here.

    In this first post of a multi-post series, I will describe the drivers and origins of the collaboration itself. The project is an evolution of contributions from these two groups that I have previously written about here in e-Literate. In the next post, I will describe some of the key architectural concepts of the project. And in subsequent posts, I will explore some of the implications.

    On the nature of academic collaborations

    Before I get into the details of the project, it’s worth taking a moment to talk through some lessons learned about fostering EEP-style collaborations. First, they take time and energy to bring to fruition. While EEP generated a lot of excitement from the beginning simply by helping academic groups discover colleagues in different organizations who were doing complementary work, helping these initial connections to evolve from casual cross-fertilization to deliberate and sustained collaboration takes a lot of time and hard work on everyone’s part. This is not a surprise. But I am learning a lot about the specific kinds of facilitation necessary to bring collaborative projects to fruition. (This topic probably deserves its own blog post at some point.)

    Second, one of the complexities that need to be considered is the internal diplomacy that each stakeholder has to manage. Note that the partners I have listed here are ETX and OLI, not ASU and Carnegie Mellon. Likewise, the title of this blog post is not the much sexier “ASU and Carnegie Mellon collaborate on…”. Both institutions are rightfully protective of their respective brands and how their involvement is characterized in various projects. And that’s just the tip of the iceberg. Each participant has to manage a whole host of intra-institutional concerns that affect when and how they talk about such collaborations publicly.

    The collaboration I will be writing about in this series has been underway for some time and is quite advanced at this point. Yet I could not write about it before now without complicating important conversations that were (and still are) proceeding inside these institutions. Even now, this blog post is not an official press release and should not be considered an official statement from either university. I have asked the major project leaders at each institution to review my first two blog posts in the series to help me characterize the project appropriately. That said, if and when you see a press release jointly issued from the two institutions, you should consider that more official statement to be another milestone in the evolving relationship.

    With those caveats out of the way, I’m going to walk through the process by which contributions led to a ground-breaking collaboration. In the process, I’ll provide a first sketch of exactly what that collaboration is.

    Carnegie Mellon’s OpenSimon contribution

    In the spring of 2019, Carnegie Mellon University used the EEP summit that they hosted to announce OpenSimon, a collection of open-source projects built over the years by their faculty to advance their research and application of applied learning science—or, as they call it, “learning engineering.” The investment these projects collectively represented was well north of $100 million in grant money, staff time, etc. This was an official, university-sanctioned contribution. They issued a press release. University Provost Jim Garrett kicked off the announcement at the summit.

    People at the EEP summit and the in wider EdTech world were simultaneously excited and a little bewildered. The toolkit provides a sprawling array of functionality in a variety of areas. Some of its components tilt more toward research, while others are immediately useful in teaching applications. It was hard to know where to look for the low-hanging fruit. Even for me.

    One of the crown jewels in OpenSimon is the OLI software, which is developed and maintained by the eponymous OLI team. Originally conceived almost 20 years ago, the OLI platform set the template for modern mainstream courseware while the research on the courses built on that platform established some of the earliest and most robust evidence we have for the effectiveness of this kind of teaching tool. Without OLI, there probably would be no Pearson Revel, no Cengage MindTap, no McGraw-Hill Education Connect or SmartBooks, Wiley+ or Knewton, and so on. Today, OLI hosts somewhere in the range of 50 Carnegie Mellon-designed courses for over 50,000 formal course enrollments annually, not including courses and lessons created by faculty at other institutions or the many students who use OLI courses for self-study outside of a formal course adoption.

    My coverage of OpenSimon to-date has left out one critical update. OLI is in the process of rebuilding the platform, incorporating lessons learned from the past two decades in how to design such a system and reimplementing the platform on a modern architecture that takes advantage of the cloud services and technology stacks available in 2020. The new platform, code-named “Torus,” has already been used during the university’s 2020 summer school program in which students practiced building simple course modules. OLI plans to migrate the first tranche of their courses in production onto the new platform by the fall of 2021 and the remainder by the fall of 2022. In other words, within 18 months, this next-generation platform will be hosting all of the 100,000+ student enrollments in production.

    As with the other OpenSimon projects, Torus is being released as an open-source project under the MIT license.

    ETX’s courseware interoperability initiative

    About a month after the OpenSimon announcement, I was approached by Ariel Anbar, the ASU Presidential Professor and Director of ETX. I have been a long-time admirer of Ariel and his ETX colleagues, having written about their Habitable Worlds course as far back as 2013. From early on, ETX used the Smart Sparrow courseware platform to create their immersive digital lessons and courses in a pedagogical style that they call “guided experiential learning.” When I ran into Ariel at the 2019 ASU+GSV conference, he had recently received a $2.5 million grant from the Department of Education to build OER courseware that follows this pedagogical style on Smart Sparrow. But he foresaw a problem. While his content would be released under a Creative Commons license, the platform required to run it was proprietary. To his mind, this violated the spirit of OER.

    Furthermore, having worked very closely with Smart Sparrow to build over $15 million in grant-funded content already (from funders including, NASA, NSF, and the Bill and Melinda Gates Foundation), Ariel recognized that all such platforms were hurt more than they were helped by that lock-in. Without a wealth of content that could be “played” on a proprietary platform, the companies that make the platforms will always be financially at-risk and therefore risky to use. If the stand-alone courseware platform product category is ever to become sustainable, it needs an ecosystem of interoperable content.

    Ariel hired me, under the auspices of EEP, to convene a coalition to draft interoperability standards for courseware content that could be submitted to a standards body as a first draft of an official specification. In doing so, ETX was offering to make an EEP-style contribution. They were extending their project in ways that advance the public good by enabling other educators and educator-supporting companies to benefit and contribute.

    As I wrote about in a blog post late last year, our first meeting included OLI, Lumen Learning, Smart Sparrow, and CogBooks as potential platform implementors, with ETX acting as the convener and some allies at Maricopa Community Colleges who were involved with the grant providing an access-oriented academic perspective. ((At the time, Lumen Learning and Smart Sparrow were both EEP sponsors, and Carnegie Mellon was our 2019 academic sponsor.)) The group met in September 2019 to workshop the scope and basic structure of several kinds of interoperability standards. IMS Global CEO Rob Abel offered his blessing to the project and some free consultation hours from one of his engineers. The group had the outline of an action plan shortly after our first meeting and was off to a promising start.

    Fate intervenes

    Shortly after the interoperability kick-off meeting, I learned that Pearson had acquired the assets of Smart Sparrow. ((Disclosure: Pearson is a current EEP sponsor.)) They wanted Sparrow’s next-generation version of the platform for internal projects and did not intend to continue supporting the product for existing customers. It’s not my place to characterize the relationship between Pearson and Sparrow’s customers regarding this transition, other than to make the observation that it seems to be proceeding in a supportive manner that has satisfied the Sparrow customers I have spoken with. But the most important point for the ETX-led interoperability project is that the abstract concern it was intended to address suddenly became concrete. ETX had over $15 million of content development efforts invested in Smart Sparrow courseware and plans for over $10 million more. They were already supporting somewhere in the neighborhood of 50,000 student enrollments a year (many of which came through their Inspark network). They needed a way to move their content to a new platform that would preserve the pedagogical intent and learning design. It wouldn’t be enough to move the assets—the text, videos, quiz questions, and so on. Those assets were assembled in particular ways to create specific and differentiated learning experiences for students. ETX needed a way to preserve their past and future investments in guided experiential learning.

    We needed to accelerate our timeline by not only designing but also implementing interoperability with at least one of the partners in the collaboration. Of these, OLI turned out to be the best fit in terms of timeline and development ambitions. Both groups needed to implement new platforms in roughly the same time frame. (ETX needs to migrate its content off of Sparrow by June of 2022, although the goal to finish the migration sooner.) Both groups were highly collaboration-minded and open to cross-pollination. At first blush, the fit seemed good.

    (CogBooks has also been involved in the interoperability discussions, although I’m not at liberty to characterize their past or prospective involvement at the moment other than to give them credit for coming to the table and investing effort into early interoperability prototypes.)

    But there were—and still are—challenges to be faced too. First, the alignment of the two groups’ timelines is a double-edged sword. On the one hand, it is good to have collaboration partners with well-defined requirements and similar delivery schedules. On the other, neither group could afford to allow the time cost of collaboration to throw off their respective delivery schedules. Both ETX and OLI have responsibilities to internal and external stakeholders that they must meet. Further, Sparrow deviates from the course design archetype that OLI created more substantially than many other platforms in the market. The architecture of the new platform would have to accommodate divergent design approaches. And the two teams did not know each other or their respective work very well.

    The fit looked promising. But the project would face substantial challenges.

    The collaboration comes together

    ETX engaged Unicon—another EEP sponsor—to help with the software development on their side. The first step was to prototype a harness for running ETX lessons outside of Sparrow in a way that would preserve the integrity of the learning design and could be launched from courseware platforms using a variant of the IMS LTI interoperability standard. Unicon has completed several increasingly sophisticated versions of the harness. ETX also continued to engage me—again, under the auspices of EEP—to help facilitate the growing collaboration. ETX and OLI respectively invested significant amounts of internal staff time to articulate requirements, think through architectural designs, align road maps, and get to know each other. Needless to say, this took time and good will on all sides.

    The collaborators have reached a level of mutual confidence that the project has been greenlit. They plan to expand OLI Torus to accommodate the authoring and delivery of Smart Sparrow-style—and, more importantly, ETX-style—courses. Both organizations expect to support substantial numbers of students and faculty in courses migrated from their respective legacy platforms by the fall of 2021, with ongoing work continuing into 2022. Both organizations are making substantial engineering investments in the project; OLI through its internal development team and ETX through its contract with Unicon (as well as ETX-internal project management and learning design subject matter expertise). They are collaborating on project management, and both have their (respectively formidable) learning design/learning engineering experts involved with the design. ETX also continues to contract EEP to facilitate the collaboration and help keep the two universities’ product plans and goals aligned and as complementary as possible. Torus will be released under a commercial-friendly open source license. Further, it will act as a reference implementation for draft courseware interoperability specifications. (Again, these last two sentences may come across as bags of words to you, but they are vital to the project’s ambitions. I will unpack them in subsequent blog posts in this series.)

    This is an incredibly ambitious project with many miles to go (and many critical decisions to make) before it is complete. But to be clear, it is happening. Resources have been committed. Development is well underway. Several interoperability prototypes have been built according to increasingly detailed specifications. And the innovations that are resulting from the collaboration are considerable. The current generation of courseware platforms is a collection of highly idiosyncratic applications that were developed in relative isolation, are generally fairly old, and are often maintained by organizations with challenging or problematic business models. This project is gaining insights by comparing two very different and very innovative platforms in a collaboration between two very different but very accomplished academic course design teams with respective histories of supporting faculty and students at a reasonably large scale across a diverse range of learning contexts.

    It is thrilling.

    In my next post in this series, I will describe several of the architectural and design innovations that have emerged from the collaboration so far.

  • Reminder: This Week’s Blursday Social with Phil Hill

    Phil and Michael, bustin’ some moves.

    Gang, don’t forget that this week’s Blursday Social guest is Phil Hill. Blursday Socials are conversational, so we’ll all talk about whatever the folks who show up want to talk about. But here are a few topics on my mind for a conversation with Phil:

    • Curricular materials trends: Phil recently updated his cost analysis of curricular materials. But there’s some weirdness in the numbers due to the convoluted way in which publishers bundle products. I’d like to tease out what we know about adoption rates and pricing of courseware and digital homework products.
    • Online education trends during COVID: Phil has been tracking the enrollment numbers that we have so far and sifting through them to find trends among types of institutions, types of programs, etc. I’d like to get his latest take on what we do and don’t know, as well as what numbers he’s watching for.
    • The resurrection of Blackboard…? While I care a lot less about LMS market share these days, I am interested in why different companies are doing better or worse and what that tells us about what campuses value in their LMS and LMS vendor these days. Phil recently wrote about a big win in the US for Blackboard. I want to know more.

    But most of all, I want to catch up with my old friend in the company of other friends, probably with a beverage in-hand. You should come. It’s fun.

    Remember, Blursday Socials are much more conversation and participation than broadcast. Think of it as being like trivia night at your favorite bar. There will be some organized activity, but a lot of it will be just connecting with friends. Some folks will be regulars, others will be semi-regulars, and still others will drop in occasionally. They start every Thursday at 4 PM Eastern Time. The bar closes at 5:30. 

    The dress code is casual; BYOB.

    I recommend that you also subscribe to the iCal feed, which you can do by going to this page and clicking on the “iCal Feed” button on the bottom. I have events tentatively lined up for the handful of weeks, which I will populate to the calendar as they are finalized.

    Phil Hill’s session is Thursday, November 12th at 4 PM ET. RSVP here.

  • The Anatomy of a Scaled Digital Seminar

    In my last post, I argued that scaling the seminar is an equity issue:

    [S]caling the seminar is essential for equity if we want to avoid a two-track system of education—one at expensive universities and another for the rest of us. This is already a serious problem that the current situation is likely to make worse.

    Scaling the Seminar

    I also argued that we are experiencing a failure of imagination about how to scale the seminar because EdTech has become obsessed with solving the wrong problem:

    We’ve been pursuing the solutions that have helped improve student self-study by providing machine-generated feedback on formative assessments. While these solutions have proven beneficial in some disciplines and have been especially important in helping students through developmental math, they have hit a wall. Even in the subjects where they work well, these solutions usually have sharply limited value by themselves. They work best when the educators assigning these products use their feedback to reduce (and customize) their lectures while increasing class discussion and project work.

    Despite the proliferation of these products, flipped classrooms and other active learning techniques are spreading slowly. And meanwhile, there is a whole swath of disciplines for which machine grading simply doesn’t work. This most broadly applies to any class in which the evaluation of what students say and write is a major part of the teaching process. EdTech has been trying for too long to slam its square peg into a round hole.

    Scaling the Seminar

    In this post, I will outline an approach for scaling social feedback and peer assessment as the means for scaling the seminar. I will use the example of an English composition course for two reasons. First, I recently had the pleasure of presenting at a virtual workshop on the topic hosted by the University System of Georgia and supported by the APLU. So I have already done some thinking about this particular topic. Second, English composition is one of the hardest seminar-style classes to scale. Students need frequent and intensive feedback. And particularly in access-oriented contexts, many of these first-year students do not come to college with the skill sets necessary to provide high-quality writing-oriented peer review. So it’s a good test case.

    Setting the goalposts

    Let’s be clear about the boundary between ambitions and fantasies. I do not believe it is currently possible to scale English composition courses with quality in even middle-of-the-road access-oriented colleges using any method that I can think of. We do not have a viable alternative to frequent feedback from an expert human to teach students that don’t have a good grasp of grammar, are weak at constructing arguments, don’t know how to cite evidence, and have not learned how to give and take feedback on writing.

    I do believe that a technology-supported approach to social feedback and peer assessment can potentially accomplish the following goals:

    • Improve the outcomes in access-oriented composition courses by supporting effective pedagogical models and evidence-based practices
    • Make writing-across-the-curriculum programs more practically implementable.
    • Scale effective seminar-style classes in cases where students are better prepared for them (including second-year students at access-oriented institutions who have had a year of experience in a writing-across-the-curriculum program).
    • Provide scaffolding for faculty in all disciplines to “flip” their classes more easily and effectively

    The basic rationale is that social feedback and peer assessment enable groups to learn and work more effectively regardless of the knowledge domain. These two functions are at the heart of seminar-style pedagogy. Anyone lucky enough to have taken a great seminar on any subject will likely remember the fantastic conversations among the whole class. There may or may not have been lectures. But whenever content is provided to the students—as an instructor presentation, a reading, or something else—the point is generally to catalyze peer-to-peer conversation and learning.

    Equally importantly, these skills transfer to the workplace. With all the buzz over the years about preparing students for the future of work, we fail to recognize that the future of knowledge work—a future that has already arrived—is a workplace environment designed to facilitate social feedback and peer assessment in a project-based learning context. For example, in software development, Agile software development is all about improving software quality by baking social feedback and peer assessment into the day-to-day work of building software. The very first tenet of the Agile Manifesto—which only has four tenets in total—is “individuals and interactions over process and tools.” Building novel and complex software always requires a learning process. It’s a major reason why development timelines are notoriously unpredictable. Teams learn as they build. They discover unanticipated problems to which they have to invent solutions. Older software development methods perpetually led to disappointment because they failed to recognize that knowledge work entails learning and discovery. Waterfall development was optimized for predictable schedules. It failed at that goal. In contrast, Agile software optimizes for team learning and responsiveness in the face of unpredictability. It has a much better track record.

    Today, Agile software development is used to create and improve many of the largest and most complex software applications in human history. These projects succeed by deploying—and continuously fine-tuning—social feedback and peer assessment at scale in a project learning-based context. The rule of thumb with Agile software development teams is that they are usually seven people on average. Larger projects are scaled by creating more teams, not larger ones. The various Agile methodologies all emphasize team roles, regular check-ins, and frequent constructive feedback from teammates. Agile development is organized in sprints, which are short periods of time—typically two weeks—in which teams attempt to meet very specific project goals that were defined at the beginning of the sprint. (You can think of them as project-based learning objectives.) At the end of each sprint, teams conduct “demos” to bring in important stakeholders and get peer review on what they accomplished. They also conduct “retrospectives,” in which the team discusses what went well and what went poorly in each sprint, with a goal of identifying ways to improve their craft in future sprints. They might conduct formal exercises like “the five whys” which are designed to improve the quality of social feedback.

    Agile can teach us a lot about what it takes to improve both the quality and the scale of purposeful collaborative learning work. Experienced practitioners will tell you that Agile is easy to explain but hard to do well. Managers of development shops often need years of practice and regular coaching from experts to cultivate high-performing collaborative teams reliably. And importantly, many Agile teams do use specialized tools to support their collaborative learning journey. The Agile Manifesto may prioritize individuals and interactions over processes and tools, but most Agilistas are all for processes and tools designed to support and scaffold individuals and interactions.

    You may even be familiar with one or two of these tools. Trello—a collaborative brainstorming and planning tool with a whole section of its website devoted to educators—was originally developed as an Agile software development tool to simulate sticky notes moved around on a whiteboard with swim lanes drawn on it. It is owned by Atlassian, which is probably the single best-known company for creating products designed to support Agile software development.

    I’m not arguing that every class should ape Agile development methodologies. It’s the opposite, actually. Every Agile development methodology is an example of a project-based collaborative learning methodology. While we don’t need to fetishize software development, we certainly can learn from it. The first two lessons are (1) collaborative, active learning can be accomplished effectively at scale, and (2) doing so is hard, requiring lots of practice for everyone and good supporting tools.

    Individuals and interactions over process and tools

    That last bit—about the approach being hard and requiring lots of practice for everyone—is a buzzkill. The popular assumption is that that that neither professors nor students will work hard and persistently, especially if it requires them to change their habits. Rather than interrogating that assumption, the sector gets seduced into magical thinking. “Humans are too hard to teach. Why don’t we teach machines to do the work instead? That will be much easier!” Which, I must say, is not a terribly productive attitude for education in general. But it’s particularly bad for teaching difficult and uniquely human skills like good communication and critical thinking. Let’s take a moment to explore that dead end before moving on to the approach I’m proposing.

    As part of my preparation for the aforementioned workshop on scaling English composition, I took a look at the notes the group had been taking from previous sessions. They (inevitably) included a tools section that (inevitably) included a list of programs that auto-assess writing. To be clear, I use tools like these in my own writing. But I am vehemently opposed to their use in education as unsupervised assessment tools. These programs all regularly provide mistaken diagnoses and poor advice. I am an experienced enough writer to know when to ignore them. Many students are not. The students who are in most dire need of frequent feedback on their writing are also least well equipped to know when to ignore the software’s advice. And the software is often poorly equipped to handle second language learners or neurodivergent students with unusual language or communication challenges.

    To illustrate this point, I tested a few text examples with Hemingway, one of the tools listed in the workshop notes. For my first test, I asked Hemingway what it thinks of a sentence that Noam Chomsky famously created to illustrate the difference between syntax and semantics.

    Hemmingway likes grammatical but meaningless sentences.

    This sentence is perfectly grammatical and perfectly meaningless. Hemingway likes it.

    Maybe that’s an unfair test. Let’s try some text from Hemingway’s fellow giant of American literature and notorious macho man, Herman Melville:

    Hemingway thinks Moby Dick is bull. And not in a good way.

    Hemingway likes “Call me Ishmael.” After the first three words, though, things go downhill fast.

    So, maybe that’s not fair either. Let’s try some non-fiction. How about a great speech?

    Is Hemingway a hater?

    Hemingway is not a big fan of Abraham Lincoln either, as it turns out. It thinks the Gettysburg Address is “very hard to read.” And maybe that’s true. But I don’t think that Lincoln would have written a better speech had he followed Hemingway’s advice.

    At this point in my experimentation, I had given up on finding another great American writer that Hemingway likes. So just for fun, I threw it a Robert Frost poem:

    Hemingway likes poetry

    Huh. Hemingway was not thrown off by the line breaks of the poetry. Does it “understand” them? Or does it fail to “see” them at all? I don’t know.

    And this is the real problem with the machine assessment of writing. It’s not just that the software gets the analysis wrong a lot. That’s bad enough. But the bigger problem is that it gets its analysis inscrutably wrong a lot. How do you teach students when to trust an unreliable tool if you don’t know when it will be reliable yourself? The more advanced a textual analysis algorithm is, the greater the chances are that even the people who programmed it don’t always know why it makes the assessments that it does. This is because artificial intelligence software reprograms itself as it “learns” in ways that even AI experts can’t always follow.

    Machines are unpredictably unreliable writing tutors. And keep in mind that Hemingway is only trying to evaluate writing style here. Software that tries to evaluate the accuracy or coherence of a written argument is even more problematic.

    If your software cannot evaluate complex speech acts, it is probably useless at evaluating many types of complex thought. And isn’t complex thinking…um…what we’re supposed to be learning how to do in college? Might that be one reason that the seminar is considered the pinnacle of a high-quality college education?

    Collaborative feedback

    Throughout this post, I have been using the phrases “collaborative feedback” and “peer evaluations.” I consider them to be separate competencies, with collaborative feedback being a prerequisite of peer evaluation. These skills are building blocks for any kind of social learning pedagogy, including but not limited to the seminar.

    Collaborative feedback is essentially class discussion that actively managed by an educator (or facilitator if the exercise is in the workplace). Participants both practice both giving and receiving constructive feedback. In contrast to other styles of class discussion, collaborative feedback sessions are student-centric. The goal of the instructor is to encourage peer-to-peer conversation and then intervene primarily to help keep the participants on track and constructive. The the group has achieved competence when the instructor either starts to function as just another peer in the group or fades out of the conversation altogether.

    This is a common strategy for teaching English composition. By way of example, let’s revisit the Hemingway evaluation of the Gettysburg Address:

    Formative assessment? No. Collaborative feedback? Yes!

    Students achieve competency at writing collaborative feedback when they can have a constructive conversation about the writing passage without a lot of help. This takes time and practice. The instructor might start by modeling commentary. What does that first sentence mean? What would be a simpler way of writing it? Why did Lincoln write it that way instead of putting it more simply?

    Students are learning a whole cluster of skills in this exercise, including but not limited to the following:

    • How to ask good questions about writing quality (starting with, “Do I actually understand this sentence?”)
    • What intelligent answers to those questions look like
    • What constructive answers for the purposes of revision look like

    Writing instructors do this all the time. Sometimes they use passages from readings, while other times they use student writing samples. They don’t need a tool like Hemingway to do this work. But notice how it can help. It scaffolds some of the questions and calls students’ attention to sentences that deserve attention. With an educator present to facilitate, Hemingway could be an excellent aid for fostering collaborative conversation around writing. (If I were facilitating such a conversation using this passage in Hemingway, I might point out that one of the uses of passive voice is contained in a quote from the Declaration of Independence. So there’s an opportunity to teach students the skill of knowing how to use tools like Hemingway and when to ignore them.)

    In collaborative feedback sessions, participants are always learning and refining two sets of skills: one that is domain-specific and another that is general to collaborative work. It is not enough for participants’ answers to be insightful. They must also be constructive. Instructors are often scaffolding both of these skill sets. First, they model. Then, they facilitate. And finally, they step back and let the students lead.

    Scaling an access-oriented English composition seminar is incredibly difficult because it often takes an entire term to get students proficient enough at collaborative feedback that they will be ready for peer evaluation, which is the point at which social teaching methods can scale. (In fact, access-oriented writing classes in challenging environments routinely fail to get all the students across this particular finish line. It’s too much to accomplish in one course.)

    But I can easily see how both synchronous and asynchronous technologies could enable instructors to teach collaborative feedback skills more effectively. The Hemingway example I just gave is a good starting place. A good textual analysis tool could help the instructor rapidly scaffold many different examples for class discussion. The Riff Analytics tool that I wrote about in my Engageli first look post is another example. Riff may not be able to evaluate the quality of the student feedback, but it can provide feedback on the collaborative conversation quality. Some education-oriented discussion boards are starting to develop interesting affordances that also help students to cultivate this skillset. For example, Ment.io provides analysis of how close or far apart students are from each other in their opinions on a subject, based on textual analysis:

    Thought distance

    How accurate is this algorithm? I don’t know. But it’s fairly low-stakes. It simply encourages students to reflect on the dynamics of agreement and disagreement. Even if students ultimately decide that the algorithm is misreading a conversation, it still will have accomplished its purpose. These kinds of analytics can be used in a synchronous discussion, an asynchronous discussion forum, or a social text annotation tool.

    In fact, one of the most promising aspects of collaborative feedback EdTech affordances is their potential to become ubiquitous. They can be embedded in many different educational tools to help teach many different subjects and disciplines. All technology-enabled classes could include affordances that make facilitating rich and productive class conversations easier. As a result, instructors could be more likely to incorporate such discussions into their course designs more frequently.

    Peer assessment

    Once students are proficient in participating in collaborative feedback, they are ready to start learning peer assessment skills. To restate my earlier point, this is the milestone at which social learning pedagogies can begin to scale effectively. Why has EdTech pushed so hard on machine assessment? Because providing students with frequent, high-quality feedback is the bottleneck to scaling classes while maintaining quality. Machines have proven excellent at providing frequent and timely feedback to students, but often at the expense of quality. If students can be taught how to provide high-quality feedback to each other, then they can solve the assessment bottleneck more effectively than the machines can. Even better, the students will be learning essential workplace and life skills in the process.

    To be clear, the path that I’m proposing is hard. Every academic knows from direct experience that even people with PhDs are not reliably capable of providing good peer review (or receiving it, for that matter). Anyone who has tried to implement peer assessment in even a small classroom knows that it is a lot harder to pull off than giving a decent lecture. Going back to the Agile example, software development shops that transition to Agile methodologies usually need a couple of years and some active, persistent coaching from an expert to become proficient at it.

    To deal with this problem, a substantial majority of the rituals and artifacts in Agile methodologies are designed to improve these skills in the group. To cite another example, Agile teams will often “play the cards” at the beginning of each development sprint. A key part of reliable software development is accurately estimating how much effort—and therefore time—will be required to develop a feature. So at the beginning of a sprint, all team members may be given a set of cards representing different estimates for the amount of effort they think each feature will require. For each feature, the team members literally lay their cards on the table, comparing how much effort each person thinks a given feature will require. The goal is remarkably similar to the “thought distance” graph above. The team is trying to figure out the level of agreement about how much work is required. Each team member then has to defend her estimate. The team debates the merits of the various opinions and strives to arrive at a consensus estimate.

    We do have EdTech tools that facilitate, scaffold, and scale peer assessment. There is some evidence that they can be effective, although I would like to see more research in this area. One of the (relatively) better-known products in this category is Peerceptiv. It can help organize students into groups, scaffold their peer evaluation work, and then scaffold peer evaluation of how supportive and effective their peers were during the exercise.

    Putting your cards on the table

    Peer assessment can be enhanced using a number of techniques. The simplest one is to assign a small but meaningful percentage of students’ grades based on the feedback quality scores that their peers give them. In other words, reward students for giving feedback that their peers find useful. Second, the instructor can selectively hop from group to group and act as a participant in some peer reviews (particularly if these reviews are asynchronous). She doesn’t have to do this with every group every time. But having an instructor present and modeling quality peer evaluation from time to time helps a lot. She’s essentially acting like a coach, moving from team to team. And finally, we can some of the same technological aids that I described in the previous section to continue acting as supplemental “guides on the side.”

    One of the harder parts of this approach is designing good rubrics that students can be taught to apply consistently and effectively. The upside is that, if they are designed well, they can often be reused in various contexts. For example, I can imagine a shortlist of writing evaluation criteria that are included in every rubric template. To the degree that non-writing instructors adopt the peer evaluation tool and template, they will automatically implement a basic writing-across-the-curriculum program. Even if they are uncomfortable evaluating students’ writing, the students themselves will coach each other.

    Making the transition

    My proposal is ambitious. It likely will work best when applied consistently in different courses across different programs. Faculty would have to redesign their classes and think differently about their pedagogical approaches. Much of the technology required may exist but has yet to be stitched together into a coherent end-to-end experience that supports this approach. And we don’t have a lot of research on how well these strategies will work, especially in concert in a technology-enabled context.

    But the raw materials exist. Research on the effectiveness of collaborative learning exists. Various technologies that could scaffold, support, and scale the pedagogy exist. And I reject the notion that instructors or students would not be willing to put in the work required en masse. The truth is that a discussion-based seminar in which ideas flow freely is almost the Platonic ideal of education for educators and students alike. A good seminar is a joy for everyone involved. We should not abandon the idea that we can provide this kind of opportunity for all students. Nor should we shirk our responsibility to teach students the skills required to participate in such learning experiences as an essential part of a college education.

    Nothing I have written here should be read as a blanket repudiation of machine assessment or EdTech that focuses on helping students with self-study. Those approaches are necessary for scaling educational access with quality.

    But they are not sufficient.

  • Scaling the Seminar

    Scaling the Seminar

    In my recent first look at Engageli, I wrote about the importance of scaling the humanities seminar. The short version is that budget pressures will force universities to make cuts to programs that are the most costly to run. Since STEM programs tend to generate more grant money and social sciences programs can often teach at least the lower-level courses in large numbers, the humanities are vulnerable. The pedagogy of the seminar constrains the size of even the 100-level composition classes that all students take (in contrast to the vast majority of other 100-level courses). We are not good at scaling seminar-style courses with quality yet. Consequently, programs that rely on seminar-style pedagogy are vulnerable during hard economic times. Faculty are more likely to be let go, and programs are more likely to be cut.

    This action can create a vicious cycle because teaching these disciplines with quality currently depends on having sufficient faculty. The more faculty are cut, the weaker the program becomes. The weaker the program is, the fewer students it attracts. The fewer students there are in the program, the less financially healthy the program becomes. As a result, cutting into a program can easily lead to the program’s eventual elimination altogether. While specious rhetoric around humanities degrees not being career-relevant doesn’t help, the real pressure on humanities departments will come from the institution’s per-student cost.

    These are good reasons to find ways to scale the seminar, but there are others. In fact, I’m going to argue that scaling the seminar is essential for equity if we want to avoid a two-track system of education—one at expensive universities and another for the rest of us. This is already a serious problem that the current situation is likely to make worse. I’m also going to argue that it is not an oxymoron to talk about large seminars. I think it’s possible to approximate—and even innovate on—the pedagogical affordances of a seminar with significantly more students in one class section.

    Seminars shouldn’t only be for rich white dudes.

    Finally, I will argue that one major reason we haven’t arrived at this solution before now is that EdTech is stuck in something of a cul-de-sac. We’ve been pursuing the solutions that have helped improve student self-study by providing machine-generated feedback on formative assessments. While these solutions have proven beneficial in some disciplines and have been especially important in helping students through developmental math, they have hit a wall. Even in the subjects where they work well, these solutions usually have sharply limited value by themselves. They work best when the educators assigning these products use their feedback to reduce (and customize) their lectures while increasing class discussion and project work.

    Despite the proliferation of these products, flipped classrooms and other active learning techniques are spreading slowly. And meanwhile, there is a whole swath of disciplines for which machine grading simply doesn’t work. This most broadly applies to any class in which the evaluation of what students say and write is a major part of the teaching process. EdTech has been trying for too long to slam its square peg into a round hole.

    I am calling for a new research agenda. One in which we focus on scaling educational conversation and human-to-human engagement with as much energy as we have been investing in machine assessment. I am far from the first person to call for a focus on this problem. For a variety of structural and historical reasons, it hasn’t gained traction. But the situation has changed. Now is the time for such a research agenda. Now is the time for us to figure out how to augment and scale educator-facilitated pedagogy. We can’t just keep trying to automate assessment and hoping the rest will work out. Scaling the seminar will not be easy. But I will argue that the challenges are less daunting than those we face by continuing down the current path while the results could produce higher quality and more equitable education.

    This post is the first in a multi-part series.

    The gold standard

    If we’re being honest, the seminar is still considered the de facto gold standard for all disciplines. However much we may pretend that large lectures are acceptable in teaching some disciplines, the truth is that rich colleges and universities minimize them and market that lack of scale as one of the hallmarks of an elite institution.

    On the undergraduate level, one way—perhaps the dominant way—in which top-tier colleges and universities measure and express the quality of the education they offer is through their student/faculty ratio. This is a proxy for the seminar/large-lecture ratio.

    This pattern holds online as well. Take a look at the differences in the design of online graduate programs from top-branded universities versus those from older, more access-oriented programs. We can use 2U as a proxy for the former since they essentially built their company by persuading top-tier universities that they can build top-tier online graduate programs. 2U’s motto is “No back row,” and their course designs are heavily weighted toward synchronous classes of constrained size. 2U scales graduate seminars by offering many sections of the identically designed course with more junior (i.e., less expensive) instructors trained to teach the same course design. One way to express this strategy is that 2U is scaling the seminar “horizontally.” They are still offering normal-sized seminars but are pushing down the cost of incrementally adding more sections.

    One of the reasons they have been able to sell this approach to faculty senates that would have rejected a larger and more asynchronous design is that 2U has preserved the seminar’s basic structure. Keep in mind, though, that this model tends to work best with one-year degree prices in the $40,000 or $50,000 range. It incrementally improves the scalability of seminars. That’s not a criticism of 2U. It’s simply an acknowledgment that this approach to scaling does not bend the cost curve radically enough to solve the problems that I am calling out.

    The reality is that we have to substantially lower the cost of higher education in the United States while increasing the number of available seats in degree programs if we are going to meet equity goals. And we also have reasons to do so for institutional sustainability. If colleges can serve more students—with quality—at a lower cost per student, then we can achieve both equity and sustainability goals.

    This shift cannot work if it begins and ends with a mantra to “do more with less.” Access-oriented institutions are already doing more than they can sustain with less than they need to be sustainable. Rather, we need to create tools that are “force multipliers.” We can think of that metaphor in the military sense, i.e., increasing the effectiveness of the human forces we have on the ground, or in the physics sense, i.e., increasing the force that one human can apply by using a pully or an inclined plane or an engine. Technology externalizes techniques into tools that enable us to apply those techniques more effectively with less effort. EdTech should externalize pedagogical techniques in ways that enable us to apply them to help students learn more effectively with less effort per student on the part of the educator.

    I believe in the transformative potential of EdTech. I also believe that we have been thinking too narrowly about the kinds of tools we can create and how we can apply them.

    The cul-de-sac

    EdTech has generally approached scaling by trying to replicate the growth of large lecture classes, i.e., by scaling the machine scoring of assessments. On one level, this makes sense. We know that students do better when they get frequent and timely feedback, and we also know that providing that feedback is incredibly time-consuming. The feedback loop is the fundamental unit of teaching and learning. Students try something, see what happens, and adjust accordingly. Educators give students something to try, watch what happens, and adjust accordingly. The more rapidly and frequently students can get feedback, the more likely students are to learn, and the more rapidly they tend to progress. So scaling feedback loops—both the number that can be provided to each student and the number that can be supported for an entire class at one time—is an important problem to solve for both quality and cost reasons. Automating assessment is a straightforward way to scale feedback loops without burdening the instructor.

    Keep in mind that this trend started long before “EdTech” was a word. Universities figured out that TAs are less expensive to use as graders than professors and that Scantrons are less expensive than TAs. Universities were using Scantrons and similar machine grading technologies to scale assessment when I was a student back in the 1980s, at a time when personal email addresses were still relatively uncommon.

    Unfortunately, scaling via machine assessment is problematic. Some types of student output—like writing or comments in a discussion or projects—are difficult for machines to evaluate. To do even a passably reliable job, machine assessment inevitably flattens the very notion of feedback. This approach can work OK for helping students master foundational knowledge and skills that are low on Bloom’s Taxonomy but fall apart quickly when trying to provide feedback on the kinds of complex analysis and problem-solving skills that we associate with a college education. There are underutilized methods for pushing at the boundaries of machine assessment somewhat, such as educational games and inquiry-based courseware designs, but these approaches have their limits too.

    Think about the feedback that students get in a seminar. Classroom conversation is rich with feedback. In addition to direct feedback from both the instructor and fellow students, there’s indirect feedback that results from multiple people engaged in purposeful conversation and collaborative problem-solving. This high-bandwidth environment for learning feedback doesn’t easily lend itself to simulation or replacement by software-assisted self-study. Likewise, when evaluating student expressions of sophisticated ideas, even the most enthusiastic proponents of writing assessment software have to grant that today’s algorithms are left in the dust on the subtleties compared to even a mediocre human reviewer. And again, while writing and humanities are the most obvious cases, I’ve had math and physics professors tell me that they can facilitate a deeper understanding of even foundational concepts in a small, discussion-based seminar than they can in any other way.

    So while conventional machine assessment enables us to help more students achieve passable literacy levels in reasonably well-structured knowledge domains, it’s not consistently good at teaching critical thinking in these domains. And it’s virtually useless at teaching fluent expression or collaborative problem solving—two key skills for the modern workplace (not to mention the modern democracy, such as it is).

    The fundamental limiter to scaling the class is the human instructor, who has only so many hours in the day. Augmenting instructor feedback with machines the approach that EdTech has emphasized so far. But what if we could scale the student peer-to-peer feedback with quality? While instructors do not scale as class sizes grow, peers scale as the class size grows by definition. Instead of putting all our eggs in the one basket of trying to make computers provide feedback that is as good as humans, what if we focused more energy on getting students to provide feedback that is as good as the instructor’s?

    The new “blended” and the new “flipped”

    COVID-19 is teaching us a lot in a hurry about the quality of that high-bandwidth educational feedback machine that we call the classroom. And we are already beginning to see products like Engageli begin to respond to those lessons learned. We can begin to see a world in which those rich synchronous educational conversations can take place fully online or even in some random and unpredictable mix of some participants being online and others being in a physical room together. We can aspire to a new type of “blended” classes that mix synchronous and asynchronous experiences rather than a mix of online and face-to-face.

    As I wrote in my Engageli first look post, that would be great but not enough. In an environment where software mediates the face-to-face as well as the online synchronous class experience, the software could scaffold pedagogy in ways that a physical classroom cannot—for both the educator and the students. At the tail end of my Engageli review, I started to explore ways in which software can help improve the quality of human-to-human feedback. I used the example of Riff Analytics, which helps students to recognize whether they are speaking out, taking turns, and affirming statements made by their peers in group conversations. In other words, Riff helps students to become more effective at collaborative, purposeful, and equitable peer conversations.

    We can take this idea much further, particularly if we blend synchronous and asynchronous tools. The ideal we should be trying to replicate and improve on is not just the blended classroom but the flipped classroom. One of the perennial challenges with flipping the classroom is that facilitating consistently productive group work among students is incredibly hard. It’s a lot of work that sometimes fails regardless of the instructor’s skill and best efforts. It’s hard even in a small class when conducted by an instructor who practices it regularly. It’s really hard in a large class when conducted by an instructor who is mostly trained and experienced in a conventional lecture model.

    We shouldn’t be surprised at the stories of instructors who tried to flip their classes and had horrible experiences. We haven’t provided them with the tools that they need. With its competency-based analytics, courseware can help faculty prepare by giving them a sense of the foundational knowledge that students may be either mastering or struggling with. But by itself, that information is not sufficient for a successful flipped classroom. The hardest part about flipping a class is getting the groups to collaborate effectively and consistently. I can’t think of any widely adopted EdTech tools that are specifically designed to help with this part of the challenge.

    I think it’s possible to scale the seminar and active learning across most disciplines. I think we should put more energy into improving the quality of peer feedback rather than single-mindedly focusing on improving machine feedback. There is research, and there are products that can help to achieve this goal. Both have been chronically undernourished because all the glory (and money) has gone to machine assessment. But this historic moment we are all living is creating an opportunity to rebalance our efforts. COVID-19 has forced EdTech—and many educational institutions—to think about the richness of human-to-human conversational experiences with more clarity and specificity.

    In my next post in this series, I’ll describe how we might tackle this challenge using the hardest one of the courses to scale that I can think of—English Composition—as my example.