e-Literate

Present is Prologue

Author: Michael Feldstein

  • Major Changes at e-Literate and MindWires

    Major Changes at e-Literate and MindWires

    All good things must come to an end.

    After nearly seven great years as business partners and an even longer run as co-bloggers, Phil and Michael, a.k.a. the Statler and Waldorf of ed tech, are undergoing a process that Gwyneth Paltrow calls “conscious uncoupling.”

    (Is it weirder that we’re referring to ourselves in the third person or that we’re quoting Gwyneth Paltrow? Anyway…)

    We’ve had great fun together and hopefully done a little good in the world, but we each need to be doing different things at the moment. Phil will be keeping MindWires as well as the LMS Market Analysis subscription service. He will be forking to his new blog home at http://PhilOnEdTech.com, but all historical posts will remain at e-Literate. He will continue to cover ed tech and online / hybrid education market trends in general, and basically all the stuff that you’re used to reading about from him. Email subscribers to e-Literate will initially be subscribed to Phil’s blog but will be given clear options to opt out. That said, you should follow his blog. It’s going to be great.

    Michael is going to stay right here at e-Literate. His new business will help universities and education companies adopt the innovations that are being contributed by the Empirical Educator Project network and, more generally, helping organizations to improve their educational impact. In keeping with that new venture, e-Literate will be returning to its roots of likewise focusing on the nuts and bolts of education, only covering technologies, companies, and markets from angles that set the context for discussions of educational impact. Expect a blog post from him here on e-Literate with more details soon.

    This does not mean that you have heard the last from Statler and Waldorf. We remain friends and, more importantly, one of us owes the other money. (We won’t say who.)

    Keep your ears open. From time to time, you will still hear some cackling from the cheap seats.

  • EEP 2019 Will Be Live Audiostreamed

    Interest in the Empirical Educator Summit (EEP) has been off the charts. We want and intend to include everybody, but only when we can include people in a way that is useful to them. So we are being intentional about the pace and ways in which we are growing.

    That said, we know a lot of people are very interested. We had already planned to release video of much of the summit after the fact. We’ve decided that we’re going to try to live stream the audio as well. (My experience with live-streaming video is that there isn’t much value in the visuals unless your setup is better than we will be able to manage, so we’d rather focus on trying to get you a solid audio stream.)

    We have a placeholder page set up at http://empiricaleducators.net/2019-eep-summit/. Between now and Monday, we will be posting an agenda of the summit and putting up a widget for the audio streaming on that page. Check there periodically for updates. For planning purposes, I can tell you now that the audio streaming will be from 9 AM to 3:30 PM EST on Monday, May 6th and from 9 AM to 12 PM EST on Tuesday, May 7th. Again, the agenda will be posted on the EEP summit page soon. This is a last-minute addition driven by demand, so we’re winging it a bit.

    We also invite you to discuss the summit on Twitter as it is streamed. We will not have the luxury of a dedicated social media person to monitor and respond to the conversation live, but we will be encouraging the on-site community to participate and will definitely be looking at what you have to say afterward to see what we can learn from your input. The hashtag for the event is #EEP2019.

    We’re adding two more hashtags for more specific input, since EEP is ultimately about doing things together. If you use these, please be sure to catch the early sessions on Monday that explain the goals of EEP so that your input is on point. The first hashtag, #EEP2019ideas, is for suggestions about how EEP members—both current and prospective—can work together to accomplish the goals of the network. The second, #EEP2019challenges, is for obstacles you want us to be aware of as we think about how to build out the collaborative network.

    To prepare you for the streaming of the event, I’m going to assign you some homework. The main reading is very short. I just published a piece in Forbes about Carnegie Mellon’s contribution. It’s not what you’re used to reading from me in that Forbes required the piece to be only about 800 words and strictly enforced a requirement that readers shouldn’t need to have any knowledge of higher education or software whatsoever in order to understand the article. The downside of these requirements is that I had to flatten and truncate some details and nuances that e-Literate readers are used to getting from me. (One example that I particularly want to get off my chest is that I briefly described the fruits of Lumen Learning’s collaboration with Carnegie Mellon but wasn’t able to give them proper credit.) But there were some benefits to those restrictions too. I think the piece captures something of the sense of professional identity and culture that both Carnegie Mellon and EEP seek to foster. Also, did I mention that it’s probably the shortest piece by me that you will ever see? Go read it.

    Beyond that, if you want to get a deeper sense of the train of thought behind the effort, take a dip into the archive of EEP-related blog posts here at e-Literate.

  • Carnegie Mellon and Lumen Learning Announce EEP-Relevant Collaboration

    Late last week, Carnegie Mellon University (CMU) and Lumen Learning jointly issued a press release announcing their collaboration on an effort to integrate the Lumen-developed RISE analytical framework for curricular materials improvement analysis into the toolkit that Carnegie Mellon announced it will be contributing via open licenses (and unveiling at the Empirical Educator Project (EEP) summit that they are hosting in May).

    To be clear, Lumen and Carnegie Mellon are long-time collaborators, and this particular project probably would have happened without either EEP or CMU’s decision to contribute the software that they are now openly licensing. But it is worth talking about in this context for two reasons. First, it provides a great, simple, easy-to-understand example of a subset of the kinds of collaborations we hope to catalyze. And second, it illustrates how CMU’s contribution and the growth of the EEP network can amplify the value of such contributions.

    RISE

    The RISE framework is pretty easy to understand. RISE stands for Resource Inspection, Selection, and Enhancement. Their focus is on using it to improve Open Educational Resources (OER) because that’s what they do, but there’s nothing about RISE that only works with OER. As long as you have the right to modify the curricular materials you are working with—even if that means removing something proprietary and replacing it with something of your own making—then the RISE framework is potentially useful.

    From the paper:

    In order to continuously improve open educational resources, an automated process and framework is needed to make course content improvement practical, inexpensive, and efficient. One way that resources could be programmatically identified is to use a metric combining resource use and student grade on the corresponding outcome to identify whether the resource was similar to or different than other resources. Resources that were significantly different than others can be flagged for examination by instructional designers to determine why the resource was more or less effective than other resources. To achieve this, we propose the Resource Inspection, Selection, and Enhancement (RISE) Framework as a simple framework for using learning analytics to identify open educational resources that are good candidates for improvement efforts.


    The framework assumes that both OER content and assessment items have been explicitly aligned with learning outcomes, allowing designers or evaluators to connect OER to the specific assessments whose success they are designed to facilitate. In other words, learning outcome alignment of both content and assessment is critical to enabling the proposed framework. Our framework is flexible regarding the number of resources aligned with a single outcome and the number of items assessing a single outcome.


    The framework is composed of a 2 x 2 matrix. Student grade on assessment is on the y-axis. The x-axis is more flexible, and can include resource usage metrics such as pageviews, time spent, or content page ratings. Each resource can be classified as either high or low on each axis by splitting resources into categories based on the median value. By locating each resource within this matrix, we can examine the relationship between resource usage and student performance on related assessments. In Figure 2, we have identified possible reasons that may cause a resource to be categorized in a particular quadrant using resource use (x-axis) and grades (y-axis).

    Figure 2. A partial list of reasons OER might receive a particular classification within the RISE framework.

    By utilizing this framework, designers can identify resources in their courses that are good candidates for additional improvement efforts. For instance, if a resource is in the High Use, High Grades quadrant, it may act as a model for other resources in the class. If a resource falls into the Low Use, Low Grades quadrant, it may warrant further evaluation by the designers to understand why students are ignoring it or why it is not contributing to student success. The goal of the framework is not to make specific design recommendations, but to provide a means of identifying resources that should be evaluated and improved.

    Let’s break this down.

    RISE is designed to work with a certain type of common course design, where content and assessment items are both aligned to learning objectives. This design paradigm doesn’t work for every course, but it works for many courses. The work of aligning the course content and assessment questions with specific learning objectives is intended to pay dividends in terms of helping the course designers and instructors gain added visibility into whether their course design is accomplishing what it was intended to accomplish. The 2×2 matrix in the RISE paper captures this value rather intuitively. Let’s look at it again:

    Each box captures potential explanations that would be fairly obvious candidates to most instructors. For example, if students are spending a lot of time looking at the content but still scoring poorly on related test questions, some possible explanations are that (1) the teaching content is poorly designed, (2) assessment questions are poorly written, or (3) the concept is hard for students to learn. There may be other explanations as well. But just seeing the correlation that students are spending a lot of time on particular content are still doing poorly on particular related assessment learning questions leads the instructor and the content designer (who may or may not be the same person) to ask useful questions. And then there is some craft at the end about thinking through how to deal with the content that has been identified as potentially problematic.

    This isn’t magic. It’s not a robot tutor in the sky. In fact, it’s almost the antithesis. It’s so sensible that it verges on boring. It’s hygiene. Everybody who teaches with this kind of course design should regularly tune those courses in this way, as should everybody who builds courses that are designed this way. But that’s like saying everybody should brush their teeth at least twice a day. It’s not sexy.

    Also, easy to understand and easy to do are two different things. Even assuming that your curricular materials are designed this way and that you have sufficient rights to modify them, different courses live in different platforms. While you don’t need to get a lot of sophisticated data to do this analysis—just basic Google Analytics-style page usage and item-level assessment data—it will take a little bit of technical know-how, and the details will be different on each platform. Once you have the data, you will then need to be able to do a little statistical analysis. There isn’t much math in this paper and what little there is isn’t very complicated, but it is still math. Not everybody will feel comfortable with it.

    The typical way the sector has handled this problem has been to put pressure on vendors as consumers to add this capability as a feature to their products. But that process is slow and uncertain. Worse, each vendor will likely implement the feature slightly differently and non-transparently, which creates a greater challenge for the last point of friction. Features like this require a little bit of literacy to use well. Everybody knows the mantra “correlation is not causation,” but it is better thought of as the closest thing that Western scientific thinking can get to Zen koan. ((Outside of quantum mechanics, at least.)) If you think you’ve plumbed the depths of meaning of that phrase, then you probably haven’t. If we want educators to understand both the value and the limitations of working with data, then they need to have absolute clarity and consistency regarding what those analytics widgets are telling them. Having ten widgets in different platforms telling them almost but not quite the same things in ways that are hard to differentiate will do more harm than good.

    And this is where we fail.

    While the world is off chasing robot tutors and self-driving cars, we are leaving many, many tools like RISE just lying on the floor, unused and largely unusable, for the simple reason that we have not taken the extra steps necessary to make them easy enough and intuitive enough for non-technical faculty to adopt. And by tools, I mean methods. This isn’t about technology. It’s about literacy. Why should we expect academics, of all people, to trust analytical methods that nobody has bothered to explain to them? They don’t need to understand how to do the math, but they do need to understand what the math is doing. And they need to trust that somebody that they trust is verifying that the math is doing what they think it is doing. They need to know that peer review is at work, even if they are not active participants in it.

    Making RISE shine

    This is where CMU’s contribution and EEP can help. LearnSphere is the particular portion of the CMU contribution into which RISE will be integrated. I use the word “portion” because LearnSphere itself is a composite project consisting of a few different components that CMU collectively describes as “a community data infrastructure to support learning improvement online.” I might alternatively describe it as a cloud-based educational research collaboration platform. It is probably best known for its DataShop component, which is designed to share research learning research data sets.

    One of the more recent but extremely interesting additions to LearnSphere is called Tigris, which provides a separate research workflow layer. Suppose that you wanted to run a RISE analysis on your course data, in whatever platform it happens to be in. Lumen Learning is contributing the statistical programming package for RISE that will be imported into Tigris. If you happen to be statistically fluent, you can open up that package and inspect it. If you aren’t technical, don’t worry. You’ll be able to grab the workflow using drag-and-drop, import your data, and see the results.

    Again, this kind of contribution was possible before CMU decided to make its open source contribution and before EEP existed. They have been cloud hosting LearnSphere for collaborative research use for some time now.

    But now they also have an ecosystem.

    By contributing so much under open license, along with the major accompanying effort to make that contribution ready for public consumption, CMU is making massive declaration to the world about their seriousness regarding research collaboration. It is a magnet. Now Lumen Learning’s contribution isn’t simply an isolated event. It is an early leader with more to come. Expect more vendors to contribute algorithms and to announce data export compatibility. Expect universities to begin adopting LearnSphere, either via CMU’s hosted instance or their own instance, made possible the full stack being released under an open source license. This will start with the group that will gather at the EEP summit at CMU on May 6th and 7th, because one has to start somewhere. That is the pilot group. But it will grow. (And LearnSphere is only part of CMU’s total contribution.)

    With this kind of an ecosystem, we can create an environment in which practically useful innovations can spread much more quickly (and cheaply) which vendors regardless of size or marketing budget can be rewarded in the marketplace based on their willingness to make practical contributions of educational tools and methods that can be useful to customers and non-customers alike. Lumen Learning has made a contribution with the RISE research. They now want to make a further contribution to make that research more practically useful to customers and non-customers alike. CMU’s contributed infrastructure and the EEP network will give us an opportunity reward that kind of behavior with credit and attention.

    That is the kind of world I want to live in.

  • EEP, EDwhy, and Seeds

    So the news broke today about the Empirical Educator Project’s (EEP’s) year two experimental design, which we’re calling EDwhy. The “ED” stands for Educational Design,” so the full name means, basically, “Why is your educational design the way that it is?” It invites educators to interrogate their own designs and aspires to give them the tools to do so. Here is the press release.

    We have some good coverage to start you off from Inside Higher Ed and EdSurge. At IHE, Lindsay McKenzie goes broad. She starts with some good shoe leather work at Carnegie Mellon with some interviews. Pay close attention to the interview with Ken Koedinger, as he talks about (but does not name) a research finding called the doer effect, which I’m going to use as an example later in this blog post. She also provides a good refresher of the open source versus proprietary question that universities often face with substantial software intellectual property that they develop, and then touches lightly on EEP’s role with the EDwhy announcement at the end (although with a clutch statement from Duke’s Matthew Rascoff, who always seems to say the right thing with a lot of intellectual and moral clarity in very few words). If you’re looking to find a way into this story from the beginning in a compact way, Linday’s story one good route in.

    Meanwhile, Jeff Young at EdSurge has dug a little deeper into significance behind the EDwhy idea and mechanics. I think the question that is on everyone’s minds is, “OK, $100 million dollars, lots of software, cool learning science-y things, but really, how is this going to be made useful?” Jeff begins to explore that question, and I’m going to take a deeper dive in this post. He also has some commentary from me about why we chose the name we did. You’ll have to go read it on EdSurge to get those details, but I’ll say this much here: On e-Literate, where one of our major roles is to critique hype and protect against the dangers of  bad actors, we have an ethical obligation to throw some sharp elbows. With EEP, where we are not watching from the sidelines but actually entering the fray, we are mindful that our obligation shifts as our role shifts. We take the e-Literate lessons to heart while also attempting to be humble both about the accomplishments of those before us and how easy it is for us to fall into the same traps that very smart people before us have fallen victim to.

    But I don’t want to write about the naming decision too much here. Instead, I want to write about how we are going to attempt to live up to the humbling confidence that Carnegie Mellon expressed in us when they chose us as a partner in their grand project. Obviously, when they offered to make their enormous contribution through our fledgling organization, it both forced and empowered us to rethink how we would go about the project in Year 2. We had always planned to stop, evaluate, and iterate on the design after the first year, but this opportunity demanded a pretty dramatic rethink in approach which, to be honest, is still ongoing. We have an idea that I’m going to share with you now that I believe makes sense in concept but does not yet have a fine-grained implementation plan. We are working hard with our Carnegie Mellon friends to have a foundation in place by the time of the summit. We will also workshop the idea at the summit with the cohort to refine our approach. This is going to be a year-long project. So we expect to spend some time after the summit continuing to put pieces in place and fine-tuning as we go. At the end of the year, we will do a progress check, evaluate, and iterate.

    The Hackathon

    I am always mindful about appropriating terms from Silicon Valley culture because I think it tends to be reflexively idealized. That said, there is a lot to like about the educational value of a hackathon. It is a social, time-bounded, self-organizing, problem-based learning exercise. A group of people will get together to solve a defined problem over a period of time. That group is often cross-functional. They might have software engineers, user experience designers, end users, and so on. Hackathons have a tangible and several intangible goals. The tangible goal in the canonical case is a piece of software, but we can think of it more broadly as an artifact that has been tested and demonstrated to solve the problem that was the goal set out at the beginning of the exercise. The intangible goals often include learning how to work in a cross-functional team, learning how to solve difficult problems with unexpected wrinkles, and learning particular craft-related skills necessary to solve the problem (e.g., programming tricks or software testing techniques).

    This is a good model for the kind of culture building that EEP has always aspired to achieve and, I believe that inspired Carnegie Mellon to see us as a good fit for their own ambitions. While I want to be clear that I do not speak for them, my understanding of their goals from our conversations thus far is that it would be a mistake to interpret their primary goal to be broader adoption of their software and other tools. Sure, they want to see that happen. But my read is that they see that as a second-order effect, or maybe as means to an end. What I hear from them in our conversations is that they really want to make their approach to improving education broadly accessible and meaningfully useful. They call that approach “learning engineering,” which they seem comfortable with me characterizing as one flavor or methodology within a broader developing family that we call “empirical education.” The hackathon works to support this goal because it creates an environment in which people habitually self-organize in cross-functional groups to improve educational design in ways that empower greater student success. It brings together the right people around the right kinds of goals and conversations. If we can then empower them with the right tools and methods, we are on your way to promoting learning engineering. If we can achieve that,  we can unlock the real power of the big release, which is to help democratize the science of education.

    While I said I didn’t want to dwell on our name choice here, it’s probably worth spending a little time on the word “design” in the way we are using it in EDwhy. A number of different overlapping but distinct stakeholder groups in academia tend to compete for mindshare around this word—Design Thinking practitioners, Instructional Designers, Learning Designers, User Experience Designers, and others. Making sense of how these all connect yet are distinct from each other is non-obvious even before we get to culturally local differences in usage. To give one example, Herb Simon, in addition to being the father of Learning Engineering, is considered by some to be the grandfather of Design Thinking. These are two compatible but distinct and non-interchangeable disciplines. In most places outside of Carnegie Mellon, their practitioners tend to be either completely ignorant of each other or find themselves cast as rivals in educational solution design.

    “Design” in the EDwhy context is a holistic and colloquial term meaning, simply, the way you decided to put something together. A cross-functional EDwhy hackathon team might include people with knowledge of Design Thinking, Instructional Design, Learning Design, User Experience Design, and/or Learning Engineering. Who is at the table will depend on the specific nature of the challenge being tackled and the kinds of expertise needed to take it on.

    At any rate, as we started thinking about how to help our network digest Carnegie Mellon’s $100 million contribution—never mind the sum of all possible contributions from all current and future EEP participants—we started thinking about both the digestive process and coming up with a form that is digestible. Verbs and nouns.

    The hackathon is the verb. Theoretically, the hackathon is flexible enough to allow for projects of different sizes and ambitions, whether inter- or intra-institutional. We still very much want to encourage inter-institutional collaboration, but one lesson we learned last year is that inter-institutional collaboration is incredibly hard, even with a lot of work done by third parties to lower barriers. We have to build a gentle slope toward that level of collaboration. The hackathon is a form that lets people start small and grow in ambition. At some point, they will outgrow the form and need to form something more like a traditional project with more formal management structures.

    We aspire to reach the point where we have that problem. For now, we are focused on culture-building, and we hypothesize that the hackathon is a good ritual for accomplishing that while also delivering immediate educational utility.

    The Seeds

    The hackathon idea is simple enough to grasp in the abstract. The hard part is putting it together with the right packages that help people identify and solve new problems using the contributions from Carnegie Mellon or other participants. For this, we’ve developed the concept of an EDwhy “seed.” This is one of the pieces I will want to workshop with the EEP cohort, but there’s enough here conceptually that the general idea should be clear.

    We start with a general area of interest where some research has been done but where there are more questions to be answered. For example (and as I mentioned earlier, Ken Koedinger and his CMU colleagues have done some research into something called “the doer effect.” It means pretty much what it sounds like. The researchers were able to demonstrate, using solid, quantitative methods that learning by doing is, for example, about six times more effective than learning by watching a video.

    (Side note for all you liberal arts folks out there who are suspicious of this data stuff: This study more or less just made the case for constructivism. Using numbers and computers and statistics and stuff.)

    That’s an interesting finding, if not a shocking one, but it also highlights a lot that we don’t know. For example, is doing always better than watching a video (or reading) for learning? Should we throw out all books and videos? If not, then how much watching or reading is good? In what order? Does the subject matter make a difference? The expertise of the learner? Other characteristics of the learner? Other characteristics of the overall course design? Or course goals?

    Let’s make this more concrete. One of my favorite course designs is Habitable Worlds by ASU’s Ariel Anbar. There is a lot of learning by doing in that problem-based course, but also liberal use of video. It would be interesting to do some testing and experimentation to find out how to make the most out of the doer effect and find the optimal balance of the course elements.

    As it turns out, Carnegie Mellon’s contributions include the software that was used to conduct the original doer effect research. (The IHE article mentions LearnSphere. Spend a little time exploring that site if you’re curious.) That software includes a data repository with access to (appropriately anonymized) data that could be used to replicate the results (or try to run different analyses on the data), a visual workflow that makes the study easily repeatable with different data, and access to the underlying R packages (for those who can understand them) to make the research methods completely transparent. If you put together the original studies, the software, the workflows, the data to practice reproducing the results, the transparency of the methods, and wrap in some documentation, some training, and a number of suggested starter questions for investigation, you have a seed. A self-organizing community could take up that seed and develop a hackathon project. If there were also a community forum where the hackathon group could ask questions of statisticians, cognitive psychologists, and psychometricians, as well as some technical support folks, as well as share lessons learned with each other, then you could really have something.

    I’m guessing the net result might turn out to be what would call an “intermediate” seed. Not every team would have the capability to self-organize around something this complex. We’d like to develop beginner, intermediate, and advanced level seeds, where beginner seeds are approachable by non-technical groups, intermediate seeds might require some technical skill and some knowledge of experimental design, and advanced seeds are really for folks who have some serious specialist expertise in their groups. The I’ll defer on the final difficulty ratings of each seed, including the one I just described, to the creators and the early adopters. One skill set we will be learning in the EDwhy experiment is how to package up a seed to make it accessible and useful to different sorts of audiences. Eventually, we may develop profiles of hackathon teams that are richer than just beginner/intermediate/advanced.

    At any rate, our goal for the year is to prove out and refine the approach through some pilot seeds and hackathons. We don’t imagine that we will be able to address the entire surface area of Carnegie Mellon’s $100 Million contribution in the one-year time frame, but we do aspire to prove out a novel and sustainable support and diffusion mechanism, not only for the software but for the methods and the culture. And during this time, we will also invite other EEP members to develop and contribute their own seeds, some of which will be less technical or tackle entirely different types of educational problems than Carnegie Mellon’s seeds will. This is a general mechanism we will be trying out. Interestingly, another arrow that CMU has in its quiver is the Open Learning Initiative (OLI) authoring and delivery platforms. So we may very well find their contributions to seed development goes well beyond the open source software code, which I think is the way in which people are naturally tending to think about the contribution at this early stage in the process.

    Both learning and science—or any path to enlightenment, really—starts with a simple admission: “There is so much that I don’t know, and so much that I would like to understand better.” Big announcements like this generally run against the grain of that admission. We have an ingrained cultural notion that, after spending a $100 million, you are supposed know all the answers. After spending 7 years in graduate school, you are supposed to know all the answers. After getting all the press and all the buzz, you are supposed to know all the answers.

    Nope. Sorry. It doesn’t work that way.

    There is so much that we don’t know, and so much that we would like to understand better. If you keep repeating that mantra to yourself every time you hear something new about Carnegie Mellon’s contribution or about EEP or the EDwhy initiative, each new piece of information will make a lot more sense to you.

  • Empirical Educator Project/CMU Updates

    Empirical Educator Project/CMU Updates

    As promised, I’m returning to Carnegie Mellon University’s announcement of the $100 million contribution of tools, software, and content to support the democratization of learning science, as well as the relationship of that contribution to the Empirical Educator Project.

    Just that first sentence shows how much meat there is yet to be put on the bones. “Contribution?” “Democratize learning science?” What does all that mean? These are fair questions. Jeff Young has some good reporting up at EdSurge that connects some dots on what’s public so far. I’m going to do my best to explain why the answers are not yet as clear as they might be, when and how they will be come clearer, and then give some reminders of past conversations here on e-Literate that provide some breadcrumbs leading in the direction of where all this is going. There will be more news in the press that will drop on Wednesday, followed by some more analysis by me on Thursday. From there, the story will build in pieces, through the Empirical Educator summit on May 6th and 7th, and afterward.

    Let’s start with the big picture. Carnegie Mellon has been a pioneer in various types of theoretical and applied educational and cognitive science research—which is now fashionable to roll up into a ball and call “learning science”—for decades. While their work often doesn’t get the publicity it deserves, the breadth and depth is astounding. I’ve had the privilege of visiting a few times and taking deep dives. ((When I say deep, I mean deep. See, for example, https://mfeldstein.wpengine.com/can-there-be-microscope-of-mind/.)) There are very few institutions in the world that are in their league in terms of the scope of what they do.

    The university made a decision to take a lot of their work product—over $100 million worth, in fact—and make it broadly accessible to the academic community. Honoring this commitment in a real and practical way is…hard. Very, very hard. It’s about more than just releasing the source code to software, which is hard enough to do right in and of itself. This is about making science—not just the output but the practice—accessible and useful to non-scientist educators. On top of that, the components of the contribution were not all designed together as a single software platform. They consist of many research projects, developed by different teams and that may be more loosely or tightly related to each other. Imagine if a top research laboratory decided to turn itself into the Smithsonian Institution, making itself a hands-on science museum accessible to everyone without dumbing itself down. That’s not exactly the goal that Carnegie Mellon’s Simon Initiative has set for itself, but its’ the same spirit. It’s incredibly ambitious and not easily imagined, much less described in a single press release.

    In fact, doing this well is itself going to take some empirical experimentation. And that’s where the partnership with the Empirical Educator Project comes in. Carnegie Mellon had already offered to host this year’s summit on May 6th and 7th. I’m honestly not sure how long they’ve been mulling over this idea of the giant release, but at some point they came to the conclusion that we would be good pilot partners for their effort. This week, we will be describing our high-level approach to our pilot design for helping to make CMU’s contribution accessible. And not just CMU’s contribution, either, since EEP is based on the premise that many academic institutions have innovations to contribute if only we can get better at sharing them. The scale of CMU’s contribution has given us an opportunity to challenge our assumptions about how we should be going about this work. It’s a once-in-a-lifetime opportunity.

    As I said, more details while emerge in the coming days, weeks, and months, first as we learn how to tell such a big story and later as we learn from the grand experiment as it unfolds. For now, I want to leave you with two video playlists. The first is a set of interviews of Carnegie Mellon faculty that I recorded while visiting a few years ago. At the time, I had no agenda other than to capture their individual and collective views on “learning science” and its relationship to classroom teaching. The second is a set of interviews from participants after the first day of the first EEP summit. Taken together, I think you will see why there is a good fit.

    CMU interview YouTube playlist.

    EEP interview YouTube playlist.

  • Carnegie Mellon’s $100 Million Announcement

    This is going to be a short post, in part because I’m traveling, but I need to call your attention to a developing story, both because it’s huge in its own right and for its importance to the Empirical Educator Project (which, by the way, has a new website).

    Carnegie Mellon has announced a $100 million contribution in tools, software, and content that “that is intended to catalyze a new era of progress in educational effectiveness that is equal to the challenge of rapid change and growth in 21st century educational needs. The suite of tools is the product of over $100 million of research and development from a wide variety of funders.”

    The suite of tools will be released in stages over the next year and represents a major departure from the “silver bullet” or “moon shot” efforts to revolutionize education with technology in recent years. Instead, the contribution is intended to democratize the science of learning and empower educators across the world to become citizen scientists. Carnegie Mellon’s goal is to provide knowledge of how to conduct applied educational research that classroom educators, researchers and educational technology companies can learn, apply, extend and share with the global educational community.

    “We live in a moment when our educational institutions are in danger of a catastrophic failure that we cannot afford. College and university closures are becoming regular occurrences, even as tuition and student debt rise to record levels,” said Norman Bier, executive director of the Simon Initiative.

    “This is happening at the same time that even highly educated people need to continue learning in order to keep their skills up-to-date, and when people all over the globe need increasing access to high quality educational opportunities through technology. In the face of these institutional and structural challenges, demonstrably improving outcomes and learning for students must be our foremost concern,” Bier said.

    I have lots of good things to say about this approach, but for a preview, you can go back and review my post about ed tech hype being in remission. This announcement is a little hard to parse because it’s just a down payment on a complex story, and because it’s a big price tag thing from a big engineering school, but trust me: this is not the same old thing.

    The full list of what is being released has yet to be announced, but I’ve seen it, and it is mind-blowing. The breadth and depth are pretty astonishing. In fact, one of Carnegie Mellon’s  biggest challenges will be explaining all of what’s in it. This isn’t a tool or a platform. It’s a collection that’s in the process of being knitted together into an ecosystem. And the way that people inhabit that ecosystem is what will really matter.

    There will be a lot more to say on that in the near future. The university is going to be revealing a lot of the details of their contribution at our second annual Empirical Educator Project summit, which they are graciously hosting on May 6th and 7th. We will have some announcements between now and then, likely a flurry of announcements (from us and from other parties) around the time of the summit itself, and will be releasing video of many of the talks after the summit afterward. There are also some reporters working this story, so I will keep running updates of those stories as they come out over at the new Empirical Educator Project site and periodically collect them in my updates here as well.

    Watch this space.

  • A Blackboard Debt Update, and a Lesson on Public Relations

    The short version of this story is that Blackboard has sold off their Transact business and moved their corporate headquarters out of Washington, DC. Both of these are sensible moves that give them an opportunity to reduce their debt load and get on better financial footing. Not being a debt expert with access to the information to the information that credit rating agencies have, I can’t comment at this point on how much this improves their outlook.

    That should be the sum total of this post. Unfortunately, because Blackboard handled some of our previous coverage poorly, the news also provides evidence that they weren’t entirely forthright with us in previous conversations about the tools they had at their disposal for handling their debt. So I’m obliged to complicate the news with an accountability story. Since I am weary of writing accountability stories after 14 years of writing them, I prefer to turn this story into a lesson about how ed tech companies can handle difficult PR situations better, in the hopes that I will have to write fewer of these stories in the future.

    If what you mainly care about is Blackboard’s financial health, then you can stop here. If you’re interested in learning more about how ed tech companies accidentally get themselves into unnecessary trouble, and how they can minimize their chances of doing so, then read on.

    Asking for Trouble

    Back in the summer of last year, I got into two consecutive spats with Blackboard, neither of which I saw coming. The first was about our data showing that Instructure Canvas had surpassed Blackboard in US market share. Now, everybody, including Blackboard, knew that those market share lines where moving closer together. Counting methodologies being complex and legitimately debatable, it’s possible to have honest disagreements about how close they are or whether they have crossed at any given moment. But we considered this more of a milestone that summed up a long-term trend than a big story in and of itself.

    There were a couple of ways that Blackboard could have handled this. One was to play it to the hilt. Being branded—pun intended—as the corporate juggernaut has been a problem for Blackboard for a long time. They could have finally turned the tables on Instructure and portrayed themselves as the scrappy underdog. Or they could have just kept quiet and let the story pass. It would have disappeared down the memory hole in about three days. Instead, they engaged in a strong public pushback against our numbers. That doesn’t make them bad people; they have a right to defend themselves as they see fit. But from a purely strategic perspective, their choice had the effect of both prolonging their bad news cycle and provoking responses from us.

    In one of those responses, I tried to explain—again—that the market share lines crossing really wasn’t news but rather a symbol of the ongoing trend. If you want real potential bad news for Blackboard, I argued—and this is where the danger of prolonging the bad news cycle caused further complications for them—you should look at their debt situation:

    The real issue of concern is the potential behavior of their debt and equity owners. I’ll come back to the point about Blackboard’s total product portfolio in that context….

    The second “Blackboard alert” article worth reading is the one by Katherine Doherty and Eliza Ronalds-Hannon at Bloomberg News. This one wasn’t a reaction to our piece but rather coincidental timing triggered by the same underlying concerns. I spoke with Doherty, whose beat includes companies with distressed debt.

    The debt is the real existential issue. Without it, Blackboard would just be a company that continues to struggle with its flagship product but which would have enough runway to turn itself around over time, one way or another. In the Bloomberg article, Blackboard CEO Bill Ballhaus repeats Miller’s reminder that the company sells other products and services. And as we have pointed out repeatedly here on e-Literate, the international markets are an increasingly large percentage of Blackboard’s financial picture. The fundamentals of the company may not be great, but they’re not dire either. Given time, good leadership, and low debt, a company in this position should be able to right itself.

    But because Blackboard has high debt, their situation potentially a lot more volatile. One major reason why the market share milestone matters is that it’s an apt metaphor for Blackboard’s financial waterline. At their current debt levels, the company can’t afford for their market share to continue to drop.

    Contract losses have sent Blackboard’s revenue and earnings sliding, according to people with knowledge of the matter, making it harder to carry more than $1.3 billion of rated debt. With some of Blackboard’s bonds selling at deeply distressed levels, Ballhaus is crafting a comeback, and possible options include the sale of its payment processing division, said the people, who asked not to be identified because the discussions are private.

    So, I wrote that Bloomberg news reported that Blackboard has debt levels that are high enough to cause potential problems for its business. Elsewhere, we reported that Moody’s had made some negative comments about their debt as well. Reporting these things is a little like reporting that the Weather Channel is predicting a high likelihood of heavy snow. It’s not something you argue about.

    I went on to say that, while the debt situation was serious for Blackboard, it was far from certain doom:

    Even in this situation, the results for Blackboard Learn customers won’t necessarily be bad, or even noticeable—depending on how the finances are resolved. If Blackboard sells off Transact, gets a good price for it, pays down some debt, and otherwise sticks with the current management’s plan, that could buy them some time and some ability to survive further erosion of market share around Learn. If the company’s owner, Providence Equity, decides to take more drastic steps, then the potential impact on customers is unpredictable. And Providence’s calculations regarding how much drastic action is required must be at least partly driven by their assessment of how close Blackboard is to bottoming out in LMS market share loss.

    This story was harder for the company to ignore, but they still had some options. I had said some good things about the company in that post, and some critical things about their competition. Their CEO, Bill Ballhaus, is a turnaround specialist. The Private Equity owners of Blackboard have signaled their trust in him by making him both Chairman and CEO. They had a good story line there. And they were going to have an option to engage with us in person very soon at BbWorld.

    So what strategy did they choose?

    Keep digging

    They pushed back hard. We had two meetings with Blackboard executives, including our meeting with Bill Ballhaus, in which they brought up the debt issue immediately and completely dismissed it as illegitimate. This was what poker players call a “tell.” It was so far out of the norm for this sort of analyst/executive meeting that it sent a clear signal of how concerned they were about the coverage. Now, some of that is simply due to the fact that people read what we write and react to it, and they don’t always read it carefully. So we know that Blackboard got inbound calls that were triggered by that post. But the tone of the pushback, coupled with the implausibility of the arguments, were highly inconsistent from our experience with this management team.

    This instantly changed the story line of BbWorld for us, and not in a way that Blackboard had intended. Here’s what I wrote in my follow-up post, entitled “Blackboard’s Defenses of Its Finances Are Not Persuasive,” about that meeting:

    When we were at BbWorld the week before last, Blackboard’s executive management pushed back vehemently on our analysis of how their high levels of debt could impact their business decisions. We heard their strong disagreement expressed in our very first meeting of the conference from Chief Learning and Innovation Officer Phill Miller and in our very last meeting from CEO Bill Ballhaus.

    We stand by our analysis. In fact, Blackboard’s pushback had the opposite of its intended effect. We left BbWorld more convinced that we are right rather than less….

    Ballhaus argued to us that the amount of debt that Blackboard is carrying is a strategic choice that he and the private equity investors—he used the pronoun “we”—make together. In particular, he argued, “we” could choose at any time to invest more money in the company, paying down debt in exchange for equity. Further, he argued, it’s logical to assume that Providence would do so if needed because “they only make their money if we improve.”…

    Paying down debt in exchange for equity, called “recapitalization,” is a strong vote of confidence by a private equity (PE) owner. First, since debt holders get paid before equity holders in the event of bankruptcy, it increases risk for the PE firm. Second, it would mean a substantial investment of cash, which is partly what PE firms typically try to minimize by requiring the companies that they own to take on substantial debt in the first place. When PE-owned companies find that they are in danger of being unable to make their debt payments—which both Moody’s and S&P Global Ratings have said is currently the case with Blackboard—the PE owners can and do employ a number of different strategies that are financially less risky to them in order to address the problem, either instead of or in addition to recapitalizing.

    For example, when Cengage Learning found itself with unmanageable debt levels after its acquisition by private equity, they filed for bankruptcy:

    “The decisive actions we are taking today will reduce our debt and improve our capital structure to support our long-term business strategy of transitioning from traditional print models to digital educational and research materials,” Michael Hansen, Cengage Learning’s chief executive, said in a statement.

    To be crystal clear, I am not suggesting that Blackboard is likely to file for bankruptcy. Providence Equity has other options at its disposal, some of which I will write about in the next section.

    Rather, the point is that Ballhaus’ claim that we should just assume Providence will see it as being in their interest to recapitalize Blackboard is not credible on its face to anybody with even passing knowledge of how private equity companies work. For example, the tone of the Washington Business Journal article I referenced above, which (obviously) was written by a business reporter, suggests significant skepticism that Providence will not let the company’s debt challenges impact their business decisions. The industry experts we typically consult with when writing financial or business stories like this one were even harsher in their evaluations of Blackboard’s position. Two literally laughed out loud at it.

    I’m pretty sure that wasn’t the coverage Blackboard was hoping for.

    And it was destined to get worse. Because they were now stuck in a trap of their own making.

    The Trap

    What do you do as an analyst when you believe a company has been making misleading statements to you and that there will be evidence supporting your theory of the case in the future? You lay down a marker and wait.

    Here’s what I wrote:

    [I]n fairness, there is an empirical fact of the matter here, and we do not yet have conclusive public evidence that the company’s high levels of debt will, in fact, affect their business strategy. So here’s what we’re going to do:

    1. I will summarize their position as objectively as I can.
    2. I will explain why we don’t find their position persuasive.
    3. I will lay out the signs that concrete evidence we will be looking for going forward that will either support or undermine our thesis.
    4. Phil and I will publish updates as we monitor these signs and, if there is no additional public evidence of our thesis by BbWorld 2019 (or strong evidence emerges that we are wrong before then), then we will publish a mea culpa post….

    Here are a few actions Blackboard could take in the future that would indicate Providence Equity has chosen to push Blackboard to solve its own debt problem rather than making it go away with more of Providence’s money:

    • Sell off one or more parts of the business: A Bloomberg piece written by journalists from their distressed debt desk reports, “With some of Blackboard’s bonds selling at deeply distressed levels, Ballhaus is crafting a comeback, and possible options include the sale of its payment processing division, said the people, who asked not to be identified because the discussions are private.” Said payment processing division, Blackboard Transact, is a cash cow for the company. If Blackboard sells off one of its more profitable business units at a time when the company is having trouble making debt payments, that would indicate a choice by Providence Equity to find a way to reduce debt pressure that is less risky for them in terms of cash investment but more risky for Blackboard in terms of long-term health. Particularly since Providence already tried to sell Blackboard once and has now owned the company for well past the normal sell-by date that PE companies like to follow, the sale of Transact might suggest further moves to follow.
    • Unload expenses (like office space): The Washington Business Journal article notes, “Blackboard is also interested in unloading its 70,000 square feet of office space at 1111 19th street, with 12,000 square feet already sublet, according to an April post on Tech Office Spaces. It’s unclear where Blackboard will go if it succeeds in leasing out its entire footprint. Blackboard stood to benefit from a tax rebate program for companies that agree to sign 50,000 square feet for at least a dozen years, valued at half the company’s tenant improvement costs, or a maximum of $5 million over five years.” Of course, companies take cost-cutting measures all the time, regardless of their financial health. The business reporter’s phrasing suggests that he may be detecting a whiff of desperation in the specifics of this transaction. Since that’s his expertise more than ours, we’ll be looking for additional confirmation of our thesis, such as if Blackboard were to…
    • Significantly restructure with major layoffs: If Blackboard were to move to a smaller office while also laying off employees—beyond those that might leave in a sale of a business division or the slow leak of headcount that the company has been having for a while now—that would certainly be an indicator that Providence is not ready to just give Blackboard the money the company needs to complete a turn-around and is instead pushing them to solve their own financial problems.

    I wrote that post on July 31st, 2018. Where are we seven and a half months later?

    Trap Sprung

    Last week, Blackboard announced the sale of Transact.

    Boom.

    Note that this is not an objectively bad thing for Blackboard. To the contrary, it helps them with their debt problem in a way that has little to no impact on their core customers. The problem is that it also runs against the grain of the story line that Blackboard executives pushed to us aggressively last summer. They turned what should have been a good story for them into a more complicated story.

    Also, last January, Blackboard announced they would be moving their headquarters out of Washington, DC to a new space in Reston, VA.

    Boom.

    Note that this is also not an objectively bad thing for Blackboard, also a way for them to manage their debt problem without impacting customers, and also in tension with the story they told us last year.

    The layoff evidence is less clear. There are plenty of comments on Glassdoor about ongoing layoffs (as well as voluntary talent drain) at the company, but that sort of evidence needs to be taken with a heaping teaspoon of salt. And honestly, I am hoping that I don’t see evidence of a major restructuring going forward. I never want to root for people to lose their jobs.

    So overall, there is pretty clear evidence that (a) all the debt agencies were not wrong when they said Blackboard had debt issues, (b) we were not wrong when we said that Blackboard could not expect Providence Equity to solve their problems by simply showering them with more money, and (c) Blackboard is taking necessary and reasonable steps to reduce their debt load. In other breaking news, gravity still exists.

    I let Blackboard know I would be writing an update to my previous post and gave Mr. Ballhaus an opportunity to revise and extend his previous remarks. Here’s what I got back:

    Our decision to divest in Transact is consistent with our strategic efforts to simplify our business as we move to a purely SaaS model and enhance focus on our core teaching and learning portfolio. The fact that we have tightened our strategic focus toward our current education clients and are accelerating innovation to benefit them is a stark contrast from our competitors who are looking to grow their business in areas outside of education.

    Proceeds of the sale will go toward deleveraging the company and also present potential opportunities to reinvest in the business.

    OK. This isn’t a bad statement as far as it goes. The last sentence is most directly relevant. The rest is stuff that he wants to get in, which is fine and expected. The truth is that he’s heavily constrained in what he can say about the debt management for legal and other reasons. As I said at the top, regarding the substance of the situation, I’m not a financial expert and do not have access to the information that debt agencies do, so I will wait for them to weigh in on how all of this affects Blackboard’s financial prognosis. It can’t be bad, but I’m not qualified to judge how much they’ve improved without the benefit of expert input.

    Regarding the PR situation, as far as I’m concerned, this was a fairly normal corporate answer. We’re back on terra firma. The more interesting question is how companies can avoid getting themselves into this situation in the first place. Blackboard is far from the only company that has gone through this sort of thing with us. And while we occasionally run into CEOs who are simply bad humans, usually these situations arise out of missteps that happen while the people at these companies are under enormous pressure because the companies are working their way through a rough patch. I have been on the inside of a company as an employee in that sort of a situation. It is not easy. External-facing leaders in particular need to cultivate reflexes to deal with these sorts of crisis situations. And make no mistake; an analyst or reporter raising uncomfortable questions about something your company is struggling with definitely feels like a crisis situation.

    So what’s the right reflex to cultivate?

    Honesty Works

    Executives need to understand the power of honesty as an offensive weapon. Set aside ethics for a moment. I’m talking about it from a purely strategic perspective. Let’s look at a couple of examples of how to play this card, starting with Blackboard.

    I made the analogy earlier between the debt agencies and the Weather Channel. Let’s extend that analogy a bit. I live in Massachusetts. It snows a lot up here. (Or at least, it used to.) But we have very professional road crews who know what they are doing. Most of the time, we can get a dumping of 18 inches at night and still drive on safe roads the next morning. We check the weather, but we don’t freak out about it. One way Mr. Ballhaus could have handled the debt conversation would have been to wait for us to bring the topic up—which we would have—and then say something analogous to the following:

    It’s true; the Weather Channel is predicting snow. I lived for ten years in rural Minnesota. In a house with a very long driveway. I have a truck with a snow plow on it. Here is my shovel. Any questions?

    Here’s how that looks in business terms:

    It’s true; the debt agencies are concerned about our debt load. That’s one reason why Providence Equity brought me in. I’m a Private Equity turn-around guy. This is pretty much what I do. You guys have been around the block enough to know that I can’t talk about the details of how we’re going to manage it, but we have strategies in place for strengthening the company’s balance sheet while protecting our customers.

    Every word of this is either undeniably true or reasonably plausible, and there’s not much we could have said in response. It also doesn’t make for much of a story. It becomes a “this is one of a number of factors that we’re watching” kind of thing. It doesn’t become the dominant story line coming out of BbWorld. And it softens the analysts up just a little bit. With a story like debt risk, there’s always a lot that we don’t know. The flavor of our coverage is influenced by our trust in management to be reasonably honest and open with us. In this context, a little bit of good will goes a long way.

    For another example, take the case where Pearson was trying to manage a story I was chasing of their CIO’s penchant for repeatedly talking about the company’s supposed ambition to build, in his words, the Netflix of education. I was (and still am) pretty sure that isn’t what Pearson is trying to do. But because this very high-level person continues to be allowed to publicly declare that they are trying to do this, and because Pearson is a big place where the right hand sometimes doesn’t know what the left hand is doing, only somebody who outranks the CIO can definitively put the question to rest. And there is only one person who outranks him: CEO John Fallon. After much discussion with Pearson’s PR team and some interviews with a couple of the CIO’s peers in other relevant parts of the organization, and very much to my surprise, I was given direct access to Pearson’s CEO. So here we go, I thought. This was going to be easy. He could put this story to bed, and I could spend my time writing a story about something good rather than something dumb.

    That’s not what happened. 

    But it so easily could have.

    Yeah, Pearson doesn’t want to be the Netflix of education, and Albert shouldn’t have said that we do. We do think that we need to become an internet-scale digital company, and we do think that the trend toward renting digital assets is one that is relevant to the educational market. That said, we understand that analogies are fraught in education and we don’t wish to oversimplify. In fact, we sold off our consumer businesses in part because we want supporting educators in managing those complexities to be the core of what we do. When we talk about efficacy, that’s what we mean.

    That answer not only would have spiked the “Pearson can’t stop talking about being the Netflix of education” story; it also would have turned me toward writing an update to my original efficacy story, complete with reporting on the genuinely good work the company’s efficacy team has been doing since then. The Netflix thing would have ended up getting mostly buried as a side note about how difficult and fraught it is for companies to talk about their work.

    It wouldn’t have mattered if I knew the answer was a strategic attempt to kill or divert the story. As long as it was honest, it would have been fair game. Beyond that, I would much rather write a story about a company that’s trying to do the right thing and does it imperfectly than play a game of gotcha with a company because the management is giving answers that I feel I have an obligation to police. When leadership chooses not to play the honesty card, they often accidentally trigger a gotcha game that generally doesn’t end well for them and puts them in a worse light than they actually deserve. I wanted to say definitively that Pearson isn’t trying to be the Netflix of education because I believe it to be true. I know they are doing good work because I have seen it with my own eyes. I could have written a story about that. But because Mr. Fallon chose not to play the honesty card, the story ended up being about Pearson failing to clearly disavow the Netflix analogy, and I couldn’t definitively write what I believe to be true about the company because Mr. Fallon didn’t give me the proof that I needed for the story.

    I could go on; I have many more examples like these. A positive one is when Instructure’s former CEO Josh Coates completely transformed a negative story about the company charging customers for access to their own data. Not only did he admit that the company screwed up in very blunt terms; he changed the policy, encouraged us to call customers and verify that they were satisfied with the changes, and thanked Phil for calling his attention to the problem. There are reasons why we wrote very little negative coverage of Instructure during the Coates years. One is that the leadership team really understood the power of honesty.

    The bottom line is that honesty is disarming. Good analysts always try to maintain healthy skepticism, but they also try to be good judges of character. Because good analysis is partly based on knowing how much you trust the particular version of events that a company’s management is giving you. In contrast, when leaders give in to the (understandable) temptation to deflect, they look dishonest. I want to be very clear about separating a moment of failing to be honest, which we all have had under pressure from time to time, from being a fundamentally dishonest person, of which I have met relatively few in my life. Analysts (and reporters) have to distinguish between the two based on very little information, and they have the obligation to be skeptical. Also, good analysts have instincts that lead them to poke where it hurts. The reflex of the person being poked will be to protect the sore spot. Effective leaders learn to fight that reflex in the moment. They acknowledge problems that the analyst may have uncovered (to the degree that they can) and turn that moment of vulnerability into an opportunity.