e-Literate

Present is Prologue

Author: Michael Feldstein

  • Instructure’s Better Possible Future

    Having reluctantly weighed in on Instructure’s proposed acquisition by Thoma Bravo, I would like to turn back toward positive rather than negative possible futures by describing a different potential vector for the company. Despite the criticisms of my previous post, I do not believe that Instructure is in a deep hole, inevitably heading deeper, with no way out in the foreseeable future. If there are two takeaways from that post, they are these: First, it is easy for an EdTech company to have a sudden and dramatic reversal of fortunes and of customer perceptions, especially in the downward direction. Instructure is at an inflection point where the risk of such a turn is particularly high. (I warned about this over a year ago, after Instructurecon 2018.) Second, the seemingly obvious or proximal causes of a company’s success can be misleading. When we misunderstand causality in this way, it heightens the risk. Instructure has not been a well-understood company in general, and my concern is that the current board of directors and executive management may not have a good understanding of the causes of the company’s historic success. So that post wasn’t about what will happen. It was about what could happen.

    In this post, I want to explore the company’s actual current competitive strengths as the basis for the kind of growth in value generation that both customers and shareholders would like to see. In the process, I’m going to have to spend a little more time analyzing their missteps. But beating up on Instructure’s management is not my goal. Rather, I want to offer up a case study for how value can be created in EdTech in the 2020s. In the process, I am going to spend some time on the promise and perils of data-driven affordances.

    Reminder: D2L, one of Instructure’s major competitors, recently became the first foundational sponsor of e-Literate’s Empirical Educator Project (EEP). Given that big decisions are in the process of being made about Instructure’s future right now, I am obliged to call special attention to the appearance of a conflict of interest.

    Instructure’s remaining competitive strengths

    As I highlighted in my previous post, Instructure entered the market with competitive strengths that were difficult and time-consuming for their competitors to duplicate. One was that Canvas was built from the ground up to be cloud-native. From their customers’ perspective, that’s a critical ingredient to the product’s reliability. From their competitors’ perspective, it is a very hard technical task to retrofit a cloud architecture onto a mature product that was designed to run on individual customers’ own servers. Another competitive advantage was that Canvas was built from the beginning to delight end-users, whereas older platforms were at least partly built to delight the IT managers who had made purchasing decisions in the early years of LMS sales, and who often had different concerns than the end users. This too is very hard to retrofit onto an older application. Arguably, Blackboard’s biggest mistake with Ultra has been to set customer expectations too high regarding the pace of progress. The rearchitecture and redesign they have undertaken are genuinely, seriously hard.

    But given time and motivation, these challenges can be overcome. And sure enough, Instructure’s competitors have been steadily narrowing these gaps over the years. The race is now competitive in these areas where Instructure had the field to itself for a long time. Likewise, their competitors have improved their customer service and focus on improving the end-users’ interactions with the companies.

    So what’s left?

    First, while Instructure’s brand has taken a couple of hits lately, it’s still strong enough, and customer memory of their positive experiences with the company is long enough, that lost ground can be regained. There are some customers who have had bad experiences, or growing concerns. The CEO has said potentially alarming things in public and failed to correct his mistakes. Nevertheless, I get the sense that the customer base is still rooting for the company to find its footing again.

    Second, there’s the culture. Most of the front-line people and important mid-level to senior managers who grew up in Instructure’s culture of excellence are still there. They still have the same knowledge, skills, experience, and convictions. And they have a collective muscle memory of how to work together in ways that were successful for customers in the past. Until we start seeing either talent flight or layoffs that cut deeply into the wrong parts of the company, then the heart of the original company is still beating.

    As for the core platform, while it’s no longer dramatically differentiated, it’s still solid. All things considered, nobody enjoys migrating to a new LMS. Instructure is still on a glide path to at least maintain its market share in the US and grow abroad. They would have to actively screw that up for the situation to change. And while that is easier than it may seem, it would take a while.

    Finally, while the market has caught on to the value of Canvas being designed as a cloud-based application with a focus on end-users, there is one other novel decision the company made early on that is still underappreciated. Canvas was designed to be a platform, not an application. To grasp the difference, think about the difference between a Blackberry (if you are old enough to remember those) and an iPhone. The Blackberry was a good phone that had a great keyboard. So, in a world where texting and email were becoming at least as important as voice communications, adding text communication features to a phone was a good idea. Yes, there were a few “apps,” for Blackberry. But there weren’t many, and most of them were bad. The iPhone, in contrast, was built for apps. Instead of making the primary interface a great physical keyboard, Apple made it a pane of glass. The interface was anything that could be created for a touch screen. Including interfaces and applications that Apple hadn’t even dreamed of. The iPhone is a platform. It is infrastructure that is designed for other people to write applications that run on it and through it.

    Instructure made a series of principled decisions not to develop certain capabilities in Canvas. Instead, they built both the technical infrastructure in Canvas and the relationships with integration partners to have a very strong ecosystem of EdTech companies that have deep relationships with the company and deep integrations with the platform. Rather than building a great keyboard and email app, they built a metaphorical pane of glass and app store. They created a technology platform and ecosystem that attracted other people to extend Canvas with new capabilities.

    This too was a pioneering strategy. Before Instructure, the model for LMS partner relationships had been set by Blackboard in the Chasen/Small era. They recognized that the vast majority of digital education products would have to integrate with the LMS. So they decided to monetize that by charging every integration partner a (significant) fee. In those early days, there was some business justification for that strategy. After all, Blackboard had invested in developing integration APIs—Blackboard Building Blocks—and, since they were by far the largest player in the market, integration with their platform offered substantial business opportunities in the form of access to Blackboard’s customer base. This became a significant revenue source for Blackboard that its competitors envied and, to varying degrees, tried to emulate.

    But just as Instructure was lucky to come along right when the nature of the LMS customer was shifting, and right when cloud-based development had matured enough to make building a cloud-based LMS feasible, they were also lucky to come to market right when the IMS LTI interoperability standard was starting to take off and EdTech venture funding was really taking off. The LTI technical integration standard made charging a toll harder to justify, while the proliferation of EdTech startups transformed the locus of value for an LMS from a Swiss army knife into the hub of an ecosystem. Or, as my friend Kelvin Thompson from UCF has memorably characterized it, the potato part of Mr. Potato Head. On the one hand, there is no Mr. Potato Head without the potato. On the other, the potato itself is not where the personality (and personalization) come from. Instructure was less focused on charging partners and more interested in enticing them into adding character to Canvas in new and imaginative ways.

    Dalek Mr. Potato Head
    Homer Simpson Mr. Potato Head

    Their platform differentiator is subtle in the LMS world, but one that is critical to their current situation.

    In fact, it is essential to the first of two paths Instructure has for developing a product portfolio that I’m going to describe in the remainder of this post.

    Acquisitions

    There is now a whole galaxy of small EdTech companies that have developed new products or services that touch the LMS in one way or another. Instructure arguably is in the best position of any of the players in the market to identify good companies with products that already integrate well with Canvas and snatch them up. It’s not an exceptionally strong or durable advantage, but it’s one that they could be utilizing aggressively right now.

    Former Instructure CEO Josh Coates had a philosophy about acquisitions. Specifically, he was against them most of the time. I heard him give a talk about it once. His strategy was thoughtful and well-reasoned. And it would have been justified had the company succeeded in organically developing a portfolio of compelling offerings. But that didn’t happen. So, at least for the short term, the company is going to need to be more acquisitive in order to become financially healthy. But for that strategy to work, Instructure will need to rely on the knowledge of its employees who have worked on the platform aspects of Instructure’s business to identify truly good companies. What I mean by “truly good companies” is ones that have earned customer loyalty by solving some important problem well. Instructure needs to find compelling products, which is not easy in EdTech. The company leaders will also need to clearly and consistently articulate the reasons why they are proud to have acquired those products and how the acquisitions will better serve their Canvas customers.

    Instructure has made one high-profile acquisition under CEO Dan Goldsmith: Portfolium. Here’s the relevant bit from Instructure’s press release announcing the acquisition:

    Portfolium was created to help every person realize their full potential by connecting their learning with opportunity. The company helps institutions inspire, assess, and showcase student achievements via its powerful ePortfolio network, student-centered assessment, job matching capabilities, and academic and co-curricular pathways.

    “Working with Portfolium advances our mission since it enables us to help people move from the classroom to the workplace,” said Dan Goldsmith, CEO of Instructure. “Portfolium has been a great partner of ours. With their team, and by adding their student success capabilities built on the leading learner network, we will, together, provide more value to both current and new customers.”

    Instructure Enters Into Agreement to Acquire Student Success Network Portfolium

    Honestly, I don’t know what that means. Portfolium is an ePortfolio. That blurb mostly describes what an ePortfolio does, but adding some pleasant adjectives. Shorter version:

    Instructure is acquiring Portfolium, which is an ePortfolio company. Portfolium does things that ePortfolios do. We like it and think you will too.

    Honest press release

    The only bit in there that isn’t completely generic is the sentence fragment about job matching. But they don’t do anything with it.

    In fairness, most people who haven’t written a press release don’t appreciate how hard it is to write a meaningful one. I’m a pretty good writer, but I will readily admit to having struggled with that particular genre at times. That’s exactly why smart companies don’t rely on the press release to carry all the weight. They go out and repeat and elaborate on the message. Relentlessly.

    I haven’t heard any such message about Portfolium.

    Again, in fairness, I have not been following the LMS market as closely as I used to. But the thing is, if Instructure were really doing this right, it shouldn’t have been possible for me to miss this. I should have read it in the articles in outlets like EdSurge, Inside Higher Ed, and The Chronicle. In preparing to write this post, I did a search to see if I missed anything in the coverage by these or any other outlets.

    Nope.

    No quote from the CEO beyond what was in the press release. In fact, no quote that I could find from anyone that wasn’t already in the press release. There was an interview with the Portfolium founder in the San Diego Tribune. But it was clearly a local business story, in a local outlet—Portfolium was San Diego-based—and not anything aimed at Instructure’s customers. There’s nothing. Nada. Zip.

    When I ask Instructure customers about the deal, they tell me they haven’t heard anything either. And when I have run into Instructure employees at conferences—specifically, ones who are in a position to know about Portfolium—none of them have brought it up with me. In the old days, they would have. These are the same humans. There has been no invasion of the body snatchers. So it appears that the corporate communication strategy has changed. Where it used to be the case that you could get any random Instructure employee driving a golf cart at Instructurecon ((Golf carts driven by employees at Instructurecon is an actual thing. Or at least, it used to be.)) to talk about just about anything, it is now difficult to get even senior Instructure managers to talk about…well…just about anything.

    So there is at least one and possibly two serious but very fixable problems here. First, there is definitely a communication problem. Specifically, it would be good to have meaningful communication directly from the people within the company who understand why acquisitions are being made. Instructure employees have been some of the best brand ambassadors in the sector. Circumstantial evidence strongly suggests that they are no longer empowered to speak for the company. By muzzling them, Instructure is voluntarily throwing one of its remaining competitive strengths in the garbage can. And the only senior manager who has been allowed to speak on the record about the Goldsmith era’s first important, high-profile acquisition—to anybody, apparently—is the CEO. The one who has the most to prove about actually knowing the sector and caring about it. And his only statement that I can find amounts to a spoonful of bland pap in a press release.

    I don’t know much about sports, but I know what an unforced error is.

    While it would be easy to lay much of this at the feet of the marketing department, there has been a pattern of communication in the Goldsmith era that started immediately after he became CEO and has been consistent despite some churn in marketing department senior personnel. In fact, that churn may be an indicator in itself. The leader sets the tone.

    The second possible problem is harder to assess precisely because of the effects of the first one. I can’t tell if Portfolium is a good acquisition or not. A lot depends on what problems Instructure’s managers think it solves for customers and how they intend to make it more useful. Which they’re not really talking about. I can’t tell if this acquisition was a paint-by-numbers decision or if there is a real effort to increase value for the customers in a mindful, meaningful way. Instructure has people who know which partners are good potential acquisitions and why. They know their partners and customers really well. I can name a handful of them off the top of my head. In general, they are not being heard externally, which is unfortunate. If they are also not being heard internally—and again, I can’t tell one way or another—that would be a lot worse.

    I am confident that Instructure has the right ingredients and the right chefs to cook up an effective acquisition strategy. But the proof of the pudding is in the eating. If Instructure doesn’t both prioritize the acquisition of companies that will serve their customers well and communicate the reasoning and intentions behind the acquisitions, that could easily mean the difference between customers who are hungry for more and ones who are left with a bad taste in their mouths.

    Insights

    The other area for growth is in providing educators and students with better insights that support student success. It is impossible to overstate both how important and how fraught a topic this is.

    On the one hand, colleges and universities fail huge swathes of students all too often. There are students who want to go to college, get admitted to college, but haven’t had anybody to teach them how to succeed in college. Students who never make it to the first day of class. Or who get partway to a degree and drop out, due to lack of skills and support or tough personal circumstances. Students who need to come back to school and reskill while they are holding down full-time jobs and raising families.

    Many colleges and universities are not very good at serving some or all of these groups and routinely fail them. This used to be “only” a moral failing. Now it is an existential one, because the most obvious path to long-term sustainability for an increasing number of colleges and universities is to serve more students in their area more effectively for 20 or 40 years rather than for two or four. Institutions of higher education need to learn to perform better. And they need to learn it urgently. To do that, they need new insights. To get new insights, they need better information—that is, better data—and better ways of analyzing it.

    On the other hand, we live in an era when people have good reason to be concerned about the misuse of data in a seemingly ever-increasing variety of deeply troubling ways. Educators are responsible for their students. Many of them take this responsibility very seriously indeed. And particularly when it comes to the experimental use of data—even for the best of purposes—that responsibility is hard-wired into institutional processes in very particular ways.

    In academia, educational research falls into a larger bucket of “human subjects research.” That category also includes research on topics like how to perform open-heart surgery, how to deal with intractable clinical depression, and how people can be manipulated or fooled by social media. Think about the possible unintended consequences of poorly designed experiments in any of those three areas. We have canonical examples from bygone eras. The Milgram experiment. The Stanford Prison Experiment. Today, any academic research conducted in the United States that involves human subjects must submit its experimental design and protocols for experimental subjects’ informed consent to a rigorous peer review and approval process before the experiment can be undertaken. Educational research must be submitted to the same approval process by the same oversight board that would approve life-and-death surgery or psychological experiments.

    But unlike in medicine, an EdTech company that wants to conduct research using student data for product development purposes is required to do…nothing at all. They don’t even have to inform the students that they are being experimented on.

    In fact, EdTech companies conduct experiments on a weekly basis that would require a lengthy approval process in some universities. ((Different universities interpret the rules around such approvals differently, in part because a university that has a world-class medical school will likely understand their responsibilities differently than one with no medical school at all but a significant psychology research program, for example.)) Suppose, for example, that product developers want to test which design of a button is more likely to raise user awareness of a feature. They conduct what is known in the industry as an “A/B” test. They show some users one design, other users the other design, and track which design gets the most clicks. This is an absolutely routine software development practice. It is a foundational strategy that developers use to learn how to make their software more useful and usable.

    But let’s also suppose that the feature the button activates has a significant impact on improving student outcomes. Using the feature helps to improve learning. The developers, with the best of intentions, are trying to figure out which version of the button will get more students to use the feature that will help them. But in this experiment, the button that turns out to be worse may negatively impact the learning outcomes of students using that version of the software.

    This is exactly the kind of contingency that might trigger a requirement for an experimental design review process inside a university.

    Imagine that you’re an academic who feels responsible for your students and who lives in that kind of a culture surrounding anything that remotely looks like it could be human experimentation. Imagine that you read the following statement by the CEO of a company whose EdTech product you, personally, require your students to use extensively every single day:

    We’ve been working on the scaffolding for [DIG] for well over a year now. I mentioned in our remarks that we already have product validation towards out there in the market. We have instructors and students consuming output from some of the initial experiments with DIG. And we anticipate later this year obviously to make more announcements around specific products and offerings and how we bring them into the market. DIG ultimately is a platform first and foremost based upon machine learning and artificial intelligence. I believe that any multi tenant SaaS company born in the cloud has the opportunity once they hit a certain market share. And in fact, it may even be incumbent upon those organizations to partner with the industry and evolve that industry with new insights and predictive modeling using AI and ML. That’s what DIG is at its heart.

    We already have analytical capabilities in our Canvas platform. I want to be really clear and delineate the difference between an analytics and reporting capability, and a machine learning and AI platform. [snip]

    We have the most comprehensive database on the educational experience in the globe. So given that information that we have, no one else has those data assets at their fingertips to be able to develop those algorithms and predictive models.

    Instructure CEO Dan Goldsmith

    Heads would explode. Heads did explode. Heads are still exploding.

    Even so, as I said in my previous post, that was a very fixable problem. Dan was still new. He easily could have played the “new guy” card. A mea culpa, a couple of comments to reporters, and a brief but earnest listening tour likely would have blunted the worst of it. The company could have reset and been in a position to have productive conversations with customers about this thorny set of challenges that they need to face together. Instead, Goldsmith said nothing, and the problem festered.

    In July, Instructure VP of Higher Education Jared Stein wrote a blog post on DIG trying to settle things down and dispell some of the concerns. I trust Jared and, more importantly, Instructure customers trust Jared. He did a decent job in that post, as far as it went. But one blog post by a senior employee, three months later, is not going to undo the damage of such a bad faux pas by the CEO. A CEO trumps a VP. Therefore, a CEO’s misstatement can only be credibly corrected by the CEO. Furthermore, Pandora’s Box had been opened. Because Instructure didn’t jump on the lid the moment they saw it crack open, all the deeply difficult questions about uses of student data in EdTech have come flooding out. The company didn’t lose everybody’s trust, but they lost the trust of enough vocal customers that now they have a persistent problem.

    This too is fixable, but it must be fixed. Some damage has been done to Instructure’s reputation. Enough that a more concerted and sustained effort must now be made which includes actions and not just gestures. But the imbroglio has not yet permanently damaged the company. People have long memories of their experiences with Instructure and personal relationships with employees who still work there.

    Update: Jeff Young just published a piece out about the data concerns in EdSurge. It includes quotes from Instructure’s chief spokesperson Cory Edwards, who I don’t know well but have found so far to be a good actor, and Melissa Loble, a long-time Instructure vet and current SVP for Customer Success, who is exactly the kind of person we should be hearing from directly more often. There’s even a quote from a Thoma Bravo representative. All were responding directly to the data use uproar. So this is progress. But still nothing from CEO Dan Goldsmith. Why EdSurge was able to get Instructure’s prospective PE owner on record but not its CEO, about a problem that was set off by a comment made by that CEO…it’s just mystifying.

    Education needs insight-providing, data-driven tools to help educators better serve a wider range of students who could succeed if only we were offering them the right kind of help. Furthermore, educational institutions need productive partnerships with the private sector to get these solutions out to as many students as possible as quickly as is possible and responsible. But this partnership can only take place in a high-trust environment. Instructure has had the necessary level of trust from their clients to do this kind of work. They have damaged that trust in this particular area. But not beyond repair. There is still a sound foundation, and any cracks still can be repointed.

    This brings me to Instructure’s most compelling product. It is not Canvas. Canvas has only been Instructure’s second-most compelling product.

    Brand, brand, brand

    Most people interpret “brand” as shorthand for a banal series of tactics employed by marketing departments. Academics, in particular, are inclined to load the word with distasteful connotations of shallow window dressing at best and obnoxious disingenuousness at worst.

    Nothing could be further from the truth. “Brand” is another word for reputation. It is who people think you are. It’s how much they trust that you are who you say you are. How much they trust you, period. Brand isn’t a series of tactics. It is the outward manifestation of the character of an organization or individual, as understood by people who have come to know them based on their actions over an extended period of time. Far from being a small set of eye-rolling marketing gimmicks executed by a small set of individuals, an organization’s brand is the gestalt impression that people get from every single interaction they have with every single member of that organization and every single product, website, or other touchpoint. That gestalt is “monetizable” to the extent that people trust the organization to understand their needs and have their interests at heart. They will give you their money if and when they believe they can trust you with it.

    Instructure’s brand has, until now, been its primary and best product. It is still one of the best in the sector, even if it is getting a little ragged around the edges. Because the brand is still good, the company can still build the relationships it needs to make good acquisitions, evangelize those acquisitions to its customers, and work with its customers on even the most sensitive (and important) product research and development efforts.

    That is what I hope for Instructure and why I have expended so much energy writing these last two blog posts. I want to live in a world where EdTech vendors are successful because their customers and partners believe in them. Instructure has been that kind of company. And it still could be.

    But back to the proof of the pudding. In this 21st-Century economy, as the saying now goes, if you’re not at the table, you’re on the menu. Instructure can recover, succeed, and thrive to the degree that the company’s leadership can reignite their customers’ faith that they have a seat at the table. Every decision they make, including financial transactions, should be judged by how it helps or hinders them from doing so.

  • Instructure’s Proposed Acquisition is a Bad Risk for Everyone

    I did not want to write this post. I really didn’t.

    For starters, I’ve been delighted to get away from this sort of corporate analysis and accountability writing, knowing that Phil has it well covered on his blog. That goes doubly for LMS inside baseball. Second, having recently announced a major sponsorship from one of Instructure’s competitors, there’s no way I can write about this topic without the appearance of a conflict of interest, particularly when my assessment is negative. Third, After watching the fights about what the acquisition means breaking out on social media, I have about as much desire to insert myself in the middle of that as I do to stick my index finger into my garbage disposal.

    But people keep asking me to write about Instructure’s prospective acquisition by Private Equity (PE) company Thoma Bravo. And it’s a consequential moment. One of only a very few mainstream LMS providers is at an inflection point. Perhaps more than customers realize. I have become increasingly worried that, by going through with this particular sale at this particular time, there is a very high risk of destroying the company’s value to customers and shareholders alike.

    So here I go. Stickin’ my finger in the garbage disposal.

    In this post, I’m going to explain why I worry the current deal on offer to take Instructure private runs a high risk of being bad for everyone.

    Instructure probably needs to do this at some point

    Let’s start with a basic reality: The reason that Instructure is moving toward a sale to private equity is that the company is at an inflection point and, despite its history of success, could very easily fall apart. Most Instructure customers don’t see this; what they see is a company that keeps growing and, on the surface, doesn’t seem to have changed dramatically. But major shareholders and board members have a different perspective. They see financial problems down the road, and they also see that life under private equity could buy the company the time it needs to address these challenges in ways that staying publicly-traded might not. There is some debate among shareholders about how quickly this move must be made and how to best accomplish it. But I haven’t heard objections from these stakeholders that taking the company private is a bad idea in general. I agree with them on that point. I personally believe that a sale to the right PE firm is probably a good idea in principle, both for shareholders and for customers.

    Instructure’s competitors have made a big deal out of the fact that the company still isn’t profitable after all these years. That’s a bit of a red herring. Instructure has chosen to reinvest profits in the company. When you do that, the “extra” money the company has made from a sale is no longer called “profit.” But that is a choice.

    The real problem is that selling Learning Management Systems is, in and of itself, not a very good business. Yes, Instructure could have been profitable sooner. But not very profitable. The way that LMS companies become financially healthy in the long run—to the extent that they do—is by selling other products and services to their existing LMS customer base. They increase the number of dollars per customer that they earn by selling those customers other things. (We’re going to return to that last sentence later in this post, because even though it is true, it is framed in a way that often leads EdTech Private Equity acquirers to make very bad decisions.)

    Instructure has a strong and growing customer base (even if the rate of growth has slowed recently). That’s why investors are attracted to them. What the company lacks is a portfolio of other products and services to sell. They have a good foundation, but they need to build on it. So there are a few critical questions that I am going to address in this post which are relevant to everyone who wants to see Instructure become more valuable rather than less:

    1. How did the company manage to grow so dramatically for so long?
    2. What does the company need to do in order to avoid breaking the engine that enabled them to acquire and retain customers?
    3. What would a credible plan for increasing Instructure’s future value to customers look like?
    4. What is the gap between a credible plan and what Instructure has presented so far?
    5. Why is that gap bad for current shareholders and dangerous for customers in the context of a potential acquisition?

    Before I get to these questions, though, I want to address a source of anxiety that I am seeing on social media that I think is misplaced. There is a narrow and outdated notion of what PE does that is leading to some unrealistic fears among some concerned academics. While there are real risks to worry about here, it’s important to focus on the fears that are realistic.

    Private equity is…equity that is private

    From a definitional standpoint, a private equity (PE) company purchases equity—stock shares—of the companies it invests in, but it does so outside of the publicly traded stock markets. For many years, private equity companies tended to follow one of only a couple of well-defined playbooks. Sometimes they would be house flippers, fixing up an undervalued house and then turning it over (relatively) quickly for a profit. Sometimes they would be junk dealers, stripping a car with a broken engine for the parts that still had value. Sometimes they would be slumlords, extracting as much “rent” from the asset as they could while putting as little investment in maintenance as they could. While the word “slumlord” is (intentionally) pejorative, some of these models could work out fine for the customers and employees, depending very much on the specifics. More on that in a bit.

    Now that the private equity markets have reached over three trillion dollars (and growing rapidly), there is a lot more diversity in the kinds of investors buying equity privately. Their goals, risk tolerance, and strategies vary pretty widely. There are, for example, family offices where rich people invest their money in individual companies rather than on the public markets. Depending on the goals of the family, the behavior of these funds can be quite long-term-minded and benign. Or not. You just can’t conclude much from the label “private equity” by itself anymore.

    Here’s another way to think about it: Academics tend to be suspicious of venture capital, private equity, and IPOs. Basically, any funding is anxiety-provoking because of the potential strings attached. In Instructure’s case, they earned a spectacular reputation with customers while being venture-backed. They’ve done less well, but not horribly, since their IPO. What can we conclude, then, about the relative badness of VC-backed companies versus publicly traded ones?

    Uh…not much. It comes down to the alignment of incentives between the investors and the customers. Understanding that will help to clarify which fears about this particular acquisition are realistic and which are not.

    Instructure has no parts to fleece

    One concern I’ve read is that the new private equity (PE) owners will sell off parts of the company. To which I reply, Which parts are you worried about them selling off?

    Canvas?

    No. There would be no company without Canvas.

    Arc?

    Are people massively freaked out about the potential sale of Arc? I’m guessing not.

    Portfolium?

    I’m not saying anything bad about the product itself (or any of Instructure’s products), but is all of this hand-wringing about the possible sale of Portfolium? Doubtful.

    Bridge?

    Hmm. Let’s set Bridge aside for a moment, look at a situation with an LMS company where selling off parts actually was a viable business strategy, and then return to it.

    Blackboard’s PE owners sold off CashNet less than a year ago. It was an absolute nightmare for the company’s bread-and-butter education customers.

    Oh, wait. No, it wasn’t.

    Blackboard Learn customers, by and large, didn’t even notice that CashNet was gone. It was an asset that had more value to those customers in the form of the cash that Blackboard got for it than in the actual ownership of the product. This was a case in which selling off a part of the company was both a good strategy from a financial perspective and inconsequential at worst to customers.

    In fact, Instructure is struggling now largely because it doesn’t have enough valuable parts to sell. To anyone, including current customers. The company failed to develop a strong portfolio of products they could “cross-sell.” Going back to Blackboard for a moment, however much people may grumble about them (rightly and/or wrongly), when the conversation shifts to the topic of their Ally accessibility product, eyes light up. It solves a real and important problem. People want it. Current Learn customers want it. Prospective Learn customers want it. Customers who wouldn’t touch Learn with a 10-foot pole want it. Therefore, it has financial value.

    Instructure needs a portfolio of offerings that are compelling as Blackboard’s Ally in order to be financially viable in the long term. Because the company has thus far failed to develop such a portfolio, it developed an alternative growth strategy to sell an LMS into the corporate market instead. And not its existing LMS. No, a new one. Bridge.

    This, frankly, was a dumb idea that was never going to work. Where the LMS is a mission-critical application that inspires passion—positive or negative—from users in the education markets, it is the polar opposite in the corporate markets. Whenever corporate budgets get slashed, Training and Development is the first departmental budget to get hit. And unlike the academic LMS market, there are somewhere in the neighborhood of 173,000 corporate LMS products. OK, I’m exaggerating. A little. But it’s a large enough list that whole companies have been built on writing up catalogs that describe the different LMSs in that market. Not even reviewing them, really. Mostly just listing them all in one place.

    Ask yourself how detrimental it would be for you if Instructure sold off or killed off Bridge. If your answer is “not very,” then you don’t have much to worry about in terms of new owners selling off pieces of the company.

    It’s important to understand that a profit motive and a motive to serve students and educators well are not inherently opposed to each other. They can be. They are almost always in tension with each other. But if the best way to make money is to serve students and educators well, then good results for education can be driven by a profit motive. Money will be more likely to be invested in the right things. I say “more likely” because the other challenge is knowing how to invest that money such that it will be likely to have the net effect of serving students and educators well. My father likes to say, “Never attribute to malice that which simple incompetence can explain.” More on this later in the post.

    The PE acquisition would be to support a return to a focus on education

    One way to read the PE acquisition is as an admission of failure of the Bridge strategy. That is certainly, explicitly, the line that Instructure’s activist investors have been pushing. If you’re worried about the impact of the short-sighted view of PE, then you should also be worried about the quarter-to-quarter pressures of the public markets. Instructure’s Bridge strategy failed to produce the financial results that were promised. One argument for PE acquisition in this situation is that the public markets would not be patient with Instructure as it attempts to recover from this misstep and build the product portfolio that it should have started working on five or more years ago. The right PE firm would give Instructure time to ditch Bridge and refocus on its core market. (That’s you.)

    I don’t know much about Instructure’s PE acquirer, Thoma Bravo. Phil describes their investment style thusly:

    Thoma Bravo is widely known for the “buy and build” acquisition strategy, where a platform company with solid customer base is purchased (often for high price), enabling subsequent acquisitions of smaller companies that have lower price multiples. This is not the same as buying company A and B and combining them; rather, this strategy is based on multiple acquisitions tied to the platform company.

    That fits with what Instructure needs to do to become more sustainable in its core education markets. It needs more educational products to sell. So at first blush, Instructure being owned by Thoma Bravo could be better for customers than Instructure being publicly traded has been.

    Interestingly, though, the current Instructure shareholders who oppose this acquisition also argue that Instructure should return its focus to its core market of education. They just don’t think this particular deal is the best way to do it and are suspicious that there are other, more venal reasons for this particular merger at this particular time. Here, for example, is what Instructure investor Rivulet argued in its SEC filing protesting the merger:

    We invested in Instructure because we admire the company, and we see plenty of opportunity for continued growth and success. With Canvas, Instructure has developed the market-leading learning management software product for the higher education market. The Canvas story has provided an incredible lesson about the success that comes from a focus on innovation, putting students first, and offering a compelling alternative to a greedy incumbent that was run for the enrichment of management and shareholders. Unfortunately, with the announced sale of the company to Thoma Bravo, Instructure is providing a different sort of lesson to its customers and shareholders. Here, it appears that the company has run a rushed strategic review process that was designed to result in a sale to management’s chosen buyer at a low price…in order for management to save their jobs and enrich themselves in the years ahead. It is outrageous.

    Rivulet SEC filing

    In other words, investors both in favor of the merger and opposed to it agree that Instructure is a growth company that has stalled because it has lost focus on its core education markets. They believe the company can continue to serve its shareholders well by serving its education customers well. The debate among them is over whether this deal was really designed to serve those customers (and thus generate profit for shareholders), or whether it was corrupted by an agreement that exchanged preferential treatment of Thoma Bravo in the bidding process for particularly rich compensation for Instructure CEO Dan Goldsmith and his sister, who Goldsmith hired as a senior executive of the company.

    I will not write about the details or the merits of those accusations in this post. That’s not e-Literate‘s beat anymore, and I have no insights to offer about them that would add any value. If you want to read more insightful and in-depth coverage of that fight than I could offer, then go read Phil Hill’s coverage.

    Rather than speculating about motives or analyzing the finances, I am going to write about strategy and execution. Does Instructure’s current executive team have a credible strategy and a strong track record of execution that is consistent with its previous, incredibly successful strategy for growing and retaining its installed base? And does it have a credible, market-validated strategy for increasing the value that it provides to its customers such that customers will likely be willing to pay more for that value? If the answer to these questions is “yes,” then current shareholders can have increased confidence that they will receive a good price for their shares while the end users of Instructure’s products and services can have increased confidence that they can continue to count on the company to serve them well.

    Unfortunately, it looks to me like the answers to these questions are “no.”

    Why Instructure has succeeded—and failed—to date

    We have long argued at e-Literate that Instructure came to market with three critical advantages that incumbent competitors were slow to see and which were difficult, time-consuming, and/or expensive for them to replicate. Those advantages are as follows:

    1. Reliability (implemented as cloud-native infrastructure)
    2. Ease of use (implemented as a rethink and redesign of existing LMS functionality)
    3. Customer rapport (implemented through a variety of formal and informal ways and tied together by a strong company culture)

    These three advantages can be collected under one unifying strategic insight of the original executive management team: They brought a consumer software sensibility to what had been treated by incumbent vendors as an enterprise software business.

    Instructure understood who their customer really was in a changing market

    For readers who are not software industry aficionados, one way to get at this difference is to ask yourself a few questions about software products being used at your institution that are generally treated as enterprise software products. How involved were you in selecting your registrar software? How broadly and intensively were different stakeholder groups involved in the selection process? Who made the final decision? Do you even know the answers to any of these questions? Do you know how the decision was made?

    Generally speaking, registrar software is part of a larger package of software that includes modules for things like payroll and expense tracking. There is usually a very small group of people who both make the decisions on the procurement of such software and who also decide who else will have input on the process. This is typical with enterprise software selection—even for critical enterprise software that has many end-users who depend on it and use it daily. For enterprise software companies, their customers are those few decision-makers. They only have to care about reliability, ease of use, and end-user happiness to the degree that the decision-makers do. And often, those decision-makers have other priorities, like cost, or arcane but important support for regulatory requirements, or making their bosses happy in some way.

    In the early years, LMSs were procured like enterprise software. CIOs generally selected the LMS vendor and decided on whose input they would take when making their selection. But by the time Instructure came on the scene, that was changing. Faculty were much more intensively involved with LMS selection, and it was not unheard of for a university to form a student advisory committee or conduct focus groups as part of the selection process. Instructure made and kept three promises to these stakeholder groups. First, they would build and run the LMS such that it would be extremely unlikely to go out for two weeks at the beginning of the semester or in the run-up to finals week. Second, it would be less painful and time-consuming to use than the alternatives which existed at the time. And finally, end-users would be treated as human beings, with good customer service and both visibility and input into the product roadmap.

    Given the pedigree of the executive team at that time, these changes should not be surprising. CEO Josh Coates’ previous gig was running Mozy, a startup that was essentially the precursor to Dropbox. Think about what life was like trying to back up or share locally stored files across multiple devices before Dropbox. There was FTP. There were email attachments. And there was Sharepoint. But in order for a Dropbox product to replace those well-entrenched incumbents, it had to be rock-solid reliable, be dead simple to use, and feel completely approachable and safe.

    Instructure has been spectacularly successful not because it has been a great education company but because it has been a great consumer software company. And the early management team was very clear about this distinction. Here’s what I wrote in a 2012 post:

    The founders of Instructure aren’t educators, and they think that’s a good thing. The way they see it, educators tend to build software that solves their own problems from their own perspective, and the results are often idiosyncratic. As technologists coming in with no strong opinions about how teaching “should” be done, the Instructure leadership feel they have an advantage of humility and open-mindedness when considering solutions. As CEO Josh Coates put it, educators “are experts in their context. But their context is one or two classes in a specific university in a specific part of the country. Software developers don’t even pretend to know the domain.” So they have to go out and do the research. Instructure co-founder Brian Whitmer added, “And we know that we have to do it. We know that we have to validate it against a bunch of different people.” And they do. Instructure, as a company, is extremely attentive to the conversations among educators, to the point where they have Twitter feeds from various ed tech folks sucked right into the IRC channel that all their developers use for internal communications. They do their homework.

    Educators tend to get hung up on the profit motive and, ironically, miss the disciplinarity of starting a company. Co-founder Devlin Daley talked about a “new style of development” that was “not true in the ’90’s,” and all three of them talked a lot about Agile development. As they see it, a lot of companies that implement Agile miss the fundamental aspect of the methodology that is about getting closer to customer needs.

    e-Literate, “What Are Ed Tech Entrpreneurs Good For?

    Instructure lost its way after solving the obvious problems

    In a lot of ways, Instructure got a big leg up by being late to the market. The product category was well defined through the experiences of the early incumbents. Does an LMS need a gradebook? Definitely. A test engine? For sure. A blogging system? Ehh…. Instructure’s product team could go to a lot of people who were already using LMSs and ask them, “What do you hate about this thing and what are you really trying to accomplish that is possible but painful in it?”

    And, in fact, that is exactly what Instructure did. They figured out what problems people were trying to solve when they used an LMS and then, rather than building yet another, more feature-packed version of the same tool, they set out to build solutions to those problems. For example, they figured out that instructors who were entering grades wanted to do so as quickly as possible and focus their energy on adding educationally useful comments. So rather than cramming more features into the same spreadsheet-style gradebook that every other LMS had implemented, they built SpeedGrader, a novel user experience that helped instructors focus on the aspects of grading that they cared about while moving other functions out of the way. Why should instructors have a grade curve-setting widget taking up screen space when they’re trying to comment on a student’s paper?

    But once Canvas had caught up functionally with the other LMSs, Instructure seemed to run out of ideas. Internal to the LMS, we haven’t seen many truly major customer-facing improvements in the past four years. In fairness, Instructure claims to have been doing a lot of under-the-hood work that will yield future benefits. But I’m still fairly unclear on what those future benefits will be.

    Meanwhile, what new customer problems have they solved outside of the LMS? What new products have they given us?

    Arc. They gave us Arc. A lecture capture tool. Another lecture capture tool. Is it good? Yeah, it’s pretty good. Did it set the world on fire? No, it did not.

    I don’t want to overplay the we-don’t-need-another-one-of-those card, because Canvas itself could have been dismissed on those grounds when it first came out. In fact, it was dismissed on those grounds. I, personally, dismissed it on those grounds.

    Nor is it the only example of such a product. Not by a long shot. Consider Zoom. It dominates the webconference market, both in education and elsewhere. How did that happen? Cisco already had a very mature and very widely adopted product in Webex. Google had Hangouts. Adobe had Connect. Blackboard had the education-specific Collaborate solution. There are multiple open source options, like Big Blue Button. There are other options like GoToMeeting and Bluejeans. There was no good reason to believe that a startup with yet another webconferencing solution would survive, nevermind thrive.

    And yet, it did.

    Why? This is the critical question for Instructure to both maintain its current strength and build new ones. Maybe Instructure just got lucky by coming in when they did. Maybe they weren’t nearly as talented as they seemed to be. Humans tend to over-credit skill and under-credit timing and luck.

    Even so, I don’t believe that luck and good timing were the primary reasons for Instructure’s success. Permit me to repeat a snippet from the quote above of my 2012 post:

    Co-founder Devlin Daley talked about a “new style of development” that was “not true in the ’90’s,” and all three of them talked a lot about Agile development. As they see it, a lot of companies that implement Agile miss the fundamental aspect of the methodology that is about getting closer to customer needs.

    I believe Instructure had an approach to building their business that worked. They lost some mojo because, among other reasons, the pressure for them to grow pushed them into flattening what had been a philosophy and corporate culture into something closer to a paint-by-numbers formula. This is one reason why sales have slowed. Instructure is no longer performing as a truly great company. And if they don’t find their mojo again before they enter into an acquisition, then things are far more likely to get worse—possibly catastrophically worse—than they are to get better, because building a culture of product development excellence is not one of the things that PE knows how to do well. Whatever patterns are there when the company is purchased are likely to stay. Whatever direction the CEO has set will stay. And without the bulwark of leadership protecting a strong culture of excellence, the tendency to paint by numbers is likely to get worse rather than better. PE firms may or may not understand education, but they all understand numbers and find a high level of comfort in them.

    Product-led Growth

    Back when Instructure was being started, the software product development buzzwords were “Agile” and “Lean.” Today, a new buzz-phrase that also fits with that quote above is “product-led growth.”

    “Product-driven company” can be defined narrowly or broadly. The narrower version involves some fremium model where people adopt a free version of the product through a self-service process, use it a lot and, once again through self-service with no human salesperson involved, upgrade to a paid version. Zoom works like this. So does Dropbox. I had a free Dropbox account for a long time. To borrow a phrase from Apple, it “just worked.” I used it to the point where I needed more space. So I paid for an upgrade. At peak Dropbox for me, I had a personal paid account as well as a larger one for my company.

    Spreadsheet jockeys like product-driven companies because they have low sales and marketing costs. People adopt the product because it’s easy to do so. It spreads because people tell their friends and colleagues about how good it is. It makes money because people love and use the product so much that they will pay for an upgrade. The customers do much of the sales and marketing work and pay for the privilege.

    But as with Agile, people who implement a product-driven company design miss the fundamental aspect of the methodology that is about getting closer to customer needs. And again, Josh Coates’ last gig before becoming Instructure’s (previous) CEO was building the precursor to Dropbox. There wasn’t a big self-service component to Canvas sales, but the core philosophy of creating a customer experience that carries a lot of the sales and marketing load by inspiring fanatical loyalty was there.

    At least, it was there in the beginning, and for quite a while. It has been less evident in recent years. When a company is under pressure to grow, either financially or just to scale up to meet the needs of new customers that are walking in the door, it is easy to slip from a focus on finding new problems to solve toward one of finding new things to sell. In EdTech, there aren’t many obvious things to sell, particularly “at scale.” Painting by numbers under these circumstances is a weak strategy that fails often.

    But education has many, many problems to solve. If you have a strong rapport with your customers, a really solid team of education specialists who understand the complex nuances of educational problems, and a leadership team that is absolutely committed to growing by understanding their customers’ needs better than anyone else does, then you can find ways to make money by solving new problems. Product-driven companies are problem-driven companies.

    A good example of solving new problems is Blackboard’s decision to acquire Ally. Was there a healthy, established product category for an educational content accessibility checking platform that would be sold to colleges and universities? Absolutely not. Could anyone calculate the total addressable market (TAM) for such a product without wildly guessing? Not in any way that I can think of. Were Blackboard’s customers begging them to build and sell such a product? I very much doubt it, since there really wasn’t any such thing on the market.

    But Blackboard saw that the little startup that had found this problem to solve. I’m sure they talked to Ally’s customers—and their own—about it to make sure that it was a real and large problem, solved by the product in a useful way. And they made a bet. It’s turning out to be a good one.

    In retrospect, you could argue that the product is adjacent to Blackboard’s LMS business. But almost everything in EdTech is adjacent to the LMS because almost everything has to connect with it or work with it. If you had grabbed a random EdTech mid-level manager off the street and asked them for their top suggestions for hot product categories that are adjacent to the LMS, I doubt that learning content accessibility would have shown up on anybody’s list. This is the kind of thinking that made Instructure, and it’s the kind of thing it needs to be doing again if it wants to find its next leg of growth while holding off its newly resurgent competitors.

    So who did Instructure’s board hire to lead the charge at this critical moment to reinvigorate that deep culture of devotion to solving the problems of the end-users? A guy whose last job was running a company that sold enterprise software to life sciences companies.

    Now, Dan Goldsmith is not his résumé. He’s a multidimensional person who is capable of learning. (And to be fair, Josh Coates was not an obvious candidate for the job either.) But Dan has been in his current job for less than two years. In order for him to establish that he is up to this challenge in this industry within that length of time, he would have had to perform extraordinarily well. This is vital right now because the very same shareholder ballot they are voting on to accept Thoma Bravo’s offer would also change Dan’s compensation package to one that would attempt to lock him in with a very generous amount of stock that vests over time. If Instructure’s board and shareholders have decided that now is the time to go private, with this CEO, then they need to be very confident that he will deliver. If they are not, then the customers are under increased risk, the new owner is under increased risk, and the current shareholders are less likely to get attractive competing bids for their shares.

    Has Dan Goldsmith met this high bar in the brief 20-month period that he has been on the job?

    In my opinion?

    No.

    A case study in the opposite of what Instructure needs to do

    Instructure hasn’t made a lot of significant moves in the education space since Dan took the helm. Off-hand, I can think of two. The first was the acquisition of Portfolium. In and of itself, that’s not an obviously genius move. The ePortfolio product category has been around almost as long as the LMS product category and has been a hot seller since…uh…never.

    It could turn out to be an interesting move. Instructure has to bet that (a) potential trends like comprehensive learner records or stackable credentials will turn into something more substantial and that (b) an ePortfolio in general and this ePortfolio, in particular, provides a good infrastructure from which to start supporting those trends. I’m troubled by the fact that I’ve not heard an articulation from the company about their vision for the product. Dan made some comments about the potential for cross-selling. But those comments characterize Portfolium as a thing to sell, not as a solution to a customer problem. I simply don’t know what they’re thinking with Portfolium.

    The other move was Dan’s announcement about their DIG learning analytics initiative in the spring of 2019, which Phil covered in his own inimitable way:

    The second initiative announced on the earnings call was DIG, a strategic move with data and analytics.

    “I am also pleased to share with you an early insight into our second growth initiative focused on analytics, data science and artificial intelligence. The code name for this initiative is DIG. And this technology platform combined with the most comprehensive SaaS database on the educational experience uniquely positions us to deliver meaningful value to our customers. And from a growth perspective, DIG has the potential to double our TAM in education.”

    Instructure started ramping up their data and analytics efforts (again) about a year ago, although the focus was described at the time as being about internal analytics – that is, making Canvas a better and more valuable LMS product. From what I have heard the product validation for DIG are consistent with this message – dashboards, surfacing useful data within a workflow, etc. But that was not how DIG is being sold during the conference call [emphasis added].

    “We’ve been working on the scaffolding for [DIG] for well over a year now. I mentioned in our remarks that we already have product validation towards out there in the market. We have instructors and students consuming output from some of the initial experiments with DIG. And we anticipate later this year obviously to make more announcements around specific products and offerings and how we bring them into the market. DIG ultimately is a platform first and foremost based upon machine learning and artificial intelligence. I believe that any multi tenant SaaS company born in the cloud has the opportunity once they hit a certain market share. And in fact, it may even be incumbent upon those organizations to partner with the industry and evolve that industry with new insights and predictive modeling using AI and ML. That’s what DIG is at its heart.

    This is brand new behavior for Instructure as a company. Previously the company was reticent to talk much about non-released products, but now they are talking not just about a new initiative, they are touting buzzwordy machine learning and artificial intelligence and predictive modeling well before any of those capabilities exist or are in customer hands. Goldsmith further clarified the DIG plans during the investor conference discussion [starting at 9:00, emphasis added].

    “We already have analytical capabilities in our Canvas platform. I want to be really clear and delineate the difference between an analytics and reporting capability, and a machine learning and AI platform. [snip]

    “We have the most comprehensive database on the educational experience in the globe. So given that information that we have, no one else has those data assets at their fingertips to be able to develop those algorithms and predictive models.”

    Goldsmith then described an example of predicting a student’s expected performance in a class and how that prediction reliability goes up over time. Then we get the vision.

    “What’s even more interesting and compelling is that we can take that information, correlate it across all sorts of universities, curricula, etc, and we can start making recommendations and suggestions to the student or instructor in how they can be more successful. Watch this video, read this passage, do problems 17-34 in this textbook, spend an extra two hours on this or that. When we drive student success, we impact things like retention, we impact the productivity of the teachers, and it’s a huge opportunity. That’s just one small example.

    “Our DIG initiative, it is first and foremost a platform for ML and AI, and we will deliver and monetize it by offering different functional domains of predictive algorithms and insights. Maybe things like student success, retention, coaching and advising, career pathing, as well as a number of the other metrics that will help improve the value of an institution or connectivity across institutions. [snip]

    “We’ve gone through enough cycles thus far to have demonstrable results around improving outcomes with students and improving student success. [snip] I hope to have something at least in beta by the end of this year.”

    Wow. Robot tutor in the sky – meet the new kid on the block.

    For those readers who don’t follow the sector broadly, Phil’s reference to a “robot tutor in the sky” is from an infamous quote by the CEO of now-defunct Knewton that they liked to think of their product as “a robot tutor in the sky that can semi-read your mind.” In other words, Phil is accusing Goldsmith of using tone-deaf language that would be explosive to educators. And sure enough, it pissed a lot of people off. They didn’t like the vague hype, and they didn’t like the seemingly out-of-the-blue implication that students’ data from their in-course activities would be used without their permission for Instructure’s product development purposes. As I mulled whether and how to write this post, I received the following reply on Twitter to my retweeting of one of Phil’s posts on the topic:

    OK, OK, so somebody on the internets wrote a mean thing about Instructure’s DIG strategy. And is still upset nine months after the announcement. Maybe that evidence doesn’t persuade you (particularly if you don’t know who Laura Gibbs is).

    How about this?

    That’s a Twitter thread from Canvas end-users who are crowdsourcing a letter encouraging their institutions to lobby Instructure shareholders about Instructure’s lack of legally binding data privacy policies. Now, the authors had voiced a range of concerns in the last draft (plus comments) that I looked at before writing this post. Some of them may have been poorly phrased, overblown, or factually questionable. But others were unquestionably legitimate, sophisticated, and serious.

    Update: The authors of the letter have posted the most recent draft here, and a signable petition here.

    I see three big take-aways from this letter which are most important for our current purposes. First, this is the first time in my long history of covering EdTech that I have ever seen educators undertake such an effort to influence shareholders. The last company I would have expected to inspire this level of concern-driven activism is Instructure. And the existence of it strikes me as one of the most singular and remarkable milestones of Goldsmith’s tenure to date. Trust me: This should be taken seriously.

    Second, one reason this foment continues to build nine months after Dan’s original comments about DIG is that, at least to my knowledge, Dan himself has yet to address these customer concerns directly, publicly, and credibly. Even in this Twitter thread, he leaves the job to an anonymous PR person to desperately try to tamp down the anger. At this point, given what’s at stake, he should be making a direct personal statement as the CEO of the company. If not on Twitter, then on the company blog. Or somewhere. Anywhere.

    Finally, the authors of the letter make clear they understand that there are no obvious, well-trodden solutions to the student data privacy conundrum. They request some specific steps at the end of their letter, but they don’t pretend that these steps solve the complex, pervasive, and important challenges that universities who work with EdTech products face. They are struggling to balance the need to protect student privacy against the affirmative ethical obligation to learn how to better help their students succeed.

    THIS IS A PROBLEM THAT CUSTOMERS NEED HELP SOLVING.

    Instead of either assuming a defensive crouch about DIG or ignoring complaints altogether, Instructure could be listening to customer concerns with an ear toward making money by serving them better. This is a hard and important problem space that will not be solved without active participation by vendors. There have been academic convenings on the student data privacy challenge by groups like Asilomar. They do critical work. But they are not enough. There are initiatives by industry groups like IMS. They too are necessary but not sufficient. This is an important and growing problem, which makes it an important and growing commercial opportunity for a vendor that can actively partner with academia in finding a solution to it.

    Dan Goldsmith could have easily turned his faux pas into a moment of leadership. There is even precedent from Instructure for this under strikingly similar circumstances. I encourage you to (re)read Phil’s 2015 post on how Josh Coates handled customer complaints about the company charging universities for access to their own students’ data. The short version is that once Josh became aware of customers’ unhappiness, he took it as an opportunity to engage with them, changed the policy, published a mea culpa post on the company blog, and encouraged Phil to independently verify with the customers that their concerns had been addressed.

    Dan’s comments were significantly less serious than the implemented commercial practice that Josh had to correct. He could have easily addressed them. And there is a real problem space here that cries out for product development. EdTech needs an architecture of privacy. Instructure’s customers are being very direct about their needs and concerns. The company even has access to multiple academic experts who could help them think through the nuances of the problem. I know this for a fact because I introduced Instructure employees to those experts last spring when I urged them to turn the DIG situation around by taking leadership.

    That was over half a year ago. And now we have customers banding together to lobby shareholders about their continuing dissatisfaction. Because apparently, they have come to the conclusion that the company leadership is not listening to them.

    No. I would not bet the company on the 20-month record of Dan Goldsmith’s leadership and articulation of strategic direction. Maybe things will be different six or twelve months from now. But at this point, he has neither embodied nor provided a sound investment thesis. There is no strong reason to believe that Instructure, sold to private equity under this leadership at this moment, would be able either to continue to grow market share of its core product or to develop new solutions to real customer problems that would lead to greater profitability.

    Just how bad it could get

    I want to wrap up this post with a couple of hypothetical but plausible examples of just how bad things could get when an LMS company with no products to cross-sell and no customer-focused, problem-solving leader sit down in a room with PE owners who have big ambitions, a lot of money, and limited experience in the space. A player like Thoma Bravo is going to make acquisitions. If they make a number of small bets that don’t pay off, then there are likely to have to cut costs and raise prices to make up for their losses.

    But what I really worry about are the kind of colossal mistakes that can happen when a CEO who doesn’t know education and investors who don’t know education lock themselves in a room together and try to come up with something really big. Phil noted in a recent post about Thoma Bravo,

    Another point in the Forbes profile is that Thoma Bravo does not acquire a company and then figure out what other companies to tuck-in or combine [emphasis added].

    “Like a good tennis player who’s worked relentlessly on his ground strokes, Bravo has made private equity investing look simple. There are no complicated tricks. He figured out nearly two decades ago that software and private equity were an incredible combination. Since then, Bravo has never invested elsewhere, instead honing his strategy and technique deal after deal. He hunts for companies with novel software products, like Veracode, a Burlington, Massachusetts-based maker of security features for coders, or Pleasanton, California-based Ellie Mae, the default system among online mortgage lenders, which the firm picked up for $3.7 billion in April. His investments typically have at least $150 million in sales from repeat customers and are in markets that are too specialized to draw the interest of giants like Microsoft and Google. Bravo looks to triple their size with better operations, and by the time he strikes, he’s already mapped out an acquisition or turnaround strategy.

    Instructure fits into this playbook, and I suspect that we won’t have to wait long to see the next move. It could be tuck-ins in the vein of Portfolium, Practice, or MasteryConnect, or it could be something bigger and market-changing. It is worth noting that Bravo raised $12.6 billion in January of this year – the stakes are getting higher.

    What would fit into the category of “bigger and market-changing”? I can think of only two plausible acquisitions, particularly since Thoma Bravo seems to be disinclined to invest in people-heavy businesses like OPMs or textbook publishers. Either one of these ideas would be so incredibly, mind-bogglingly stupid that I’m hoping this part of the post is just good for laughs. (Or maybe a Halloween-style scare. Boo!)

    The first possible merger target is Ellucian which, as Phil has pointed out, has put itself up for sale. On the spreadsheets, this could look like a great prospect for Instructure. They sell to the same client base, have a big installed base, a portfolio of products to cross-sell, and have a massive lock-in.

    Anyone who actually knows EdTech and is thinking from a product-led perspective would see this as the obviously terrible idea that it is.

    In general, the SIS market is a smoking ruin that will take at least another half a decade to move itself into the cloud. The technological heritage and history of the product category conspire to make Ellucian—and their competitors—about the furthest thing from a product-led company possible. Think about how hard it was for Instructure’s competitors to make the transition they needed to make in order to catch up. Now multiply the challenge by a kajillion. Combining Ellucian with Instructure would be a disaster.

    The only worse idea I can think of that might somehow seem plausible to the wrong people having the wrong conversation at the wrong time would be for Instructure to buy—are you ready for it?—Blackboard. Again, from a paint-by-numbers perspective, it makes perfect sense. Instructure could buy Blackboard’s remaining customer base very cheaply, along with its more successful portfolio of cross-selling products. (Their Ally accessibility platform, for example, is a well-deserved smash hit.) And since Blackboard is close to having zero financial value, with their debts being close to the intrinsic value of their assets, the sellers could unload an investment that is no longer performing for them while the buyers could get assets they need practically for free.

    And in the process, Instructure would acquire the worst brand in the product category—which is an improvement from when they were the worst brand in the history of the sector—by applying the most infamous and most loathed business strategies of their infamous former Blackboard CEO Michael Chasen—namely, buying up one of the few competitors in the market. Instructure could literally become the new Blackboard. In doing so, they would utterly destroy one of the greatest brands in the history of the sector instantaneously.

    These guesses must be wrong. (Dear Lord, please let them be wrong.) Think of them as hypothetical examples of “really, Michael, how bad could it get?” than as actual predictions. I have to believe that the board, the executive team, and Thoma Bravo would not be that dumb. But the point is that of those three entities—the board, the financial owner, and the CEO—the only one we would reasonably expect to know how bad such decisions would be in practice, the only one that all parties must be able to count on to steer the ship away from the many rocks and icebergs in EdTech, is the CEO.

    Has Instructure provided customers with a detailed and credible enough strategic roadmap to inspire confidence that they have a more compelling alternative for growth? No, they have not. Has Dan Goldsmith thus far proven, lacking such a roadmap, that his reputation for performance alone is worth betting the company on? No, he has not. No smart PE company would make an attractive counter-offer under these circumstances. There is no sound investment thesis until Instructure is able to regain its footing as a product-led company.

    Instructure has been one of the best run, most consequential companies in the history of EdTech, and it could be again. It has not yet experienced the talent flight that would make a turn-around much, much harder. But until it has both a credible strategy and a leader who inspires customers to trust the company with their most pressing and complex problems—like fulfilling their ethical obligation to protect their students’ privacy in an era of digital learning—it will be just another EdTech company that is toiling away in an unprofitable product category and promising investors that it will somehow spin straw into gold.

  • Announcing a Lesson-level Interoperability Standards Effort

    I’m delighted to announce a project aimed at making it easier to share interactive digital content at the lesson level as well as to establish baseline educational analytics for digital curricular materials. I’m tempted to call this a “courseware” interoperability effort, but its potential application is broader than that term would imply to some folks. For example, the work could support well-structured content in LMSs.

    This effort is consistent in philosophy with my recent “Content as Infrastructure” post series as well as the post of a version of my IMS talk on interoperability, learning analytics, and pedagogical intent. One of the main outputs of the project will be a white paper, released as an Empirical Educator Project (EEP) contribution, describing the standards proposal that is ultimately developed and its value to education. The project also dovetails very nicely with both previously announced and as-yet-unannounced EEP projects, and I’m very excited about the work.

    The idea for the work both grew out of and is funded through a grant from the U.S. Department of Education (ED) to develop “active OER.” In the course of the grant planning process, ASU professor and grant PI Ariel Anbar came to the conclusion that the grant would have a much broader impact if the content being developed were interoperable. He consulted with ED, and they agreed that interoperability would potentially increase the impact of the resources related to the grant. So a small fragment of the grant budget was carved off to test the viability of building a coalition that can make useful progress on proposed standards definitions that are both practically useful and likely to be adopted. At the moment, my work as a facilitator is the main budget expense for the project.

    Business and mission goals

    We had a kick-off meeting of a small group in late October. (More on who was in it, why they were chosen, and how we hope to expand later in this post.) Here are the notes I captured on the goals and ambitions for impact:

    • Reduce platform lock-in for any interactive courseware content, particularly interactive OER content, which will support the following:
      • Increase the quality of existing OER content by enabling the preservation of learning design
      • Increase the supply of interactive OER content by creating a clear and achievable interoperability standard for content developers
      • Increase the availability and value of OER content for value-added platform and service providers by lowering the cost of goods involved in converting the currently available “flat” OER resources into interactive lessons with effective learning design 
      • Enable educators to more easily mix and match interactive content at the lesson level
      • Enable the development of an ecosystem of non-OER content that could be licensed at the lesson level
    • Enable lesson-level, cross-platform, cross-content learning analytics which will support the following:
      • Data-based continuous improvement of learning content, regardless of its source
      • Baseline student learning analytics capabilities that will enable institutions to monitor student progress in an apples-to-apples way across lessons, products, and courses
      • Student- and instructor-facing analytics that will help them analyze how well their respective learning and teaching strategies impact outcomes
    • A vision for implementation and ecosystem development that incentivizes participation for a wide range of commercial and non-commercial value-added participants in order to:
      • Lower the barrier to adoption for courseware platforms by assuring customers that any content they develop or use will not be locked into the licensed platform
      • Lower the cost-of-goods and availability of high-quality pre-existing content for value-added OER curricular materials product and service providers
      • Enable micro-licensing models for commercial content vendors that develop high-value lesson-level content
      • Lower the barrier for non-profit organizations and consortia to create interactive content that is competitive in functionality and measurable quality with baseline student and teacher expectations for commercial courseware

    I’ll provide more of my personal take on these goals in a subsequent post, but there is consensus in the group that we should be working toward a set of goals that are good for everyone—students, instructors, institutions, and value-added content and platform providers.

    Functional and technical goals

    Consistent with the posts I linked to at the top of this one, we’re going to start by identifying questions that educators and students would want to answer about student progress, effectiveness of content design, and effectiveness of learning intervention. Our default atomic unit for this work is the “lesson.” Our starting point for identifying this set of questions will be the ones that the participating implementors have identified as ones that their users/adopters/customers want to answer, but I expect that we’ll expand from that base over the two-year life of the initial project.

    Once we know what questions we want to answer, then we will identify the metadata for the content that is needed to answer those questions. For example, which learning objective(s) does this assessment question assess? Is the assessment formative or summative? That sort of thing. No firm decisions have been made about how this would work on the technical level, but the basic idea is that the pedagogical intent of the learning design would be captured in some machine-readable form.

    Technically, we’d like to build on as much existing standards infrastructure as possible and propose developing as little new work as possible. While the project, as a piece of a larger ED grant, does not have a formal affiliation with an interoperability standards body, I am pleased to say that IMS Global and its CEO, Rob Abel, have been highly encouraging and offered technical support to the group as we think through the effort. IMS has a lot of the infrastructure that would be needed for the effort already baked into its existing specifications. It makes all the sense in the world to try to re-use or extend standards that are already developed and adopted.

    Ultimately, the group will produce a set of recommendations for interoperability standards along with a rationale for those recommendations. The hope and intention are that these recommendations will be taken up and carried forward by the appropriate bodies at the end of the project and that the participants will continue to work together on implementation.

    More on process and risk management

    Standards development is a tricky business. You want to get to an “everybody in the pool” moment, but at the same time, you can’t win everybody over by promising to boil the ocean. So we thought a lot about how to get this process rolling and balance different risks over time.

    At my suggestion, we started by inviting in just a few of the many implementors who ultimately should be at the table. Two—Carnegie Mellon University’s Open Learning Initiative and Lumen Learning—are long-time and active participants in the OER world. While this effort will be helpful to more than just OER, the primary purpose of the grant is for the development of (inter)active OER, so we wanted representatives who could speak to the needs and nuances of the OER ecosystem. The two other implementors we invited—Smart Sparrow and CogBooks—are courseware platform implementers that both work extensively with ASU already. Smart Sparrow is also playing a major role in this OER grant since Ariel has chosen its Inspark Education network to help manage the grant and is building the content on the Smart Sparrow platform. Also, CMU, Lumen, and Smart Sparrow have all been participants in EEP. In addition to the implementers and ASU, we had representatives from Scottsdale Community College and ED at the kick-off meeting.

    This is a small enough group with enough interconnections that we have a good chance of making progress on scoping goals without excessive amounts up-front diplomacy required but diverse enough that we would get different opinions and perspectives. It’s a good group for getting started and for testing the basic idea that what we want to accomplish is doable within a reasonable period of time. Ultimately, however, the project will need more and different folks to be involved if it going to result in broadly implemented interoperability standards. The starting group of four implementer participants is going to work toward a letter they all can sign onto that says they are committing in principle to implement any standards that ultimately flow out of this effort. The value in their commitment at this early stage is to decrease the risk of other implementors who may want to join but are skeptical that the effort will produce results. In parallel, the project is seeking additional funding that would enable us to support the participation of more stakeholders—educators, platform implementers, content developers, and standards committees (and possibly students as well).

    It is early days for this work. So far, the group has only met once. There is still a lot to do and a lot to be figured out. But I am hopeful that we can both develop useful recommendations for advancing interoperability standards and pioneer some new ways of working together on productive EdTech collaboration in the process.

  • Pedagogical Intent and Designing for Inquiry

    I’ve been asked by several folks to write up some version of the talk I gave at the recent IMS learning analytics summit. The focus was on how, going forward, interoperability standards will need to capture pedagogical intent if we are going to develop meaningful learning analytics.

    This isn’t a word-for-word transcription of that talk, but it does capture the gist. The subtitles and literary quotes are mostly from the original presentation. Many thanks to Rob Abel and Cary Brown for inviting me and giving me the opportunity to speak to the IMS community about this important inflection point.

    Interoperability is communication

    We are not here to curse the darkness, but to light the candle that can guide us through that darkness to a safe and sane future.

    John F. Kennedy

    Ten or fifteen years ago, when we talked about what we wanted from EdTech software interoperability, most of the time the things we wanted seemed like they ought to be simple. Mostly, we just wanted to transfer student roster and grade information from one system to another. When we got a little more ambitious, we asked for single sign-on and the ability to put a little window of one application into another one. This was foundational interoperability. It wasn’t “disruptive” or “revolutionary” or otherwise life-altering, but it was important.

    My first close encounter with an IMS interoperability effort was when I worked at Oracle in on Peoplesoft Campus Solutions team. We just wanted to be able to send a course roster to the LMS and have the LMS send a final grade for each student back. Going in, I couldn’t understand why that was hard or why it hadn’t already been done. All I knew was that (a) it was a problem that IMS had tried and failed to solve at least once before, since we were working on the revision of an existing standard, and (b) a consequence of that failure was that IT professionals on campuses everywhere had to cook up their own, often time-intensive, duct-tape-and-chewing-gum solutions to getting roster data into the LMS. As for grades, it made no sense to me that faculty had to copy grades from their electronic LMS grade book and manually re-enter them into their electronic SIS grade book. It seemed weird that this was a thing.

    It turned out that there was a translation problem. Registrars think about classes very differently than instructors and students do. For a registrar, if a student is taking a “statistics for psychology majors” course, and that course can be taken for credit in either the psychology or the math department, then “statistics for psychology” is actually two completely separate courses. And the decision of which of the two courses a student registers for may make the difference between that student meeting the requirements for graduation or not. On the other hand, the students and instructor experience “statistics for psychology” as one course meeting in one place on one schedule with one syllabus and one group of people.

    Good software reflects and supports the needs and expectations of its users. Accordingly, SIS software represented “statistics for psychology” as two courses, while LMS software wanted to create one course space for it. In order to have roster information show up as instructors and students expect it in the LMS and then return final grades as the registrar expects them in the SIS, the standards committee had to recognize that there was a translation problem and design a specification that could function as a two-way translator.

    This was an important lesson for me. Even seemingly simple interoperability challenges can be complicated because they often aren’t about the software so much as they are about the people who use the software. Interoperability design is at least partly a liberal art.

    Now, today, we would have a different possible way of solving that particular interoperability problem than the one we came up with over a decade ago. We could take a large data set of roster information exported from the SIS, both before and after the IT professionals massaged it for import into the LMS, and aim a machine learning algorithm at it. We then could use that algorithm as a translator. Could we solve such an interoperability problem this way? I think that we probably could. I would have been a weaker product manager had we done it that way, because I wouldn’t have gone through the learning experience that resulted from the conversations we had to develop the specification. As a general principle, I think we need to be wary of machine learning applications in which the machines are the only ones doing the learning. That said, we could have probably solved such a problem this way and might have been able to do it in a lot less time than it took for the humans to work it out.

    I will argue that today’s EdTech interoperability challenges are different. That if we want to design interoperability for the purposes of insight into the teaching and learning process, then we cannot simply use clever algorithms to magically draw insights from the data, like a dehumidifier extracting water from thin air. Because the water isn’t there to be extracted. The insights we seek will not be anywhere in the data unless we make a conscious effort to put them there through design of our applications. In order to get real teaching and learning insights, we need to understand the intent of the students. And in order to understand that, we need insight into the learning design. We need to understand pedagogical intent.

    That new need, in turn, will require new approaches in interoperability standards-making. As hard as the challenges of the last decade have been, the challenges of the next one are much harder. They will require different people at the table having different conversations.

    Data and communication are not the same

    O wonder!
    How many goodly creatures are there here!
    How beauteous mankind is! O brave new world
    That has such people in’t!

    William Shakespeare

    The quote above is from The Tempest. Here’s the scene: Miranda, the speaker, is a young woman who has lived her entire life on an island with nobody but her father and a strange creature who she may think of as a brother, a friend, or a pet. One day, a ship becomes grounded on the shore of the island. And out of it comes, literally, a handsome prince, followed by a collection of strange (and presumably virile) sailors. It is this sight that prompts Miranda’s exclamation.

    As with much of Shakespeare, there are multiple possible interpretations of her words, at least one of which is off-color. Miranda could be commenting on the hunka hunka manhood walking toward her.

    “How beauteous mankind is!”

    Or. She could be commenting on how her entire world has just shifted on its axis. Until that moment, she knew of only two other people in all of existence, each of who she had known her entire life and with each of whom she had a relationship that she understood so well that she took it for granted. Suddenly, there was literally a whole world of possible people and possible relationships that she had never considered before that moment.

    “O brave new world / That has such people in’t”

    So what is on Miranda’s mind when she speaks these lines? Is it lust? Wonder? Some combination of the two? Something else?

    The text alone cannot tell us. The meaning is underdetermined by the data. Only with the metadata supplied by the actor (or the reader) can we arrive at a useful interpretation. That generative ambiguity is one of the aspects of Shakespeare’s work that makes it art.

    But Miranda is a fictional character. There is no fact of the matter about what she is thinking. When we are trying to understand the mental state of a real-life human learner, then making up our own answer because the data are not dispositive is not OK. As educators, we have a moral responsibility to understand a real-life Miranda having a real-life learning experience so that we can support her on her journey.

    Intention matters in education

    With regard to moral rules, the child submits more or less completely in intention to the rules laid down for him, but these, remaining, as it were, external to the subject’s conscience, do not really transform his conduct.

    Jean Piaget

    The challenge that we face as educators is that learning, which happens completely inside the heads of the learners, is invisible. We can not observe it directly. Accordingly, there are no direct constructs that represent it in the data. This isn’t a data science problem. It’s an education problem. The learning that is or isn’t happening in the students’ heads is invisible even in a face-to-face classroom. And the indirect traces we see of it are often highly ambiguous. Did the student correctly solve the physics problem because she understands the forces involved? Because she memorized a formula and recognized a situation in which it should be applied? Because she guessed right? The instructor can’t know the answer to this question unless she has designed a series of assessments that can disambiguate the student’s internal mental state.

    In turn, if we want to find traces of the student’s learning (or lack thereof) in the data, we must understand the instructor’s pedagogical intent that motivates her learning design. What competency is the assessment question that the student answered incorrectly intended to assess? Is the question intended to be a formative assessment? Or summative? If it’s formative, is it a pre-test, where the instructor is trying to discover what the student knows before the lesson begins? Is it a check for understanding? A learn-by-doing exercise? Or maybe something that’s a little more complex to define because it’s embedded in a simulation? The answers to these questions can radically change the meaning we assign to a student’s incorrect answer to the assessment question. We can’t fully and confidently interpret what her answer means in terms of her learning progress without understanding the pedagogical intent of the assessment design.

    But it’s very easy to pretend that we understand what the students’ answers mean. I could have chosen any one of many Shakespeare quotes to open this section, but the one I picked happens to be the very one from which Aldous Huxley derived the title of his dystopian novel Brave New World. In that story, intent was flattened through drugs, peer pressure, and conditioning. It was reduced to a small set of possible reactions that were useful in running the machine of society. Miranda’s words appear in the book in a bitterly ironic fashion from the mouth of the character John, a “savage” who has grown up outside of societal conditioning.

    We can easily develop “analytics” that tell us whether students consistently answer assessment questions correctly. And we can pretend that “correct answer analytics” are equivalent to “learning analytics.” But they are not. If our educational technology is going to enable rich and authentic vision of learning rather than a dystopian reductivist parody of it, then our learning analytics must capture the nuances of pedagogical intent rather than flattening it.

    This is hard.

    Some more examples

    The lesson assessment example is easy enough to understand. (My post series on content as infrastructure explores it in more detail.) But the more one looks around at the full range of analytics that are truly aimed at supporting student success, the clearer the lesson about capturing intent becomes.

    Take, for example, summer melt. I recently just hosted an entire hour-long Standard of Proof webinar on this topic. (You should watch it. It’s good.) Here’s a slide from that webinar which illustrates the obstacles that first-generation students face in getting from their college acceptance letter to the first day of class:

    The Summer Melt Maze

    Credit: Lindsay Page

    Each of the text labels represents an obstacle that first-generation students in particular may struggle to overcome, because their circumstances are complex (e.g., no parent to provide a parental signature), because they don’t have parents or guardians who have the training and knowledge to help them, and/or because they’re seventeen-year-old kids. I don’t know about you, but I don’t think I had to navigate a single one of these obstacles without parental help, and I don’t know if I could or would have done so without it.

    Now think about developing a software solution to identify the specific barrier a student is struggling with and provide appropriate help. Could you solve the problem simply by sucking in enough data and running a machine learning algorithm? I very much doubt it. And even if you could, think about how invasive you would have to be to do so. Think about the privacy implications. The cure would be worse than the disease.

    Georgia State University took a different approach. They have invited students to share their intent. They use a chatbot. A chatbot is a conversational interface. Students can ask directly about the problems they were encountering. Once students specify their intent, then machine learning can be used to further disambiguate it. For example, the software can figure out that a student who writes “I have no money” may be asking for help obtaining financial aid. But only because there is a conversational interface, and because the software behind that interface has been programmed to anticipate and respond to a certain range of questions that a student might come to the chatbot for answers. Through a combination of the interface layer, the data layer, and the usage context, the educational intent was encoded into the system.

    Here’s another example that postdates my talk. ACT recently released a paper on detecting non-cognitive education-relevant factors like grit and curiosity through LMS activity data logs. This is a really interesting study that I hope to write more about in a separate post in the near future, but for now, I want to focus on how labor-intensive it was to conduct. First author John Whitmer, formerly of Blackboard, is one of the people in the learning analytics community who I turn to first when I need an expert to help me understand the nuances. He’s top-drawer, and he’s particularly good at squeezing blood from a stone in terms of drawing credible and useful insights from LMS data.

    Here’s what he and his colleagues had to do in order to draw blood from this particular stone:

    The online interaction features were generated from the LMS clickstream data. After manual inspection, we determined that the action field alone (e.g., “opened”) was insufficient to address our research questions and needed to be joined with the label of the item that the action was taken in reference to, which was a complex pairing. For example, course item values in the LMS data include “Week 12: Electrochemistry” or “CHEM 102 Practice Exam 4B,” which were easily interpretable from the course syllabus, while others (e.g., “2/27 CL” or “18.7 RQ”) required confirmation from the instructor. Hence, we created broader activity categories for these activity events in the LMS data using the course syllabus with confirmation from the instructor which resulted in 110 unique activity events in the LMS data that were recoded to a total of 21 activity categories as described in Table 4.

    First, they had to look at the syllabi. With human eyeballs. Then they had to interview the instructors. You know, humans having conversations with other humans. Then the humans—who had interviewed the other humans in order to annotate the syllabi that they looked at with their human eyeballs—labeled the items being accessed in the LMS with metadata that encoded the pedagogical intent of the instructors. Only after they did all that human work of understanding and encoding pedagogical intent could they usefully apply machine learning algorithms to identify patterns of intent-relevant behavior by the students.

    LMSs are often promoted as being “pedagogically neutral.” (And no, I don’t believe that Moodle is any different.) Another way of putting this is that they do not encode pedagogical intent. This means it is devilishly hard to get pedagogically meaningful learning analytics data out of them without additional encoding work of one kind or another.

    Interoperability without intent creates chaos

    If Jorge Luis Borges’ Library of Babel could have existed in reality, it would have been something like the Long Room of Trinity College.

    Christopher de Hamel

    I want to underscore the point that simply collecting more data and writing more clever algorithms will not help us find a way out of the problem of that last example. It is a problem of epistemic closure. Data and knowledge are not the same, and more data do not necessarily unlock more knowledge.

    “The Library of Babel” is a short story by the great Jorge Luis Borges. (It’s only nine pages long. You should read it.) The story describes a world that perfectly captures the nature of the problem we face:

    The universe (which others call the Library) is composed of an indefinite and perhaps infinite number of hexagonal galleries, with vast air shafts between, surrounded by very low railings. From any of the hexagons one can see, interminably, the upper and lower floors. The distribution of the galleries is invariable. Twenty shelves, five long shelves per side, cover all the sides except two; their height, which is the distance from floor to ceiling, scarcely exceeds that of a normal bookcase. One of the free sides leads to a narrow hallway which opens onto another gallery, identical to the first and to all the rest. To the left and right of the hallway there are two very small closets. In the first, one may sleep standing up; in the other, satisfy one’s fecal necessities. Also through here passes a spiral stairway, which sinks abysmally and soars upwards to remote distances….

    There are five shelves for each of the hexagon’s walls; each shelf contains thirty-five books of uniform format; each book is of four hundred and ten pages; each page, of forty lines, each line, of some eighty letters which are black in color. There are also letters on the spine of each book; these letters do not indicate or prefigure what the pages will say….

    The orthographical symbols are twenty-five in number. This finding made it possible, three hundred years ago, to formulate a general theory of the Library and solve satisfactorily the problem which no conjecture had deciphered: the formless and chaotic nature of almost all the books. One which my father saw in a hexagon on circuit fifteen ninety-four was made up of the letters MCV, perversely repeated from the first line to the last. Another (very much consulted in this area) is a mere labyrinth of letters, but the next-to-last page says Oh time thy pyramids. This much is already known: for every sensible line of straightforward statement, there are leagues of senseless cacophonies, verbal jumbles and incoherences….

    Five hundred years ago, the chief of an upper hexagon came upon a book as confusing as the others, but which had nearly two pages of homogeneous ines. He showed his find to a wandering decoder who told him the lines were written in Portuguese; others said they were Yiddish. Within a century, the language was established: a Samoyedic Lithuanian dialect of Guarani, with classical Arabian inflections. The content was also deciphered: some notions of combinative analysis, illustrated with examples of variations with unlimited repetition. These examples made it possible for a librarian of genius to discover the fundamental law of the Library. This thinker observed that all the books, no matter how diverse they might be, are made up of the same elements: the space, the period, the comma, the twenty-two letters of the alphabet. He also alleged a fact which travelers have confirmed: In the vast Library there are no two identical books. From these two incontrovertible premises he deduced that the Library is total and that its shelves register all the possible combinations of the twenty-odd orthographical symbols (a number which, though extremely vast, is not infinite): Everything: the minutely detailed history of the future, the archangels’ autobiographies, the faithful catalogues of the Library, thousands and thousands of false catalogues, the demonstration of the fallacy of those catalogues, the demonstration of the fallacy of the true catalogue, the Gnostic gospel of Basilides, the commentary on that gospel, the commentary on the commentary on that gospel, the true story of your death, the translation of every book in all languages, the interpolations of every book in all books.

    Jorge Luis Borges

    The rest of the story is a rumination on how humans might make sense of this infinite series of rooms—this “data lake,” in modern parlance—in absence of any information about the intent of its creator.

    Since the Library contains every possible book with that number of pages, lines and characters, somewhere in all these rooms must exist a book that explains exactly what it all means. So it’s a data search problem, right?

    Wrong. Because there also exist books that are extremely similar but differ in minor but critical details. And books that argue why the book with the Truth is actually false. For that matter, there are books that contain many of the exact same words in the exact same order but are written in languages in which the words mean different things. How can one tell which account is the Truth?

    One can’t.

    If we are going to make progress toward educationally useful analytics, then we must ruthlessly expunge all traces of magical thinking about data. There are fundamental limits to what the data can tell us. Even systems that are designed for pedagogical intent do not necessarily encode it in a way that is useful for interoperable analytics. In some cases, it may be encoded at the user interface layer. A courseware authoring platform may never label an assessment as “formative” or “summative” in the data because the intended distinction is obvious to the users. In other cases, the data may be encoded in an idiosyncratic manner that does not map well to other systems (either of software or of thought). In still other cases, it may be designed badly and incorrectly or misleadingly reflect the pedagogical intent. It would be relatively easy to create a data lake of Babel which we could explore infinitely in a fruitless search for meaning.

    That way lies madness.

    If we want useful educational analytics, then we cannot simply worship the data and the algorithms. The humans must do some of the learning.

    The “semantic web” is all about intent

    I love you. You are the object of my affection and the object of my sentence.

    Mignon Fogarty

    One of the triggers for my being invited to speak at IMS about learning analytics in the first place was a previous post I wrote which was (partly) on how the structure of Caliper, which is borrowed from the structure of the semantic web, supports new sorts of interoperability conversations. Since you can read that blog post, I won’t repeat the argument in detail here. But the gist is that we both have to and can boil down chains of inference that combine pedagogical intent into simple human language that educators can understand and articulate for themselves. The triple structure of the semantic web—a simple three-word sentence with a subject, a verb, and a direct object—is designed to enable non-technical humans to string thoughts and inferences together in ways that enable more technical humans to translate those inference patterns into data structures and interoperability requirements.

    Right now, IMS Caliper adopters are largely using this vernacular as just one more IT tool for largely old-school data centralization. So now they use a “lake” instead of a “warehouse.” It’s still a centralized and IT-specialized mindset which is not suited for thinking about making meaning from multiple applications, never mind talking with other humans about interpreting the intent of users working across multiple applications.

    This is the problem that must be solved over the next decade to make real progress on educational analytics. It is at least partly a liberal arts problem. And it will be at least a decade’s worth of work, though we don’t have to wait that long to see early results.

    You get to choose the world we live in

    O wonder!
    How many goodly creatures are there here!
    How beauteous mankind is! O brave new world
    That has such people in’t!

    Shakespeare or Huxley?

    So which will it be? A brave new world in which we experience continuous wonder at how beauteous mankind is, or one in which “learning” is defined by the ability to correctly answer a series of questions and earn some digital badges? The difference between these possible futures is not one in which we embrace or reject technology, or data. It’s one in which we either embrace or ignore the complexity of human learning and the reality that we must make a conscious effort to ensure that some of this complexity is encoded into the data if we are going to design analytics systems that are educationally useful. We need to elicit specific reactions from our students as expert educators and encode the pedagogical intent for eliciting those reactions along with the reactions themselves.

    .The play’s the thing!

    William Shakespeare
  • D2L Steps Up for EEP

    I’m delighted to announce that D2L has become the Empirical Educator Project’s first official Foundational Sponsor. The major principle behind sponsorship in EEP has always been that we only accept sponsors who have something to contribute in addition to money. The same is true for Foundational Sponsors. D2L is offering more to the academic participants in the Empirical Educator Project than just money and free resources. They have both demonstrated through participation to-date that they are good participants and offered enough value in future participation to earn pride of place. Their behavior is a model for other commercial participants in the educational community to emulate.

    To begin with, D2L VP of Market Research Kenneth Chapman, who I’ve known almost as long as I’ve been in EdTech, has been incredibly supportive of our work to-date. He arranged an integration demo to support Carnegie Mellon University’s OpenSimon announcement at our last summit. He’s sent staff to CMU’s LearnLab summer school as a follow-up. He’s made his people available for our work and has consistently and actively looked for ways to collaborate. Ken’s leadership and enthusiasm convinced me that I wanted to deepen the relationship between D2L and EEP.

    In terms of what D2L is offering as a Foundational Sponsor going forward, as I wrote earlier in this post, there’s the money (and contributions of financial value) and the participation. Let’s address the material part of the contribution first. D2L has committed to sponsoring EEP for the next three years. They have also offered up Brightspace as the EEP online community space for that period of time. This second part is going to be increasingly important as EEP work starts to become year-round and as we prepare to open up at least some of that work—both sharing and participation—to the general public. And D2L has offered up the help of its support team, including superstar Ben Campbell, to help us get the site set up and to teach us how to take maximum advantage of the affordances of the platform.

    By themselves, as generous as those offers are, they wouldn’t be enough to earn D2L a place as an EEP Foundational Sponsor. What sealed the deal was D2L’s interest in increased participation. I can’t share details yet, but at a high level, there are two aspects. First, they are actively interested in offering up Brightspace as a laboratory for experimentation. One natural place where that collaboration may go, as mentioned in D2L’s press release, is integration with Carnegie Mellon University’s OpenSimon software. As I mentioned further up in the post, there has already been a little work done in this regard, and there is plenty more to explore.

    At least as important is D2L’s offer to help bring the work and contributions of the EEP participants to the Brightspace user community. I’m really, really excited about the direction that this part of the conversation is taking and can’t wait to share more details as we nail them down.

    After EEP’s summit this spring, Bart Epstein, the CEO of the Jefferson Education Exchange and a passionate advocate for the kind of efficacy work that EEP is attempting to promote, offered up a note of reasonable skepticism in EdSurge’s coverage of the event and the contributions announced at it:

    When Tesla says that it’s making its battery patents available for free, you can be sure that all of the other car companies have incentives to invest time in reading and understanding those battery patents to read them and see if they can use them,” he said. “But when CMU opens up this software, it’s unclear who is out there that is saying, ‘Oh, there’s something in there that I want.’ We just don’t know how much impact it will have.

    Bart Epstein

    Nowhere is this concern more valid than in higher education. There are probably billions of dollars’ worth of intellectual property contributions that, practically speaking, are languishing on university servers where nobody knows that they exist, what they are good for, or how to use them. If we want to bridge this divide specifically in terms of new knowledge that could actually help students succeed in the real world, then we have to go beyond publishing papers and releasing open source software and OER (as important as those activities are). We have to develop an ecosystem and a culture for the diffusion and uptake of this knowledge.

    I invited D2L to be EEP’s first official Foundational Sponsor because their participation and other contributions will support our work in achieving this ambition.

    Stay tuned for more information as this collaboration continues to evolve.

  • Our First Standard of Proof Webinar, on Beating Summer Melt

    Our First Standard of Proof Webinar, on Beating Summer Melt

    I am just thrilled to post the archive of our inaugural Standard of Proof webinar:

    Summer Melt Standard of Proof webinar
    https://youtu.be/7NoYUa_VHfs

    It is a near-perfect encapsulation of everything that the Empirical Educator Project aspires to promote and foster: front-line educators, academic researchers, and commercial vendors working together to measurably increase student success and to do so in a way that is both persuasive in terms of the rigor of evidence and repeatable by other institutions.

    As a reminder, Standard of Proof highlights academic collaborations by the Empirical Educator Project’s commercial sponsors with academic researchers and academic practitioners. All projects must either contribute new knowledge to the commons or promote the diffusion of established evidence-backed practices.

    In this case, we have a story of a small EdTech start-up—AdmitHub—that has a total of nine randomized controlled trials either completed or in process with academic partner institutions.

    Nine.

    How many of your EdTech vendors have nine randomized controlled trials either done or in the works? How many have one? How many have you even asked this question? (If you want better answers on the first two questions, then try to improve your answer to the last one.)

    They were able to do this, in part, because institutions like Georgia State University had real leaders, like GSU’s Tim Renick, who refused to accept the proposition that disadvantaged students were already beyond help by the time they got to college. Instead, the GSU folks (and their colleagues at other academic institutions conducting research with AdmitHub) set out to study their own practices and identify areas that they were failing students in ways that could be fixed. Using data, with the help of their vendor and an academic researcher—in this case, University of Pittsburgh Professor and summer melt expert Lindsay Page—GSU demonstrated that we can make education better and more equitable. It is the perfect counter-example to the sad situation I described in my post on what I didn’t see at EDUCAUSE.

    This is the future that we all should be reaching for, folks. And it is reachable, as all three of my guests made clear. The video is well worth an hour of your time. Standard of Proof webinar production values will improve as we learn our craft, but the conversation quality just doesn’t get much better than this.

    We’re actively working on the spring webinar line-up and will have some juicy conversations scheduled by the time you return from your holiday break.

    Watch this space.

  • Summer Melt

    Summer Melt: Read this book

    This post is partly a nudge for you to sign up for the inaugural e-Literate Standard of Proof webinar coming up this Monday at 2 PM and partly a post to tell you why I’m so excited to be kicking off the series with this particular story.

    My macro thesis for a while now has been that colleges and universities are in the early stages of a transformation from having a philosophical commitment to student success toward being operationally excellent at supporting and enabling student success. That proposition has been a little abstract for some. If you want to understand what that looks like in the real world, I can’t think of a better example than Georgia State University (GSU) under Tim Renick’s leadership. And this webinar will tell the story of one of his seminal achievements.

    Summer Melt is the classic example of the kind of problem that is traditionally invisible to universities that think of student success as a philosophical commitment rather than a core operational responsibility. It’s the phenomenon where students graduate high school, apply to college, get admitted, say they’re coming, fully intend to come, and then never show up. It disproportionately hits first-generation students, students of color, and economically disadvantaged students. Why? Because getting from admission to the first day of classes is a lot harder than many of us remember. You have to fill out a FAFSA form, which as Renick put it yesterday in his IMS presentation, is basically a tax return. I don’t know about you, but I didn’t fill out my FAFSA. My dad did. Not every seventeen-year-old is lucky enough to be able to hand off that responsibility. Then there are inoculations, forms to be filled out and signed by relatives with whom you may or may not have contact, places to get to, fees to pay, and so on, and so on. In their eponymous book on the topic, researchers Benjamin Castleman and Lindsay Page tell us,

    In some school districts where as many as 40 percent of college-intending students fail to matriculate, it would be more appropriate to refer to this as a “summer flood.”

    Castleman and Page, Summer Melt: Supporting Low-Income Students Through the Transition to College

    Forty. Percent.

    And some of these problems are very solvable—if you know about them. For example, once the GSU folks realized that immunizations were a problem that was preventing students from getting to the first day of classes, they started parking free immunization trucks outside during times when those soon-to-be students would be visiting campus.

    The trick is knowing. So Tim Renick and his team partnered with an AI chatbot company called AdmitHub. It turns out that their CEO, Drew Magliozzi, had read the summer melt book too. And he thought his tool could do something about it. GSU brought in AdmitHub to try it out. But they also brought in Lindsay Page to conduct a randomized controlled trial. It’s one thing to say that you think your intervention improved a problem. It’s quite another to gather credible evidence.

    And that’s what they did. According to GSU’s web page on the topic,

    To help these students, the university identified the common obstacles to enrollment that students face between graduating high school and the start of college, including financial aid applications and documents, immunization records, placement exams and class registration, among others. Georgia State developed an approach that would help at-risk students through these obstacles by instituting a combination of a new student portal to guide students through the steps needed to be ready for the first day of classes and an artificial-intelligence-enhanced chatbot, “Pounce,” to answer thousands of questions from incoming students 24/7 via text messages on their smart devices.

    In 2016, during the first summer of implementation, Pounce delivered more than 200,000 answers to questions asked by incoming freshmen, and the university reduced summer melt by 22 percent. This translated into an additional 324 students sitting in their seats for the first day of classes at Georgia State rather than sitting out the college experience.

    GSU, “Reduction of Summer Melt

    The “Standard of Proof” webinar series is designed to tell stories like this one: Universities working with credible vendor partners to learn and share something new and important about supporting students that is a benefit to the entire sector.

    This is a canonical example of that, folks.

    Sign up for the webinar.

    The webinar: Come for the enlightenment. Bring your own food.