The “Big Picture” category covers larger trends and topics that influence both the problems that technology can help address in education as well as the barriers to implementing high-quality technology-supported education. This includes research-based topics such as learning science and program effectiveness studies, philosophical discussions such as outcomes definitions, and macro-forces such as government policy, markets, and business models.
This post is partly a nudge for you to sign up for the inaugural e-Literate Standard of Proof webinar coming up this Monday at 2 PM and partly a post to tell you why I’m so excited to be kicking off the series with this particular story.
My macro thesis for a while now has been that colleges and universities are in the early stages of a transformation from having a philosophical commitment to student success toward being operationally excellent at supporting and enabling student success. That proposition has been a little abstract for some. If you want to understand what that looks like in the real world, I can’t think of a better example than Georgia State University (GSU) under Tim Renick’s leadership. And this webinar will tell the story of one of his seminal achievements.
Summer Melt is the classic example of the kind of problem that is traditionally invisible to universities that think of student success as a philosophical commitment rather than a core operational responsibility. It’s the phenomenon where students graduate high school, apply to college, get admitted, say they’re coming, fully intend to come, and then never show up. It disproportionately hits first-generation students, students of color, and economically disadvantaged students. Why? Because getting from admission to the first day of classes is a lot harder than many of us remember. You have to fill out a FAFSA form, which as Renick put it yesterday in his IMS presentation, is basically a tax return. I don’t know about you, but I didn’t fill out my FAFSA. My dad did. Not every seventeen-year-old is lucky enough to be able to hand off that responsibility. Then there are inoculations, forms to be filled out and signed by relatives with whom you may or may not have contact, places to get to, fees to pay, and so on, and so on. In their eponymous book on the topic, researchers Benjamin Castleman and Lindsay Page tell us,
In some school districts where as many as 40 percent of college-intending students fail to matriculate, it would be more appropriate to refer to this as a “summer flood.”
Castleman and Page, Summer Melt: Supporting Low-Income Students Through the Transition to College
Forty. Percent.
And some of these problems are very solvable—if you know about them. For example, once the GSU folks realized that immunizations were a problem that was preventing students from getting to the first day of classes, they started parking free immunization trucks outside during times when those soon-to-be students would be visiting campus.
The trick is knowing. So Tim Renick and his team partnered with an AI chatbot company called AdmitHub. It turns out that their CEO, Drew Magliozzi, had read the summer melt book too. And he thought his tool could do something about it. GSU brought in AdmitHub to try it out. But they also brought in Lindsay Page to conduct a randomized controlled trial. It’s one thing to say that you think your intervention improved a problem. It’s quite another to gather credible evidence.
To help these students, the university identified the common obstacles to enrollment that students face between graduating high school and the start of college, including financial aid applications and documents, immunization records, placement exams and class registration, among others. Georgia State developed an approach that would help at-risk students through these obstacles by instituting a combination of a new student portal to guide students through the steps needed to be ready for the first day of classes and an artificial-intelligence-enhanced chatbot, “Pounce,” to answer thousands of questions from incoming students 24/7 via text messages on their smart devices.
In 2016, during the first summer of implementation, Pounce delivered more than 200,000 answers to questions asked by incoming freshmen, and the university reduced summer melt by 22 percent. This translated into an additional 324 students sitting in their seats for the first day of classes at Georgia State rather than sitting out the college experience.
The “Standard of Proof” webinar series is designed to tell stories like this one: Universities working with credible vendor partners to learn and share something new and important about supporting students that is a benefit to the entire sector.
I often get asked what themes I see coming out of EDUCAUSE. The last few years, the theme has been “there is no theme,” perhaps reflecting the fact that ed tech hype has been in remission. With the exception of the OPM product category, whose hype cycle from zero to peak to trough may have been the fastest we’ve seen yet, the market seems tired of the endlessly repeating roller coaster ride of hope and disappointment.
This year was a little different. Overall, the conference was the healthiest that I’ve seen it in a while. Attendance seemed to be up. The balance between IT and learning presentations seemed good. There were more vendors on the floor, with more big booths (but no magicians, jugglers, or sword swallowers, thankfully). Overall, the organization seems to be undergoing a revitalization under John O’Brien.
Within that big picture, there was some color. The major textbook publishers had very little presence, perhaps reflecting their current financial state or perhaps indicating that they’ve figured out the EDUCAUSE crowd isn’t really their target audience, for the most part. On the other hand, ERP/SIS vendors were back with bigger booths, including some names I didn’t recognize (like Unit4, which apparently has been around for a while but which hadn’t heard of before this year).
There also was a proliferation of small vendors with various flavors of solutions aimed at improving student retention and graduation rates. Each one had its own take on the problem. One had an integration dashboard, pulling in data from the LMS, SIS, and other applications to tell advisors when students might be struggling and need encouragement. Another took a similar approach but focused on giving students nudges directly. Yet another focused on engaging students on social media. And so on.
Whenever I saw one of these companies, I tried an experiment. I’d do a quick scan of their booth posters and video screens. Just enough to get a very rough sense of what they claimed to do. In other words, I looked about as carefully as the typical exhibition floor browser looks. Then I would approach the booth and say, “I have a rough sense of what your product does from what I see on your booth.” I have one question for you: How do you know that it works?”
I got a range of different answers.
“Our customers love it.”
Bzzt. Wrong answer.
“We have dashboards that show 67 points where students could be getting bogged down.”
“OK, but how do you know that it works? I mean, have you tested it to see if it actually impacts outcomes?”
“…Well…we have a white paper.”
Pass.
“We have a customer who improved their retention by 60%! But to be truthful, our product was one of a number of initiatives they put into place, so I don’t know how much of that improvement they would attribute to it.”
Close, but no cigar. Points for honesty, though.
“We have conducted multiple randomized controlled trials.”
“Oh, really! What result were you testing against?”
“Oh, right, of course. Year-to-year retention…”
“Thanks, you can stop there for now. May I please have your business card?”
I found one—just one—vendor who appears to be able to deliver the goods on evidence. I wouldn’t have picked up on that from my cursory scan of their booth. They didn’t look any different than any of the others I had spoken to. In fact, if you had asked me which of the products I looked at would have been likely to prove out as effective, I wouldn’t have ranked them near the top of the list. The trick that they advertise most heavily, which is that they repurpose existing advertising mechanisms on popular social media platforms to deliver productive nudges, seems clever but a little disconnected from problems specific to retention. While they mention “behavioral science” on their web site, they don’t provide a lot of detail. That said, if you read their marketing copy carefully, you’ll see that they do provide a rather stripped-down, easy-to-follow explanation of a randomized controlled trial that they conducted.
The name of that company, by the way, is Motimatic. Have you heard of them? Because I haven’t. I’m going to look into them and find out more. Maybe you should too.
Right now, higher education has very poor signals to quickly distinguish between those vendors who can prove that their products are effective and those that can’t. Between those vendors who are contributing to the general state of knowledge about how to improve student success and those that aren’t. Until we can improve the signal-to-noise ratio, we are doomed to ride the roller coaster from hell that is the hype cycle until we pass out from exhaustion.
I’m going to do something about improving the signal-to-noise ratio, and I will be announcing what that something is within the next week.
According to an article in Inside Higher Ed, California just modified its $475 million Student Equity and Achievement Program “to allow the funds to be used for emergency student aid.” Since these changes don’t entail new funding, much of the article was dedicated to hand-wringing about whether diverting existing funds from other priorities “like tutoring, peer-mentoring programs and equity-focused professional development for faculty” is, on balance, a good idea.
On the one hand, there is evidence that giving students emergency financial support is both a needed and an effective intervention:
Colleges in California and across the nation have created their own emergency aid programs. A Senate analysis of the bill notes that Pasadena City College and Grossmont College both fund their programs through external sources like foundations and fundraising.
Amelia Parnell, vice president for research and policy at the National Association of Student Personnel Administrators, said the association found in a survey that most colleges feel they aren’t fully meeting students’ emergency financial needs.
“Because emergencies are typically unexpected, it’s hard to find the right balance that’s needed,” she said, adding that she thinks the spirit of the bill is “consistent with what a lot of campuses have said.”
According to the Senate floor analyses, Chiu cites as support a February 2017 report from the Institute for College Access and Success on college costs for low-income California students. The report found that low-income students at public colleges in California can’t afford college costs with the available grants, their own resources and some working income.
It also found that community colleges sometimes have a greater net price for low-income students than four-year public schools due to the limited amount of grants available for community college students.
Chiu argued that research shows emergency aid can keep students enrolled through unforeseen challenges.
Research does show that emergency aid can keep students enrolled. See, for example, Georgia State University’s Panther Retention Grants.
But on the other hand, the other interventions that the $475 million California program has been funding up until now are important too.
However, the Senate grappled with questions of whether the Student Equity and Achievement Program funds would be best used for this purpose. The analysis asks if the bill would “set a precedent that dilutes student equity funds intended for critical academic support service,” and if expanding state financial aid programs would be more appropriate.
The Senate Appropriations Committee said the bill could redirect funds away from other student support services, which could lead to “potentially significant … cost pressure” to maintain the state’s current level of student support services.
What the article doesn’t mention is whether the legislature funded any significant research, either previously or going forward, that will help guide the colleges regarding which investments are likely to be most effective in meeting their equity goals. Because that’s the question, right? Colleges have options to spend their money to best serve their students. And given the total amount of money in play across the system—nearly half a billion dollars—one would think that a small amount of money invested in research would be a wise allocation of funding.
Maybe it’s in there and just not mentioned in the IHE article. I hope so. Past experience with the California system suggests that (a) the legislature doesn’t think this way and (b) the California Community College System is not set up well to execute programmatic research of this kind even when they are given the funding and prioritization to do so—in part because they are not given the funding and prioritization to do so as often as they should be.
If there is policy uncertainty about a consequential matter that impacts students in a meaningful way, then that risk should be approached with an experimental mindset. If you aren’t mindful about assessing the impact of different choices, then you’re just throwing dice.
It took longer than I had hoped, but you can now see most of the Empirical Educator 2019 summit presentations here. (Unfortunately, the videographers didn’t capture the last couple of presentations on OpenSimon.) I’ll return to these after I finish my series on digital curricular materials design, but in the meantime, the talks are available for your enjoyment.
EdSurge has a good piece up about the U.S. Public filing submitted by the Scholarly Publishing and Academic Resources Coalition (SPARC) with the U.S. Department of Justice opposing the merger between Cengage and McGraw-Hill. In addition to the expected fare about pricing and reduced competition, there is a surprisingly fulsome argument about the dangers of the merger creating an “enormous data empire.”
Given that the topic at hand is an anti-trust challenge with the DoJ, I’m going to raise my conflict of interest statement from its normal place in a footnote to the main text: I do consulting work for McGraw-Hill Education and have consulting and sponsorship relationships with several other vendors in the curricular materials industry. For the same reason, I am recusing myself from providing an analysis of the merits of SPARC’s brief.
Instead, I want to use the data section of their brief as a springboard for a larger conversation. We don’t often get a document that enumerates such a broad list of potential concerns about student data use by educational vendors. SPARC has a specific legal burden that they’re concerned with. I’ll briefly explain it, but then I’m going to set it aside. Again, my goal is not to litigate the merits of the brief on its own terms but rather explore the issues it calls out without being limited by the antitrust arguments that SPARC needs to make in order to achieve their goals.
Let’s break it down.
When is bigger worse?
While I’m sure that PIRG’s concerns about the data are genuine, keep in mind that they have been fighting a long-running battle against textbook prices, and that the primary framing of their brief is about the future price of curricular materials. Their goal is to prevent the merger from going through because they believe it will be bad for future prices. Every other argument that they introduce to the brief, including the data arguments, they are introducing at least in part because they believe it will add to their overall case that the merger will cause, in legal parlance, “irreparable harm.” As such, that has to be the standard for them. It’s not whether we should be worried about misuse of data in general, but about whether this merger of the data pools of two companies makes the situation instantly worse in a way that can’t be undone. That’s pretty high bar. Each of their data arguments needs to be considered in light of that standard.
But if you’re more concerned with the issues of collecting increasingly large pools of student data in general, and if you can consider solutions other than “stop the merger,” then there is a more nuanced conversation to be had. I’m more interested in provoking that conversation.
What can be inferred from the data
One question that we’re going to keep coming back to throughout the post is just how much can be gleaned from the data that the publishers have. This is a tough question to answer for a number of reasons. First, we don’t know exactly everything that all the publishers are gathering today. SPARC’s doesn’t provide us with much help here; they don’t appear to have any inside information, or even to have spent much time gathering publicly available information on this particular topic. I have a pretty good idea of what publishers are collecting in most of their products today, but I certainly don’t have a comprehensive knowledge. And it’s a moving target. New features are being added all the time. I can speak a lot more confidently about what is being gathered today than on what may be gathered a year from now. The further out in time you go, the less sure you can be. Finally, while publishers—like the rest of us—have thus far proven to be relatively bad at extrapolating useful holistic knowledge about students from the data that publishers tend to have, that may not always prove to be the case. So with those generalities in mind, let’s look at SPARC’s first claim:
Like most modern digital resources, digital courseware can collect vast amounts of data without students even knowing it: where they log in, how fast they read, what time they study, what questions they get right, what sections they highlight, or how attentive they are. This information could be used to infer more sensitive information, like who their study partners or friends are, what their favorite coffee shop is, what time of day they commute from home to school, or what their likely route is.
How much of that “more sensitive information” that SPARC claims can be inferred really logical to fear right now? Most of the scary stuff they speculate about here is location-related. Unless the application page specifically asks the student’s permission to use geolocation and the student grants it—I’m sure you’ve had web pages ask your permission to know your location before—then the best it can do is know the student’s IP address, which is a pretty crude location method. None of the place-based information is really accessible via any data that is collected through any courseware that I’m aware of today. The only exception I know of is attendance-taking software. How much of an additional privacy risk it is to know the attendance habits of students who are already known to have registered for a class in virtue of the fact that they are taking and using the curricular materials associated with the class is an open question.
The other risk SPARC references specifically is knowledge of social connections. There are products that do facilitate the finding of study partners. Actually, the LMS market, which is roughly as concentrated as the curricular materials market, may have much more exposure to this particular concern.
While I certainly wouldn’t want these data to be leaked by the stewards of student learning information, I suspect there is much better quality data of this sort that is more easily obtainable from other sources. Even in the worst case, if they got misappropriated and merged with consumer data sets, the incremental value of this information relative to what someone with ill intent could learn from the average person’s social media activity strikes me as pretty limited.
Of course, the information value is a separate question from the responsibility of care. Students are responsible for the information that they post on their social media accounts. Educators and educational institutions have a responsibility of care for data in products that they require students to use. That said, we should think about both the responsibility of care and the sensitivity of particular data. Generally speaking, I don’t see the kind of location and and personal association data that publisher applications are likely to have as particularly sensitive.
Anyway, continuing with SPARC’s brief:
“We now have real time data, about the content, usage, assessment data, and how different people understand different concepts,” said Cengage CEO Michael E. Hansen in an interview with Publishers Weekly.135 McGraw-Hill claims that its SmartBook program collects 12 billion data points on students. Pearson now allows students to access its Revel digital learning environment through Amazon’s Alexa devices—which have been criticized for gathering data by “listening in” on consumers.
Once gathered, these millions of data points can be fed into proprietary algorithms that can classify a student’s learning style, assess whether they grasp core concepts, decide whether a student qualifies for extra help, or identify if a student is at risk of dropping out. Linked with other datasets, this information might be used to predict who is most likely to graduate, what their future earnings might be, how a student identifies their race or sexual orientation, who might be at risk of self-harm or substance abuse, or what their political or religious affiliation might be. While these types of processes can be used for positive ends, our society has learned that something as seemingly innocent as an online personality test can evolve into something as far-reaching as the Cambridge Analytica scandal. The possibilities for how educational data could be used and misused are endless.
I realize that this is a rhetorical flourish in a document designed to persuade, but no, the possibilities really aren’t endless. If you can’t train a robot tutor in the sky by having it watch you solve more geometry problems, then you can’t bring Skynet to sentience that way either. I don’t want to minimize real dangers. Quite the opposite. I want to make sure we aren’t distracted by imaginary dangers so that we can focus on the real ones.
I’m particularly concerned by the Cambridge Analytica sentence. “Something as seemingly innocent as an online personality test can evolve into something as far-reaching…”. The implication seems to be that Cambridge Analytica inferred enormous amounts of information from an online personality test. But that’s not what happened. The real scandal was that Cambridge Analytica used the personality test to get users to grant them permission to enormous amounts of other data in their profile. The kind of deeply personal data that people put in Facebook but don’t tend to put in their online geometry courseware. I don’t see how that applies here.
Of course, the data that these companies collect in the future may change, as may our ability to infer more sensitive insights from it. Writ large, we don’t have to make the kind of cut-and-dry, snapshot-in-time decision that a legal brief necessarily advocates. Rather than making a binary choice between either blithely assuming that all current and future uses of student educational data in corporate hands will be fine or assuming the dystopian opposite and denying students access to technology that even SPARC acknowledges could benefit them, the sector should be making a sustained and coordinated investment in student data ethics research. As new potential applications come online and new kinds of data are gathered, we should be pro-actively researching the implications rather than waiting until a disaster happens and hoping we can up the mess afterward.
Data permission creep
SPARC next goes on to argue that since (a) students are a captive audience and essentially have no choice but to surrender their rights if they want to get their grades, (b) professors, who would be the ones in a position to protect students’ rights, don’t have a good track record of protecting them from textbook prices, and (c) nobody has a good track record of reading EULAs before clicking away their rights, there is a good chance that, even if the data rights students give agree to give away are reasonable today, there is a high likelihood that they will creep into unreasonableness in the future:
Students are not only a “captive market” in terms of the cost of textbooks, they are a captive market in terms of their data. The same anticompetitive behavior that arose in the relevant market for course materials is bound to repeat itself in the relevant market for student data.
As the market shifts toward inclusive access fees and all-access subscriptions, students increasingly will be required to use digital course materials as a condition of enrolling in a course. Even if a student is not automatically subscribed, they may be enrolled in a course using digital homework, where a portion of a student’s grade depends on purchasing an access code, accepting the terms of use, and potentially surrendering data in the process of completing assignments. This is a new dimension of the principal-agent problem. In the same way that it is a foregone conclusion that students will need to purchase assigned materials regardless of the price, it is also a foregone conclusion that they will need to accept the terms of use.
The graph of textbook prices since 1980 in Section 1.1 illustrates what can happen when publishers engage in coordinated pricing practices in a market where consumers have little power, as we discussed in Section 4.1. The same problem could repeat itself in terms of the ever expanding permissions granted under terms of use. Just as professors are sometimes unaware when the price of a textbook goes up, they may not be aware when the terms of use change in a way that may be unacceptable to their students.
Therefore, there is potential for publishers to inflate the permissions they require students to grant in exchange for using a digital textbooks in the same way that they have inflated prices through coordinated behavior. Students will not only be paying in dollars and cents, but also in terms of their data.
I find the permissions creep argument to be compelling for several reasons. First, the question of whether people should have a right to control how their data are used is separable from the question of known harm that abuse of those data could cause. Students should have right to say how their data can be used and shared, regardless of whether that use is deemed harmful by some third party.
Second, there is an argument that SPARC missed here related to human subjects research. Currently, universities are required by law to get any experimentation with human subjects, including educational technology experiments, approved by an IRB. This includes, but is not limited to, a review of informed consent practices. Companies have no such IRB review requirement under current law. Companies with more data, more platforms, and bigger research departments can conduct more unsupervised research on students. For what it’s worth, my experience is that companies that do conduct research often try to do the right thing. But that should be small comfort, for a number of reasons.
First, there is no generally agreed upon definition of what “the right thing” is, and it turns out to be very complicated. When is an activity research “on” students, and when is it “on” the software? If, for example, you move a button to test whether doing so makes a feature easier to find, but awareness of that feature turns out to make a difference in student performance, then would the company need IRB approval? If the answer “yes,” and “IRB approval” for companies looks anything remotely like what it does inside universities today, then forget about getting updated software of any significance any time soon. But if the answer is “no,” then where is the line, and who decides? There is basically no shared definition of ethical research for ed tech companies and no way to evaluate company practices. This is not only bad for the universities and students but also for the companies. How can they do the right thing if there is no generally accepted definition of what the right thing is?
Second, if IRB approval specifically means getting the approval of one or more university-run IRBs, and particularly if it means getting the approval of the IRB of every university for every student whose data will be examined, universities have not yet made that remotely possible to accomplish. Nor could they handle the volume. I believe that we do need companies to be conducting properly designed research into improving educational outcomes, as long as there is appropriate review of the ethical design of their studies. Right now, there is no way of guaranteeing both of these things. That is not the fault of the companies; it’s a flaw in the system.
Fixing the student privacy permission problem would be hard to do in a holistic way. Some further legislation could potentially help, but I’m not at all confident that we know what that legislation should require at this point. I’ve written before about how federated learning analytics technical standards like IMS Caliper could theoretically enable a technical solution by enabling students to grant or deny permission to different systems that want access to their data, similarly to the way in which we grant or deny access to apps that want access to data on our phones. But that would be a long and difficult road. This is a tough nut to crack.
The research problem is also tough, but not quite as tough as the privacy permission problem. I’ve been speaking to some of my clients about it in an advisory capacity and working on it through the Empirical Educator Project. It is primarily a matter of political will at this point, and the pressure to solve this problem is rising on all sides.
More data means more privacy risk
For our purposes, I won’t quote the entirety of SPARC’s argument on this topic, but here’s the nub of it:
It is common sense that the more data a company controls, the greater the risk of a breach. Recent experience demonstrates that no company can claim to be immune to the risk of data breaches, even those who can afford the most updated security measures. The size or wealth of a company has proven no obstacle to potential hackers, and in fact larger companies may become more tempting targets. Allowing more student data to become concentrated under a single company’s control increases the risk of a large scale privacy violation.
As a case in point, Pearson recently made the news for a major data breach. According to reports, the breach affected hundreds of thousands of U.S. students across more than 13,000 school and university accounts. Pearson reports that no social security numbers or financial information was compromised, but this is not the only kind of data that can cause damage. Compromising data on educational performance and personal characteristics can potentially affect students for the rest of their lives if it finds its way to employers, credit agencies, or data brokers.
While state and federal laws provide some measure of privacy protection for student records, including limiting the disclosure of personally identifiable information, they do not go far enough to prevent the increased risk of commercial exploitation of student data or protect it from potential breaches.
While we should be very concerned about student data privacy, I don’t think the number of data points an education company has about a student is a good measure of the threat level. Again, a merged Cengage/McGraw-Hill would not have the same kind of data that Facebook would. We have to think very specifically about these data because they are quite different from data on the consumer web. The number of hints a student asked for in a psychology exercise or the number of algebra problems a student solved do not strike me as data that are particularly prone to abuse. These sorts of information bits comprise the bulk of the data that such companies have in their databases today. There may very well be extremely serious data privacy issues lurking here, but they will not be well measured by the volume of data collected (in contrast with, say, Google).
The point about the gaps in the laws is a much more serious one. Everybody has known for years, for example, that FERPA is badly inadequate. It is only getting worse as it ages. The Fordham paper cited by SPARC has some good suggestions. Now, if only we had a functioning Congress….
Algorithms
Again, I’ll excerpt the SPARC filing for our purposes:
Algorithms are embedded in some digital courseware as well, including the “adaptive learning” products of the merging companies and some of their competitors. These algorithms can be as simple as grading a quiz, or as complex as changing content based its assessment of a student’s personal learning style….
While algorithms can produce positive outcomes for some students, they also carry extreme risks, as it has become increasingly clear that algorithms are not infallible. A recent program held at the Berkman Klein Center for Internet and Society at Harvard University concluded categorically that “it is impossible to create unbiased AI systems at large scale to fit all people.” Furthermore, proprietary algorithms are frequently black boxes, where it is impossible for consumers to learn what data is being interpreted and how the calculations are made—making it difficult to determine how well it is working, and whether it might have made mistakes that could end in substantial legal or reputational consequences.
Let’s disambiguate a little here. There are two senses in which an algorithm could be considered a “black box.” Colloquially, educators might refer to an adaptive learning or learning analytics algorithm that way if they, the educators using it, have no way of understanding how the product is making the recommendations. If an algorithm is proprietary, for example, the vendor might know why the algorithm reaches a certain result, but the educator—and student—do not.
Within the machine learning community, “black box” means something more specific. It means that the results are not explainable by any humans, including the ones who wrote the algorithm. In certain domains, there is a known trade-off between predictive accuracy and the the human interpretability of how the algorithm arrived at the prediction.
Both kinds of black boxes are very serious problems for education. In my opinion, there should be no tolerance for predictive or analytic algorithms in educational software unless they are published, peer reviewed, and preferably have replicated results by third parties. Educators and qualified researchers should know how these products work, and I do not believe that this an area where the potential benefits of commercial innovation outweigh the potential harm. Companies should not compete on secret and potentially incorrect insights about how students learn and succeed. That knowledge should be considered a public good. Education companies that truly believe in their mission statements can find other grounds for competitive advantage. This is another area that EEP is doing some early work on, though I don’t have anything to announce on it just yet.
The second kind of black box—algorithms that are published and proven to work but are not explainable by humans—should be called out as such and limited to very specific kinds of low-stakes use like recommending better supplemental content from openly available resources on the internet. We should develop a set of standards for identifying applications in which we’re confident that not understanding how the algorithm arrives at its recommendation does not introduce a substantial ethical risk and does produce substantial educational benefit. If the affirmative case can’t be made, then the algorithm shouldn’t be used.
Data monopolies
I’m going to be a little careful with this one because, again, I am recusing myself from commenting on the merits of the brief, and this particular data topic is hardest to address while skirting the question before the DoJ. But I do want to make some light comments on the broader question of when combining different educational data sets is most potent and therefore most vulnerable to abuse.
From SPARC:
One lesson learned from the rise of technology giants like Facebook is that preventing platform monopoly from forming is far simpler than breaking one up. Given the vast quantity of data that the combined firm would be in a position to capture and monetize, there is a real potential for it to become the next platform monopoly, which would be catastrophic for student privacy, competition, and choice.
For decades, the college course material market has been split between three giants. There is a large difference between a market split three ways and a market split two ways. As these companies aggressively push toward digital offerings and data analytics services, a divided market will limit the size and comprehensiveness of the datasets they are able to amass, and therefore the risk they pose to students and the market. So long as publishers are competing to sell the best products to institutions, and there is significantly less risk of too much student data ending up in one company’s hands.
I won’t characterize the danger of combining publisher data sets beyond what I’ve already covered in this post. What I want to say here is that the bigger opportunity for potential insights, and therefore the bigger area of concern for potential abuse, may be when combining data sets from different kinds of learning platforms. I haven’t yet seen evidence that combining data across courseware subjects yields big gains in understanding regarding individual students. But when you combine data from courseware, the LMS, clickers, the SIS, and the CRM? That combination of data has great potential for both benefit and harm to students because it provides a much richer contextual picture of the student.
Irreparable harm
While nothing in this post is intended to comment directly on the matter before the DoJ, the phrase that frames the anti-trust argument—”irreparable harm”—is one that we should think about in the larger context. I believe we have an affirmative obligation to students to develop and employ data-enabled technologies that can help them succeed, but I also believe we have an affirmative obligation to proceed in a way that prioritizes the avoidance of doing damage that can’t be undone. “First, do no harm.” We should be putting much more effort into thinking through ethics, designing policies, and fostering market incentives now. I don’t see it happening yet, and it’s not even entirely clear to me where such efforts would live.
Today I am sharing the first video out of the Empirical Educator Project (EEP) 2019 summit, and with it, a central concerns of the project. Much of the basic machinery our learning processes work so naturally and automatically so much of the time that they are invisible to us. So pervasively invisible, in fact, that most of us are barely aware that it even exists. And that’s a problem. If you believe that the job of education is to work within what psychologist Lev Vygotski called the “zone of proximal development”—the kind of learning challenge that would be too hard for a student to learn on her own but not so hard that she can’t learn it at all—then we have to have a very finely tuned understanding of that learning machinery, to the point where we can accurately find each student’s zone of proximal development with a high level of consistency.
We fail to do this all the time. Some students are bored while others struggle. The more heterogeneous the student population is, the bigger a problem this is. As higher education as a sector becomes more committed to serving post-traditional students, first-generation students, and students with 40-year educational relationships to the school rather than 4-year relationships, then this need to be able to see and understand these invisible learning processes becomes more acute. For this reason among others, fostering academic literacy around the mental machinery of learning—making the invisible visible—is one of the central goals of EEP. I therefore wanted to start the 2019 EEP summit by highlighting this challenge. So I invited three Carnegie Mellon University (CMU) professors with complementary areas of expertise to participate in a panel that could highlight several dimensions of the problem.
This wasn’t the first time I had interviewed these three particular academics. I had been fortunate enough to be invited to a CMU press fellowship three years earlier. I brought my video camera along and happened to be able to get some air time with these very three people, two of whom I had never met before. The interviews turned out to be formative for me, particularly with regard to my thinking about EEP. I’m going to write a little about the complimentary insights that these three academics gave to me and then share both the interview video from the summit and the original interview videos from two years ago.
Expert blind spots
As we get old and forgetful, we like to joke that our minds have to make room for the new information by clearing out old information. It turns out that there’s truth behind this joke in multiple ways. First, we have different kinds of memory. If I asked you to list the steps required to tie your shoe, those steps would probably not come tripping off your tongue. Does that mean that you don’t know how to tie your shoe? No, it doesn’t. It means that you’ve moved that knowledge to a more efficient memory space in your brain. One that’s quick and efficient enough that you can easily bend down and tie your shoes while performing other, more demanding cognitive tasks. But that knowledge is not accessible to your conscious mind. It is “tacit” knowledge. Your brain is very efficient at shunting information that it needs to access but does not need to consciously examine into a different compartment than the one it was in when you were learning a skill.
There was a time when you could list the steps in tying your shoe, because that was how you first learned those steps. Your brain put that information into a box once it no longer needed conscious access to it. Chances are good that you don’t remember that time well and that you don’t remember the experience of those steps fading from your conscious memory. I tried to recreate this experience recently for myself. I am learning to swim. In the first weeks, I was thinking about about very basic aspects of moving my arms and, separately, moving my legs. That period was about nine months ago. I decided to try a little experiment with memory encoding in the process. Every two weeks, I would try to remember the steps that I learned in my first lesson. I didn’t try to memorize those steps. That would be triggering a different memory process and would invalidate the experiment. I just tried to reconstruct the steps in my mind. Meanwhile, I spent most of my time at the pool learning to be a better swimmer.
As the weeks went on, I found myself thinking less about what my arms and legs were doing separately and more about what my whole body was doing. I also found it harder and harder to remember what the original steps were that I learned in my first lesson. Nine months in, I barely remember anything about how I first thought about what I was doing. If I had to teach somebody to swim from scratch, I couldn’t just reproduce the lesson that was taught to me. I’d have to make something up. Nor could I reproduce the learning steps I took—many of which I made on my own, without my instructor—to get from my beginner’s understanding to the level of expertise I have achieved as of today. I might be able to draw on some of my knowledge and experience, but I would have to invent more of my teaching moves than most teachers like to admit, through trial and error, by working with students.
So our brains do, in fact, make room for new information by boxing up old information and putting into storage. In addition to the memory changes, we also process information differently as our domain knowledge gets more sophisticated. When you’re learning math, or cooking, or yoga, or any other discipline with integrated skills that build on each other, at first, you’re learning each skill separately. Over time, your mind integrates steps and makes general rules. As novice cooks become expert cooks, their way of thinking about cooking looks less like meticulously following one out of hundreds of completely separate recipes and more like following some generalized principles that they’ve drawn from their experience of making so many recipes. They stop thinking algorithmically and start thinking heuristically.
We don’t generally notice these changes in our cognition as we move from novices to experts in a topic. They’re not directly observable and not usually consciously experienced. They just happen. This is a problem for teaching because professors, as experts, have undergone all of these changes in their learning processes. They no longer think they way their students do. They don’t think about cooking as following individual recipes. Further, because their evolution as thinkers was largely silent, and because most professors have no professional development in these processes, it’s not always obvious to them the extent to which their brains process information in fundamentally different ways than those of their students. Ironically, it is their very expertise that causes them to struggle sometimes to understand how their students think about their subjects or how to work with them in that zone of proximal development. CMU Professor Ken Koedinger, Director of LearnLab at the Pittsburg Science of Learning Center, is an expert in this conundrum.
Expert teaching blind spots
There’s a related phenomenon that I’ll call an expert teaching blind spot, even though I don’t think that’s an official term of art. Just as it is possible to not consciously know what you know in any domain of knowledge, it’s possible to have tacit knowledge specifically in teaching. In addition to the reasons above, I’ll add another one: Interpersonal skills, including teaching skills, are somewhere in the middle of learning spectrum between things that we are hardwired to learn without anyone specifically teaching us (like spoken language as young children), and something that is an intellectual creation which must be consciously learned (like political science). Many educators have what we colloquially refer to as teaching “instincts,” and that word is not far from the truth. We have tacit interpersonal knowledge, sometimes including tacit knowledge about learning processes of our students. We know some things about how to teach in a very real sense, but that knowledge is not fully consciously accessible to us.
As a result, it can be very difficult to talk to even highly skilled teachers about what they do, because in many cases they’ve never even tried to put what they do into language. They just do what seems right and obvious to them. And if they do verbalize what they’re doing, they usually aren’t using terms of art because they usually haven’t been taught any. Their insights seem personal because nobody has talked to them that beyond the personal and phenomenological there could be a sharable, learnable, teachable body of knowledge that their instincts are tapping into. CMU’s Marsha Lovett, Director of Eberly Center for Teaching Excellence & Educational Innovation is an expert in this problem domain.
If we don’t have a coherent answer, then we make one up
If you put all of this together, it adds up to a very significant challenge to serious educators. They don’t have easy ways of knowing how they think differently than their students or easy access to their own cognitive journeys that got them from novice learners to expert learners. And yet, most of us have vivid memories of our formative experiences as students. On top of that, teachers teach, and students learn. It happens all the time. Humans are such incredible learning machines, and the machinery is so well hidden from us, that many people tend to assume that there really isn’t much to it (when nothing could be further from the truth). Most professors are good at academic learning. That’s how they ended up as professors.
And they usually had at least one experience that really inspired them to learn about their chosen field. That association is often all it takes for educators to attribute causality. “Well, I had an amazing experience in Professor Smith’s class, and Professor Smith did X, so X must be a great way to teach.” Given that most professors diligently worked through five to seven years of graduate school without being exposed to the tiniest hint of any of the above and then were expected to somehow magically know how to teach well, what tends to happen is that professors make up their own stories about what effective teaching is based on their own personal experiences—which is the only data they have, really—and they go on that. And they don’t change their minds about it very much or very easily. CMU anthropologist and Simon Research Faculty Lauren Herckis has conducted some fascinating research in this area.
We have a literacy problem
If you put all of this together, it’s clear that we’re not going to make substantial progress on improving education until educators are taught to see that which is currently invisible. We have to develop a common cultural understanding that learning involves a complex set of cognitive processes, that being an expert in a knowledge domain is not sufficient to be a good teacher of novices, that good teaching instincts are often based on tacit knowledge which we can make explicit and therefore more sharable and useful. Only by doing this together, as a sector, can we make substantial progress on improving student success. One of the main goals of Empirical Educator Project is to begin fostering the cultural infrastructure that we need in order to do that.
Here are the three original video interviews I conducted of Marsha, Ken, and Lauren two years ago:
e-Literate TV CMU Interviews
I got lucky with those interviews. The coherence in the interviews is a product of the coherent body of work at CMU’s Simon Initiative as represented by the three people who happened to be available to interview rather than through some master plan of mine.
At the summit, I chose to frame up both the discussion and the event more consciously. In addition to their work, I asked the three to reflect on their personal journeys as educators to embrace views about teaching and learning that may have seemed surprising or even counter-intuitive to them:
The journeys that these experts describe are emblematic of the bigger picture that EEP is all about. And not just in classroom work specifically, but in every aspect of serving students.
I have said before that academia needs to move from a philosophical commitment to student success toward operational excellence at supporting student success. The implied gap is knowhow. It will show up differently in the classroom than it will in, say, advising, but the pattern is going to be the same, and I think academics will be most comfortable thinking about it as starting with a literacy problem. There is some discipline, either new or existing, that they must learn to some degree of competence in order to serve their students well. They might not have to be expert in it—they don’t have to have PhDs in cognitive psychology, for example—but they do need to be literate in it.
A few weeks back, I had the pleasure of attending the IMS Learning Impact Leadership Institute (LILI). For those of you who aren’t familiar with it, IMS is the major learning application technical interoperability organization for higher education and K12 (and is making some forays into the corporate training and development world as well). They’re behind specifications like LIS, which lets your registrar software automagically populate your LMS course shell with students, and LTI, which lets you plug in many different learning applications. (I’ll have a lot more to say about LTI later in this post.)
While you may not pay much attention to them if you aren’t a technical person, they have been and will continue to be vital to creating the kind of infrastructure necessary to support more and better teaching and learning affordances in our educational technology. As I’ll describe in this post, I think the nature of that role is likely to evolve somewhat as the interoperability needs of the sector are beginning to evolve.
The IMS is very healthy
I’m happy to report that the IMS appears to be thriving by any obvious measure. The conference was well attended. It attracted a remarkably diverse group of people for an event hosted by an organization that could easily be perceived as techie-only. Furthermore, the attendees seemed very engaged and the discussions were lively.
On more objective measures, the organization’s annual report bears out this impression of strong engagement. They have strong international representation across a range of organization types.
From the IMS Global 2018 Annual Report
Whether your measure is membership, product certifications, or financial health, the IMS is setting records.
From the IMS Global 2018 Annual Report
This state of affairs is even more remarkable given that, 13 years ago, there was some question as to whether the IMS was financially sustainable.
From the IMS Global 2018 Annual Report
If you look carefully at this graph, you’ll see three distinct periods of improvement: 2005-2008, 2009-2013, and 2013-2018. Based on what I know about the state of the organization at the time, first period can most plausibly be attributed to immediate changes implemented by Rob Abel, who took over the reins of the organization in February of 2006 and likely saved it from extinction. Likewise, the magnitude of growth in the second period is consistent with that of a healthy membership organization that has been put back on track.
But that third period is different. That’s not normal growth. That’s hockey stick growth.
I am not a San Franciscan. By and large, I do not believe in heroic entrepreneur geniuses who change the world through sheer force of will. Whenever I see that kind of an upward trend, I look for a systemic change that enabled a leader or organization—through insight, luck, or both—to catch an updraft.
There is no doubt in my mind that the IMS has capitalized on some major updrafts over the last decade. That is an observation, not a criticism. That said, the winds are changing, in part because the IMS has helped move the sector through an important period of evolution and is now helping to usher in the next one. That will raise some new challenges that the IMS is certainly healthy enough to take on but will likely require them to develop a few new tricks.
The world of 2005
In the first year of the chart above, when the IMS was in danger of dying, there was very little in the way ed tech to interoperate. There were LMSs and registrar systems (a.k.a. SISs). Those were the two main systems that had to talk to each other. And they did, after a fashion. There was an IMS standard at the time, but it wasn’t a very good one. The result was that, even with the standard, there was a person in each college or university IT department whose job it was to manage the integration process, keep it running, fix it when it broke, and so on. This was not an occasional tweak, but a continual effort that ran from the first day of class registration through the last day of add/drop. If you picture an old-timey railroad engineer shoveling coal into the engine to keep it running and checking the pressure gauge every ten minutes to make sure it didn’t blow up, you wouldn’t be too far off. As for reporting final grades from the LMS’s electronic grade book automatically to the SIS’s electronic final grade record, well, forget it.
If you ignore some of the older content-oriented specifications, like QTI for test questions and Common Cartridge for importing static course content, then that was pretty much it in terms of application-to-application interoperability. Once you were inside the LMS, it was basically a bare-bones box with not much you could add. Today, the IMS lists 276 officially certified products that one can plug into any LMS (or other LTI-compliant consumer), from Academic ASAP to Xinics Commons. I am certain that is a substantial undercount of the number of LTI-compatible applications, since not all compatible product makers get officially certified. In 2005, there were zero, because LTI didn’t exist. There were LMS-specific extensions. Blackboard, for example, had Building Blocks. But with a few exceptions, most weren’t very elaborate or interesting.
My personal experience at the time was working at SUNY Systems Administration and running a search committee for an LMS that could be centrally hosted—preferably on a single instance—and potentially support all 64 campuses. For those who aren’t familiar with it, SUNY is a highly diverse system, with everything from rural (and urban) community colleges to R1s to everything in between, with some specialty schools thrown into the mix like the Fashion Institute of Technology, a medical school or two, an ophthalmology school, and so on. Both the pedagogical needs and the on-campus support capabilities across the system were (and presumably still are) incredibly diverse. There simply was not any existing LMS at the time, with or without proprietary extensions, that could meet such a diverse set of needs across the system. We saw no signs that this state of affairs was changing at pace that was visible to the naked eye, and relatively few signs that it was even widely recognized as a problem.
To be honest, I came to the realization of the need fairly slowly myself, one conversation at a time. A couple of art history professors dragged me excitedly to Columbia University to see an open source image annotation tool, only to be disappointed when they discovered that the tool was developed to teach clinical histology, which uses image annotation to teach in an entirely different way than is typically employed in art history classes. An astronomy professor at a community college on the far tip of Long Island, where there was relatively little light pollution, wanted to give every astronomy student in SUNY remote access to his telescope if only we could figure out how to get it to talk to the LMS. Anyone who has either taught a been an instructional designer for a few wildly different subjects has a leg up on this insight (and I had done both), but even so, there are levels of understanding. The art history/histology thing definitely took me by surprise.
A colleague and I, in an effort to raise awareness about the problem, wrote an article about the need for “tinkerable” learning environments in eLearn Magazine. But there were very few models at the time, even in the consumer world. The first iPhone wasn’t released until 2007. The first practically usable iPhone wasn’t released until 2008. (And we now know that even Steve Jobs was secretly skeptical that apps on a phone were a good idea.) It is a sign of just how impoverished our world of examples was in January of 2006 that the best we could think of to show what a world of learning apps could be like was Google Maps:
There are several different ways that software can be designed for extensibility. One of the most common is for developers to provide a set of application programming interfaces, or APIs, which other developers can use to hook into their own software. For example, Blackboard provides a set of APIs for building extensions that they call “Building Blocks.” The company lists about 70 such blocks that have been developed for Blackboard 6 over the several years that the product version has been in existence. That sounds like a lot, doesn’t it? On the other hand, in the first five months after Google made the APIs available for Google Maps, at least ten times that many extensions have been created for the new tool. Google doesn’t formally track the number of extensions that people create using their APIs, but Mike Pegg, author of the Google Maps Mania weblog, estimates that 800-900 English-language extensions, or “mash-ups,” with a “usable, polished Google Maps implementation” have been developed during that time—with a growth rate continuing at about 1,000 new applications being developed every six months. According to Pegg, “There are about five sites out there that facilitate users to create a map by taking out an account. These sites include wayfaring.com, communitywalk.com, mapbuilder.net—each of these sites probably has hundreds of maps for which just one key has been registered at Google.” (Google requires people who are extending their application to register for free software “keys.” Perhaps for this reason, Chris DiBona, Google’s own Open Source Program Manager, has heard estimates that are much higher. “I’ve seen speculation that there are hundreds or thousands,” says DiBona, noting that estimates can vary widely depending on how you count.
Nevertheless, even the most conservative estimate of Google Maps mash-ups is higher than the total number of extensions that exist for any mainstream LMS by an order of magnitude.
There seemed little hope for this kind of growth any time in the foreseeable future. By early 2007, having failed to convince SUNY to use its institutional weight to push interoperability forward, I had a new job working at Oracle and was representing them on a specification development committee at the IMS. It was hard, which I didn’t mind, but it was also depressing. There was little incentive for the small number of LMS and SIS vendors who dominated specification development at that time to do anything ambitious. To the contrary, the market was so anemic that the dominant vendors had every reason to maintain their dominance by resisting interoperability. Every step forward represented an internal battle within those companies between the obvious benefit of a competitive moat and the less obvious enlightened self-interest of doing something good for customers. This is simply not the kind of environment in which interoperability standards grow and thrive.
And yet, despite the fact that it certainly didn’t feel like it, change was in the air.
Glaciers are slow, but they reshape the planet
For starters, there was the LMS, which was both a change agent in of itself and an indicator of deeper changes in the institutions that were adopting them. EDUCAUSE data shows that the US LMS market became saturated some time roughly around 2003. At that time, Blackboard and WebCT had the major leads as #1 and #2, respectively. The dynamic for the next 10 years was a seesaw, with new competitors rising and Blackboard buying and killing them off as fast as it could. Take a look at the period between 2003 and 2013 in Phil’s squid graph: ((By the way, if you haven’t subscribed to Phil’s new blog yet, then you really, really should. Like, right now. I’ll wait.))
It was absolutely vicious.
None of this would materially affect the standards making process inside the IMS until, first, Blackboard’s practice of continually buying up market share eventually failed (thus allowing an actual market with actual market pressures to form) and, second, until the management team that came up with this decidedly anti-competitive strategy…er…chose to spend more time with their respective families. (I’ll have more to say about Heckle and Jeckle and their lasting impact on market perceptions in a future post.)
But the important dynamic during this period is that customers kept trying to leave Blackboard (even if they found themselves being reacquired shortly thereafter) and other companies kept trying to provide better alternatives. So even though we didn’t have a functioning, competitive market that could incentivize interoperability, and even though it certainly didn’t feel like we had one, some of the preconditions for one were being established.
Meanwhile online education growth was being driven by no fewer than three different vectors. First, for-profit providers were hitting their stride. By 2005, the University of Phoenix alone was at over 400,000 enrollments. Second, public access-oriented institutions, many of which had been seeded a decade earlier with grants from the Sloane Foundation, were starting to show impressive growth as well. A couple were getting particular attention. UMUC, for example, may not have had over 400,000 online enrollments in 2005, but they had well over 40,000, which is enough to get the attention of anyone in charge of an access-oriented public university’s budget. More quietly, many smaller schools were having online success that were proportional to their sizes and missions. For example, when I arrived at SUNY in 2005, they had a handful of community colleges that had self-sustaining online degree programs that supported both the missions and the budget of the campuses. Many more were offering individual courses and partial degrees in order to increase access for students. (Most of New York is rural, after all.)
The third driver of online education, which is more tightly intertwined with the first two than most people realize, is that Online Program Management companies (OPMs) were taking off. The early pioneers, like Deltak (now Wiley Education Services), Embanet, Compass Education (now both subsumed into Pearson), and Orbis (recently acquired by Grand Canyon University) had proved out the model. The second wave was coming. Academic Partnerships and 2Tor (now 2U) were both founded in 2008. Altius Education came in 2009. In 2010, Learning House (now also owned by Wiley) was founded.
Counting online enrollments is a notoriously slippery business, but this chart from the Babson survey is highly suggestive and accurate enough for our purpose:
If you’re a campus leader and thirty percent of your students are taking at least one online class, that becomes hard for you to ignore. Uptime becomes far more important. Quality of user experience becomes far more important. Educational affordances become far more important. Obviously, thirty percent is an average, and one that is highly unevenly distributed across segments. But it’s significant enough to be market-changing.
And the market did change. In a number of ways, the biggest one being that it became an actual, functioning market (or at least as close to one as we’ve gotten in this space).
When glaciers recede
Let’s revisit that second growth period in the IMS graph—2008 to 2013—and talk about what was happening in the world during that period. For starters, online continued its rocket ride. The for-profits peaked in 2010 at roughly 2 million enrollments (before beginning their spectacular downward spiral shortly thereafter). Not-for-profits (and odd mostly-not hybrids) ramped up the competition. ASU launched its first online 4-year degree in 2006. SNHU started a new online unit in 2009. WGU expanded into Indiana in 2010, which was the same year that Embanet merged with Compass Knowledge and was promptly bought by Pearson. (Wiley acquired Deltak two years later.)
Once again, the more online students you have, the less you are able to tolerate downtime, a poor user interface that drives down productivity, or generic course shells that make it hard to teach students what they need to learn in the ways in which they need to learn. Instructure was founded in 2008. They emphasized a few distinctions from their competitors out of the gate. The first was their native multitentant cloud architecture. Reduced downtime? Check. The second was a strong emphasis on usability. The big feature that they touted which was their early runaway hit was Speed Grader. Increased productivity? Check.
Instructure had found their updraft to give them their hockey stick growth.
But they also emphasized that they were going to be a learning platform. They weren’t going to build out every tool imaginable. Instead, they were going build a platform and encourage others to build the specialized the tools that teachers and students need. And they would aggressively encourage the development and usage of standards to do so. On the one hand, this fit from a cultural perspective. Instructure was more like a Silicon Valley company than its competitors, and platforms were hot in the Valley. On the other hand, it was still a little weird for the education space. There still weren’t good interoperability standards for what they wanted to do. There still hadn’t been an explosion of good learning tools. This is one of those situations where it’s hard to tell how much of their success was prescience and how much of it was luck that higher ed caught up with their cultural inclination at that exact moment.
Co-evolution
The very same year that Brian Whitmer and Devlin Daley founded Instructure, Chuck Severence and Mark Alier were mentoring Jordi Piguillem on a Google Summer of Code project that would become the initial implementation of LTI. In 2010, the same year that Instructure scored its first major win with the Utah Education Network, IMS Global released the final specification for LTI v1.0. All this time that the market had felt like it had been standing still, it had actually been iterating. We just hadn’t been experiencing the benefits of it. Chuck, who had been thinking about interoperability in part through his work on Sakai, had been tinkering. Students like Brian and Devlin, who had been frustrated with their LMS, had been tinkering. The IMS, which actually had a precursor specification before LTI, had been tinkering. While conditions hadn’t become visible on the surface of the glacier, way down, a mile below, the topology of the land was changing.
Meanwhile in Arizona, in 2009, the very first ASU+GSV summit was held. I admit that I have had writer’s block regarding this particular conference the last few years. It has gotten so big that it’s hard to know how to think about it, much less how to sum it up. In 2009, it was an idea. What if a university and a company that facilitates start-ups (in multiple ways) got together to encourage ed tech companies to work more effectively with universities? That’s my retrospective interpretation of the original vision. I wasn’t at many of those early conferences and I certainly wasn’t an insider. It was hard for me, with my particular background, to know what to make of it then and even harder now.
But something clicked for me this year when it turned out that IMS LILI was held at the same hotel that the ASU+GSV summit had been at a couple of months earlier. How does the IMS get to 523 product certifications and $8 million in the bank? A lot of things have to go right for that to happen, but for starters, there have to be 523 products to certify and lots of companies that can afford to pay certification fees. That economy simply did not exist in 2008. Without it, there would be no updraft to ride and consequently no hockey stick growth. ASU+GSV’s phenomenal growth, and the ecosystem that it enabled, was another major factor influenced what I saw at IMS LILI this month.
There is a lot of chicken-and-egg here. LTI made a lot of this possible, and the success LTI (and IMS Global) have experienced would not have been possible without a lot of this. The harder you stare at the picture, the more complicated it looks. This is what “systems thinking” is all about. There isn’t a linear cause-and-effect story. There are multiple interacting feedback loops. It’s a complex adaptive system, which means that it doesn’t respond in linear or predictable ways.
Update: I got a note from Rob Abel noting that a lot of the growth in the last leg came from an explosion of participation in the K12 space. That’s good color and consistent with what I’ve seen in my last couple of LILI conference visits. It’s also consistent with the rest of this analysis. K12 benefitted from all of the dynamics above—the maturation of the LMS market, the dynamics in higher education online that pushed toward SaaS and usability, the massive influx of venture funding, and so on. All of those developments, plus the work inside IMS, made the K12 growth possible, while the dynamics inside K12 added another feedback loop to this complex adaptive system.
But respond it finally did. We have some semblance of a functioning market, and with its rise, blockers preventing the formation of a vibrant interoperability standards ecosystem of the type we have today have largely fallen. Now we have to address the blockers of the formation of the vibrant interoperability ecosystem that we will need tomorrow. Because it will be qualitatively different. Tomorrow’s blockers are not market formation problems but rather collaboration methodology problems. They are about creating meaningful learning learning analytics, which will require solving some wicked problems that can only be tackled through close and well structured interdisciplinary work. That most definitely includes the standards design process itself.
After the glacier comes the flood
What I saw at the IMS LILI this year was, I think, a milestone. The end of an era. Market pressures now favor interoperability. The same companies that were the most resistant to developing and implementing useful interoperability standards in 2007 are among the most aggressive champions of interoperability today. This is not to say that foundational interoperability work is “over.” Far from it. Rather, the conditions finally exist where it can move forward as it should, still hard but relatively unimpeded by the distortions of a dysfunctional market.
That said, the nature and challenges of interoperability our sector will be facing in the next decade are fundamentally different from the ones that we faced in the last one. Up until now, we have primarily been concerned with synchronizing administration-related bits across applications. Which people are in this class? Are they students or instructors? What grades did they get on which assignments? And how much does each assignment count toward the final course grade? These challenges are hard in all the ways that are familiar to anyone who works on any sort of generic data interoperability questions.
But the next decade is is going to be about data interoperability as it pertains to insight. Data scientists think this is still familiar territory and are excited because it keeps them at the frontier of their own profession. But this will not be generic data science, for several reasons. (I will tell you right now that some of them disagree with me on this. Vehemently.) First, even in the most richly instrumented fully online environments that we have today, they are highly data impoverished relative to what we need to make good inferences about teaching and learning. For heaven’s sake, Amazon still recommends things that I have already bought. If I just bought a toaster oven last month, then how likely is it that I want to buy another one now? And I buy everything on Amazon. If they don’t know enough to make good buying recommendations on consumer products, then there’s no way that our learning environments are going to have enough data to make judgements that are orders of magnitude more sophisticated.
Well then, some answer, we’ll just collect more data! More more more! We’ll collect everything! If we collect every bit of data, then we can answer any question. (That is a pretty close paraphrase of what one of the IMS presenters said in one of the handful of learning analytics talks I went to.)
No. You won’t collect “everything”—even if we ignore the obvious, glaring ethical questions—because you don’t know what “everything” is. Computer folks, having finally freed themselves from the shackles of SQL queries and data marts, are understandably excited to apply that newfound freedom to the important problem space of learning. But it is not a good fit, because we don’t have a good understanding of the basic cognitive processes involved in learning. As I wrote about (at length) in a previous post, we have to employ multiple cutting-edge machine learning techniques just to get glimpses of learning processes even when we are directly monitoring students’ brain activity because these are extraordinarily complex processes with multiple hidden variables. Trying to tease out learning processes inside a student’s head based on learning analytics from running machine learning algorithms on LMS data is a little like trying to monitor the digestive processes of a flatworm on the bottom of the Marianas Trench based on studying the wave patterns on the surface of the ocean. There are too many invisible mediating layers to just run a random forest algorithm on your data lake—it all sounds very organic, doesn’t it?—and pop out new insights about how students learn.
That doesn’t mean we should just throw up our hands, by any means. To the contrary, IMS Global has some extraordinarily good tools close at hand for tackling this problem. But it does mean that they are going to have to take some of the stakeholder engagement strategies they’ve been working at diligently to the next level, to the point where the standards-making process itself may evolve over time.
Theory-driven interoperability
There is an excellent data and processing resource that the learning analytics folks have yet to think deeply about how to leverage, as far as I can tell from the conference. The computational power is impressive (and impressively parallel). It is the collective intelligence of educators and learning scientists. Because there are too many confounds to making useful direct inferences from the data, educational inferencing needs to be theory-driven. You need to start with at least some idea of what might be going on inside the learner’s head. One that can be either supported or disproven based on evidence. And you need to know what that evidence might look like. If you can spell all that out, then you can start doing interesting things with learning analytics, including machine learning. There is room for learning science, data science, and on-the-ground teaching expertise at the table. In fact, you need all those kinds of expertise. But the folks with those respective kinds of know-how need to be able to talk to each other and work together in the right ways, which is really hard.
The IMS has an outstanding foundation for this sort of work, because their Caliper specification turns out to provide the basis for a perfectly lovely lingua franca. To begin with, its fundamental structure is triples, which is the same basic idea as the original concept behind the semantic web. If you’re not a computer person and this is starting to make your eye’s glaze over, don’t worry, because this is plain English. Three-word sentences, in fact. Noun, verb, direct object. Student takes test. Question assesses learning objective. Student highlights sentence. Sentence discusses Impressionism.
IMS Caliper expresses learning analytics in statements that can easily be translated into three-word plain-English sentences. These sentences can be strung together into coherent paragraphs. Notice, for example, how the last two example sentences are related. Three-word sentences in this format can be chained together to form longer thoughts. New thoughts. With this one, very simple grammatical structure, we have a language that is generative in the linguistic sense. As long as you have words to put into these grammatical placeholders, you can string thoughts together. Or “chain inferences,” to sling the lingo. And it turns out, unsurprisingly, that Caliper has a mechanism for defining these words in ways that both humans and machines can understand them.
That has to be the bridge. Humans have to understand the utterances well enough to be able express their theories on the front end and understand whatever the machine is telling them it may have learned on the back end. Machines have to understand them specifically enough to be able to parse the sentences in their own, literal, machine-y way. Theoretically, Caliper could be an ideal language to enable educators and computer scientists to discuss theories about how to better support students as well as how to test those theories together.
The challenge is that the IMS community, at least based on what I saw in the sessions I attended, is not using the specification as an interdisciplinary communication tool in this way yet. What I saw happening instead was a lot of very earnest data scientists pumping as much Caliper data as the can into their data lakes. They come to the conference, give a talk and, to their credit, shrug their shoulders and admit that they really don’t know what to do with those data yet. But then they go home and build bigger pipes, because that’s their job. That’s what they do.
It’s not their fault. I’ve been friends with some of these folks for a very long time indeed. There are good people here. But if you work in the IT department, and you’re not a learning scientist or a classroom educator, and the faculty are somewhere between dismissive and disdainful of the idea of talking to you about working together to improve teaching and learning, then what can you do? You do what you know how to do and hope that things will change for the better over time.
It’s not the IMS’s fault either. The conference I attended was called the IMS Learning Impact Leadership Institute. That’s not a new name. Caliper has board that helps guide its direction. That board includes educators who are the kind of advocates that I would like to see on such a body. They are productive irritants in the best possible way. But that’s not enough anymore. This is just a really hard problem. It’s the challenge of the next decade. To meet it, we need to do more than just make sure the right people are in the room together. We need to develop new ways of working together. New roles, methodologies, ways of talking with each other, and ways of seeing the world.
I’m going to preview a bit of a post that I have in my queue for…I’m not sure when, but some time soon…by mentioning “learning engineering.” This term has gotten a lot of buzz lately, along with some criticism. I’ll be writing up my own take on it, but for now I’ll say that one reason I think the term is gaining some currency is that it represents a set of skills for being a mediator in the kind of collaboration that I’m describing here.
As it turns out, it was coined by Nobel prize-winning polymath and Carnegie Mellon luminary Herb Simon, after whom Carnegie Mellon University’s Simon Initiative was named. And, as it also turns out, the Simon Initiative hosted this year’s EEP summit and made some news in the process by contributing $100 million worth of open source software that they use in their research and pratice of…wait for it…learning engineering.
Here’s a slide that they used in their talk explaining what the heck learning engineering is and what they are doing when they are doing it:
Copyright Carnegie Mellon University, CC-BY
(By the way, the videos of all talks from the summit will be posted online, as promised. Please be patient a little longer.)
This post has already run long, so rather than unpacking the slide, I’ll leave you with a question or two. Think about this graphic as representing a data-informed continuous improvement methodology involving multiple people with multiple types of expertise. What would that methodology need to look like? Who would have to be at the table, what kinds of conversations would they have to have, and how would they have to work together?
I’m not suggesting that “learning engineering” is a magical conjuring phrase. But I am suggesting that we need new approaches, new competencies, and likely a new role or two if we are going to get to the next updraft.
