Whatever else you think of the election, it has been the mother of all teachable moments for many of us. It has raised questions about what we thought we knew about our democracy, our neighbors, our media…and apparently learning analytics. The shock of the polls being “wrong” has raised a lot of questions about how much we can really trust data analytics. Audrey Watters has written the most fleshed out critique that I’ve seen so far. But Dave Cormier tweeted about it as well. And I have had several private conversations along these lines. They all raise the question of whether we put too much faith in numerical analysis in general and complex learning analytics in particular. That is an excellent question. But in doing so, some of these arguments position analytics in opposition to narratives. That part is not right. Analytics are narratives. They are stories that we tell, or that machines tell, in order to make meaning out of data points. The problem is that most of us aren’t especially literate in this kind of narrative and don’t know how to critique it well.
This is going to be a wide-ranging post that goes a little lit crit at times and dives into an eclectic collection of topics from election polling to the history of medicine. But even the most pragmatic, b-school-minded entrepreneur or VC may find some value here. Because what I’m ultimately talking about is a fundamental limiter on the future growth of the ed tech industry. The value of learning analytics, and therefore the market for them, will be limited by the data and statistical literacy of those who adopt it. The companies that are focused on developing fancier algorithms are solving the wrong problem—at least for now. These tools will have limited adoption until they are put into the hands of educators who understand their uses and limitations. And we have a long way to go in that department.
We now have the second of our personalized learning explainer videos out. As a reminder, here’s the first one, which reframes personalized learning as a set of approaches for addressing the teaching problems of reaching hard-to-reach students:
The new one starts to talk about the how, giving flipped class, learning analytics, and adaptive learning as examples of tools or approaches that can help teachers reach those hard-to-reach students:
Some folks continue to be confused about just how these things are supposed to help. We described some of our thinking in our original post on the first explainer and our recent piece in Inside Higher Ed. We are well aware of a number of hard limits on what we can accomplish, including but not limited to the following:
A couple of 3-minute videos are not going to change the world all by themselves.
The levers and barriers for change are different at different institutions.
Vendors are going to be out selling, the press is going to be out hyping, and institutional stakeholders are going to be looking for silver bullets no matter what.
But we also find ourselves in the fortunate position of having some credibility with academic administrators, ed tech advocates, and vendors. If we can help our readers to nudge the inevitable and already ongoing conversations about courseware products in a more healthy direction, away from robot tutors in the sky and toward enabling teaching practices that reach more students, that would be a good thing. We’re creating these explainers as tools that can be used by any of the parties to those conversations to reframe the discussion. If we’re lucky, we’ll influence the ways in which the vendors pitch these products in the first place while also preparing educators to take advantage of the different pitch to ask more educationally relevant questions and nudge the procurement process in a healthier direction. We won’t always be so lucky. But we won’t always be unlucky either. A lot comes down to each of you and what you can accomplish in your respective contexts. We’re just trying to give you a few more tools to work with. You’re the change agents.
After Monday’s post on my confusion with Blackboard’s overall Learn strategy, I thought I would follow up with a reminder that there is one really important area where there are strong early signs that Blackboard is doing something right in a very important area: learning analytics. Learning analytics is one of those areas where there are many, many people talking and very few who are actually making sense. Blackboard has been hiring people that I usually call up when I have learning analytics questions and want to talk to somebody whose answers will actually make sense. To start with, they hired John Whitmer, who I have praised on this blog before. To get a flavor of who he is, here’s a talk that he gave when he was still back working at CSU:
If you’ve heard John give a talk post-hire (as you will have a chance to do a little later in this post), you’ll know that he is just as straight-talking, funny, and insightful now as then. And from our observations both at BbWorld 2015 and since, he appears to be in a position of significant influence within the company.
More recently, the company has hired Mike Sharkey to run their whole analytics group as part of their acquisition of his company, Blue Canary. Like John, he is one of those all too rare people who is both very good at explaining how learning analytics work and very comfortable calling BS on hype. It was a little tougher to find a recording of one of Mike’s talks for some reason, but here’s one of him when he was at the University of Phoenix:
Also recently, the company acquired X-Ray, a learning analytics product that was developed for Moodle and that Blackboard eventually intends to make available for Learn. Although the product itself is interesting, one of the motivations for the acquisition was that it included creator Sasha Dietrichson. I haven’t met Sasha yet but he has a reputation similar to that of John and Mike. I actually had the pleasure of attending Blackboard’s international launch event for X-Ray. I held off blogging about it because, in my opinion, the product still needed a little polish as of that event, but it’s worth bringing up in this context. You can see the whole event, including John’s presentation of the product, here:
(I recommend that you fast forward past the annoying animated commercial at the beginning.)
We haven’t yet seen these hires bear fruit in big ways (except arguably with X-Ray for the Moodle customers), but there are all the early indicators of a coherent investment in an analytics strategy that could turn into a major differentiator. So far, most of Instructure’s analytics work has been limited to making data available for others to use. And the last time we checked, D2L’s analytics strategy was stuck in the mud (although we are overdue for an update in that department and will be looking into it again soon). Furthermore, one major advantage of the new Learn SaaS architecture is that it provides infrastructure for learning analytics that would just not be possible on the older architecture. It’s hard for the company to tout that now when they don’t yet have products that show off the benefits. There is little question at this point that Blackboard grossly underestimated the amount of time it would take them to get the new architecture ready for prime time.
It’s possible that part of what we are seeing going on with Blackboard’s communications…er…strategy is that their product announcements (if, indeed, Ultra is a product) were so far out in front of delivery of any demonstrable benefits that it’s hard for anyone to explain the point of them without sounding completely pie-in-the-sky. Jay Bhatt felt a need, for whatever reason, to make grand pronouncements about how Blackboard was going to “transform education.” We criticized him for those comments at the time, but it’s looking increasingly likely that the damage he did through this penchant for grandiosity was more far-reaching than we imagined. But my larger point here is that, when I say that I am confused about what is going on in Blackboard, I really mean it. There may yet be a baby floating in this tub of stinky bathwater.
In the first part of this series, I identified four design goals for a learning platform that supports conversation-based courses. In the second part, I brought up a use case of a kind of faculty professional development course that works as a distributed flip, based on our forthcoming e-Literate TV series on personalized learning. In the next two posts, I’m going to go into some aspects of the system design. But before I do that, I want to address a concern that some readers have raised. Pointing to my apparently infamous “Dammit, the LMS” post, they raise the question of whether I am guilty of a certain amount of techno-utopianism. Whether I’m assuming just building a new widget will solve a difficult social problem. And whether any system, even if it starts out relatively pure, will inevitably become just another LMS as the same social forces come into play.
I hope not. The core lesson of “Dammit, the LMS” is that platform innovations will not propagate unless the pedagogical changes that take advantages of those changes also propagate, and pedagogical changes will not propagate without changes in the institutional culture in which they are embedded. Given that context, the use case I proposed in part 2 of this series is every bit as important as the design goals in part 1 because it provides a mechanism by which we may influence the culture. This actually aligns well with the “use scale appropriately” design goal from part 1, which included this bit:
Right now, there is a lot of value to the individual teacher of being able to close the classroom door and work unobserved by others. I would like to both lower barriers to sharing and increase the incentives to do so. The right platform can help with that, although it’s very tricky. Learning Object Repositories, for example, have largely failed to be game changers in this regard, except within a handful of programs or schools that have made major efforts to drive adoption. One problem with repositories is that they demand work on the part of the faculty while providing little in the way of rewards for sharing. If we are going to overcome the cultural inhibitions around sharing, then we have to make the barrier as low as possible and the reward as high as possible.
When we get to part 4 of the series, I hope to show how the platform, pedagogy, and culture might co-evolve through a combination of curriculum design, learning design, platform design, prepared for faculty as participants in a low-stakes environment. But before we get there, I have to first put some building blocks in place related to fostering and assessing educational conversation. That’s what I’m going to try to do in this post.
Thanks to Audrey Watters I just read a new article in Science Magazine and publicly posted here by Justin Reich, the lead researcher for HarvardX (Harvard’s implementation of edX and associated research team) ((Note that Science Magazine access requires a subscription or purchase or individual article.)). Justin calls out the limitations of current MOOC research that focuses on A/B testing and engagement instead of learning, single-course context, and post hoc analysis with proper course design. While praising the field for making available cleansed data for any type of analysis, his core argument is that we need new approaches that cannot be solved just by research teams. Update: Added link to publicly-available DOCX article.
While the whole article is worth reading, there are quite a few insightful quotes should get past the journal paywall.
Big data sets do not, by virtue of their size, inherently possess answers to interesting questions.
We have terabytes of data about what students clicked and very little understanding of what changed in their heads.
It does not require trillions of event logs to demonstrate that effort is correlated with achievement.
D’Arcy Norman started a lively inter-blog conversation like we haven’t seen in the edublogosphere in quite a while with his post on the false binary between LMS and open. His main point is that, even if you think that the open web provides a better learning environment, an LMS provides a better-than-nothing learning environment for faculty who can’t or won’t go through the work of using open web tools, and in some cases may be perfectly adequate for the educational need at hand. The institution has an obligation to provide the least-common-denominator tool set in order to help raise the baseline, and the LMS is it. This provoked a number of responses, but I want to focus on Phil’s tworesponses, which talk at a conceptual level about building a bridge between the “walled garden” of the LMS and the open web (or, to draw on his analogy, keeping the garden but removing the walls that demarcate its border). There are some interesting implications from this line of reasoning that could be explored. What would be the most likely path for this interoperability to develop? What role would the LMS play when the change is complete? For that matter, what would the whole ecosystem look like?
Seemingly separately from this discussion, we have the new Unizin coalition. Every time that Phil or I write a post on the topic, the most common response we get is, “Uh…yeah, I still don’t get it. Tell me again what the point of Unizin is, please?” The truth is that the Unizin coalition is still holding its cards close to its vest. I suspect there are details of the deals being discussed in back rooms that are crucial to understanding why universities are potentially interested. That said, we do know a couple of broad, high-level ambitions that the Unizin leadership has discussed publicly. One of those is to advance the state of learning analytics. Colorado State University’s VP of Information Technology Pat Burns has frequently talked about “educational Moneyball” in the context of Unizin’s value proposition. And having spoken with a number of stakeholders at Unizin-curious schools, it is fair to say that there is a high level of frustration with the current state of play in commercial learning analytics offerings that is driving some of the interest. But the dots have not been connected for us. What is the most feasible path for advancing the state of learning analytics? And how could Unizin help in this regard?
It turns out that the walled garden questions and the learning analytics questions are related.
It would be deeply unfair of me to mock Blackboard for having a messy but substantive keynote presentation and not give equal time to D2L’s remarkable press release, pithily entitled “D2L Supercharges Its Integrated Learning Platform With Adaptive Learning, Robust Analytics, Game-Based Learning, Windows® 8 Mobile Capabilities, And The Newest Education Content All Delivered In The Cloud.” Here’s the first sentence:
D2L, the EdTech company that created the world’s first truly integrated learning platform (ILP), today announces it is supercharging its ILP by providing groundbreaking new features and partnerships designed to personalize education and eliminate the achievement gap.
I was going to follow that quote with a cutting remark, but really, I’m not sure that I have anything to say that would be equal to the occasion. The sentence speaks for itself.
For a variety of reasons, Phil and I did not attend D2L FUSION this year, so it’s hard to tell from afar whether there is more going on at the company than meets the eye. I’ll do my best to break down what we’re seeing in this post, but it won’t have the same level of confidence that we have in our Blackboard analysis.
