Pilots? We don’t need no stinkin’ pilots!

Timothy Harfield commented on Arizona State University’s approach to pilots and scaling innovation at ASU.

.@philonedtech excellent comment on the problem of scaling innovation in #HigherEd. This is a central concern for @UIAinnovation.

— Timothy Harfield (@timothyharfield) June 4, 2015

excellent comment on the problem of scaling innovation in #HigherEd. This is a central concern for @UIAinnovation.

The University Innovation Alliance is “a consortium of 11 large public research universities committed to making high-quality college degrees accessible to a diverse body of students”. I wrote about this “central concern” last summer in a post titled “Pilots: Too many ed tech innovations stuck in purgatory”, using the frame of Everett Rogers’ Diffusions of Innovations model. While the trigger for that post was on ed tech products, the same situation applies for the course design situation.

What we are seeing in ed tech in most cases, I would argue, is that for institutions the new ideas (applications, products, services) are stuck the Persuasion stage. There is knowledge and application amongst some early adopters in small-scale pilots, but majority of faculty members either have no knowledge of the pilot or are not persuaded that the idea is to their advantage, and there is little support or structure to get the organization at large (i.e. the majority of faculty for a traditional institution, or perhaps for central academic technology organization) to make a considered decision. It’s important to note that in many cases, the innovation should not be spread to the majority, either due to being a poor solution or even due to organizational dynamics based on how the innovation is introduced.

The Purgatory of Pilots

This stuck process ends up as an ed tech purgatory – with promises and potential of the heaven of full institutional adoption with meaningful results to follow, but also with the peril of either never getting out of purgatory or outright rejection over time.

Back to Timothy’s comment. He was specifically commenting on Phil Regier’s interview in the e-Literate TV case study on ASU.

Phil Hill: There are plenty of institutions experimenting with new technology-based pedagogical approaches, but pilots often present a challenge to scale with quality. ASU’s vision, however, centers on scale and access. One observation I’ve seen from what’s happening in the US is there are a lot of pilots, but that never scale to go across a school. You sound confident that you will be scaling.

Philip Regier: We kind of don’t pilot stuff here. When we did the math program, we actually turned it on in August 2012 after all of nine months of preparation working with Knewton.We turned it on, and it applied to every seat in every freshman math course at the university. And there’s a reason for that. My experience—not just mine, but the university’s experience with pilots is that they have a very difficult time getting to scale.
Pilots … have a very difficult time getting to scale.

Part of the reason is because, guess what? It doesn’t work the first time. It doesn’t work the first time, maybe not the second. It takes multiple iterations before you understand and are able to succeed.If you start with a pilot and you go a semester or two and it’s, “Hey, this isn’t as good as what we were doing,” you’ll never get to scale.

In our case, the experience with math is a very good example of that because working with a new technology is not a silver bullet. It’s not like we’re going to use this technology, and now all of the grades are going to go up by 15 percent. What you have to do is work with the technology and develop the entire learning ecosystem around it, and that means training faculty.

That’s one approach to the scaling innovation challenge that affects not just the University Innovation Alliance institutions but most schools. This approach also raises some questions. While Phil Regier stated in further comments not in the episode that faculty were fully involved in the decision to implement new programs, are they also fully involved in evaluating whether new programs are working and whether changes are needed? Does this no pilot approach lead to the continuation of programs that have fatal flaws and should be ended rather than changed?

It is, however, an approach that directly addresses the structural barriers to diffusing the innovations. Based on Phil Regier’s comments, this approach also leads to investment in and professional development of faculty members involved.