e-Literate

Present is Prologue

Tag: Empirical Educator Project

  • EEP 2019 Will Be Live Audiostreamed

    Interest in the Empirical Educator Summit (EEP) has been off the charts. We want and intend to include everybody, but only when we can include people in a way that is useful to them. So we are being intentional about the pace and ways in which we are growing.

    That said, we know a lot of people are very interested. We had already planned to release video of much of the summit after the fact. We’ve decided that we’re going to try to live stream the audio as well. (My experience with live-streaming video is that there isn’t much value in the visuals unless your setup is better than we will be able to manage, so we’d rather focus on trying to get you a solid audio stream.)

    We have a placeholder page set up at http://empiricaleducators.net/2019-eep-summit/. Between now and Monday, we will be posting an agenda of the summit and putting up a widget for the audio streaming on that page. Check there periodically for updates. For planning purposes, I can tell you now that the audio streaming will be from 9 AM to 3:30 PM EST on Monday, May 6th and from 9 AM to 12 PM EST on Tuesday, May 7th. Again, the agenda will be posted on the EEP summit page soon. This is a last-minute addition driven by demand, so we’re winging it a bit.

    We also invite you to discuss the summit on Twitter as it is streamed. We will not have the luxury of a dedicated social media person to monitor and respond to the conversation live, but we will be encouraging the on-site community to participate and will definitely be looking at what you have to say afterward to see what we can learn from your input. The hashtag for the event is #EEP2019.

    We’re adding two more hashtags for more specific input, since EEP is ultimately about doing things together. If you use these, please be sure to catch the early sessions on Monday that explain the goals of EEP so that your input is on point. The first hashtag, #EEP2019ideas, is for suggestions about how EEP members—both current and prospective—can work together to accomplish the goals of the network. The second, #EEP2019challenges, is for obstacles you want us to be aware of as we think about how to build out the collaborative network.

    To prepare you for the streaming of the event, I’m going to assign you some homework. The main reading is very short. I just published a piece in Forbes about Carnegie Mellon’s contribution. It’s not what you’re used to reading from me in that Forbes required the piece to be only about 800 words and strictly enforced a requirement that readers shouldn’t need to have any knowledge of higher education or software whatsoever in order to understand the article. The downside of these requirements is that I had to flatten and truncate some details and nuances that e-Literate readers are used to getting from me. (One example that I particularly want to get off my chest is that I briefly described the fruits of Lumen Learning’s collaboration with Carnegie Mellon but wasn’t able to give them proper credit.) But there were some benefits to those restrictions too. I think the piece captures something of the sense of professional identity and culture that both Carnegie Mellon and EEP seek to foster. Also, did I mention that it’s probably the shortest piece by me that you will ever see? Go read it.

    Beyond that, if you want to get a deeper sense of the train of thought behind the effort, take a dip into the archive of EEP-related blog posts here at e-Literate.

  • Carnegie Mellon and Lumen Learning Announce EEP-Relevant Collaboration

    Late last week, Carnegie Mellon University (CMU) and Lumen Learning jointly issued a press release announcing their collaboration on an effort to integrate the Lumen-developed RISE analytical framework for curricular materials improvement analysis into the toolkit that Carnegie Mellon announced it will be contributing via open licenses (and unveiling at the Empirical Educator Project (EEP) summit that they are hosting in May).

    To be clear, Lumen and Carnegie Mellon are long-time collaborators, and this particular project probably would have happened without either EEP or CMU’s decision to contribute the software that they are now openly licensing. But it is worth talking about in this context for two reasons. First, it provides a great, simple, easy-to-understand example of a subset of the kinds of collaborations we hope to catalyze. And second, it illustrates how CMU’s contribution and the growth of the EEP network can amplify the value of such contributions.

    RISE

    The RISE framework is pretty easy to understand. RISE stands for Resource Inspection, Selection, and Enhancement. Their focus is on using it to improve Open Educational Resources (OER) because that’s what they do, but there’s nothing about RISE that only works with OER. As long as you have the right to modify the curricular materials you are working with—even if that means removing something proprietary and replacing it with something of your own making—then the RISE framework is potentially useful.

    From the paper:

    In order to continuously improve open educational resources, an automated process and framework is needed to make course content improvement practical, inexpensive, and efficient. One way that resources could be programmatically identified is to use a metric combining resource use and student grade on the corresponding outcome to identify whether the resource was similar to or different than other resources. Resources that were significantly different than others can be flagged for examination by instructional designers to determine why the resource was more or less effective than other resources. To achieve this, we propose the Resource Inspection, Selection, and Enhancement (RISE) Framework as a simple framework for using learning analytics to identify open educational resources that are good candidates for improvement efforts.


    The framework assumes that both OER content and assessment items have been explicitly aligned with learning outcomes, allowing designers or evaluators to connect OER to the specific assessments whose success they are designed to facilitate. In other words, learning outcome alignment of both content and assessment is critical to enabling the proposed framework. Our framework is flexible regarding the number of resources aligned with a single outcome and the number of items assessing a single outcome.


    The framework is composed of a 2 x 2 matrix. Student grade on assessment is on the y-axis. The x-axis is more flexible, and can include resource usage metrics such as pageviews, time spent, or content page ratings. Each resource can be classified as either high or low on each axis by splitting resources into categories based on the median value. By locating each resource within this matrix, we can examine the relationship between resource usage and student performance on related assessments. In Figure 2, we have identified possible reasons that may cause a resource to be categorized in a particular quadrant using resource use (x-axis) and grades (y-axis).

    Figure 2. A partial list of reasons OER might receive a particular classification within the RISE framework.

    By utilizing this framework, designers can identify resources in their courses that are good candidates for additional improvement efforts. For instance, if a resource is in the High Use, High Grades quadrant, it may act as a model for other resources in the class. If a resource falls into the Low Use, Low Grades quadrant, it may warrant further evaluation by the designers to understand why students are ignoring it or why it is not contributing to student success. The goal of the framework is not to make specific design recommendations, but to provide a means of identifying resources that should be evaluated and improved.

    Let’s break this down.

    RISE is designed to work with a certain type of common course design, where content and assessment items are both aligned to learning objectives. This design paradigm doesn’t work for every course, but it works for many courses. The work of aligning the course content and assessment questions with specific learning objectives is intended to pay dividends in terms of helping the course designers and instructors gain added visibility into whether their course design is accomplishing what it was intended to accomplish. The 2×2 matrix in the RISE paper captures this value rather intuitively. Let’s look at it again:

    Each box captures potential explanations that would be fairly obvious candidates to most instructors. For example, if students are spending a lot of time looking at the content but still scoring poorly on related test questions, some possible explanations are that (1) the teaching content is poorly designed, (2) assessment questions are poorly written, or (3) the concept is hard for students to learn. There may be other explanations as well. But just seeing the correlation that students are spending a lot of time on particular content are still doing poorly on particular related assessment learning questions leads the instructor and the content designer (who may or may not be the same person) to ask useful questions. And then there is some craft at the end about thinking through how to deal with the content that has been identified as potentially problematic.

    This isn’t magic. It’s not a robot tutor in the sky. In fact, it’s almost the antithesis. It’s so sensible that it verges on boring. It’s hygiene. Everybody who teaches with this kind of course design should regularly tune those courses in this way, as should everybody who builds courses that are designed this way. But that’s like saying everybody should brush their teeth at least twice a day. It’s not sexy.

    Also, easy to understand and easy to do are two different things. Even assuming that your curricular materials are designed this way and that you have sufficient rights to modify them, different courses live in different platforms. While you don’t need to get a lot of sophisticated data to do this analysis—just basic Google Analytics-style page usage and item-level assessment data—it will take a little bit of technical know-how, and the details will be different on each platform. Once you have the data, you will then need to be able to do a little statistical analysis. There isn’t much math in this paper and what little there is isn’t very complicated, but it is still math. Not everybody will feel comfortable with it.

    The typical way the sector has handled this problem has been to put pressure on vendors as consumers to add this capability as a feature to their products. But that process is slow and uncertain. Worse, each vendor will likely implement the feature slightly differently and non-transparently, which creates a greater challenge for the last point of friction. Features like this require a little bit of literacy to use well. Everybody knows the mantra “correlation is not causation,” but it is better thought of as the closest thing that Western scientific thinking can get to Zen koan. ((Outside of quantum mechanics, at least.)) If you think you’ve plumbed the depths of meaning of that phrase, then you probably haven’t. If we want educators to understand both the value and the limitations of working with data, then they need to have absolute clarity and consistency regarding what those analytics widgets are telling them. Having ten widgets in different platforms telling them almost but not quite the same things in ways that are hard to differentiate will do more harm than good.

    And this is where we fail.

    While the world is off chasing robot tutors and self-driving cars, we are leaving many, many tools like RISE just lying on the floor, unused and largely unusable, for the simple reason that we have not taken the extra steps necessary to make them easy enough and intuitive enough for non-technical faculty to adopt. And by tools, I mean methods. This isn’t about technology. It’s about literacy. Why should we expect academics, of all people, to trust analytical methods that nobody has bothered to explain to them? They don’t need to understand how to do the math, but they do need to understand what the math is doing. And they need to trust that somebody that they trust is verifying that the math is doing what they think it is doing. They need to know that peer review is at work, even if they are not active participants in it.

    Making RISE shine

    This is where CMU’s contribution and EEP can help. LearnSphere is the particular portion of the CMU contribution into which RISE will be integrated. I use the word “portion” because LearnSphere itself is a composite project consisting of a few different components that CMU collectively describes as “a community data infrastructure to support learning improvement online.” I might alternatively describe it as a cloud-based educational research collaboration platform. It is probably best known for its DataShop component, which is designed to share research learning research data sets.

    One of the more recent but extremely interesting additions to LearnSphere is called Tigris, which provides a separate research workflow layer. Suppose that you wanted to run a RISE analysis on your course data, in whatever platform it happens to be in. Lumen Learning is contributing the statistical programming package for RISE that will be imported into Tigris. If you happen to be statistically fluent, you can open up that package and inspect it. If you aren’t technical, don’t worry. You’ll be able to grab the workflow using drag-and-drop, import your data, and see the results.

    Again, this kind of contribution was possible before CMU decided to make its open source contribution and before EEP existed. They have been cloud hosting LearnSphere for collaborative research use for some time now.

    But now they also have an ecosystem.

    By contributing so much under open license, along with the major accompanying effort to make that contribution ready for public consumption, CMU is making massive declaration to the world about their seriousness regarding research collaboration. It is a magnet. Now Lumen Learning’s contribution isn’t simply an isolated event. It is an early leader with more to come. Expect more vendors to contribute algorithms and to announce data export compatibility. Expect universities to begin adopting LearnSphere, either via CMU’s hosted instance or their own instance, made possible the full stack being released under an open source license. This will start with the group that will gather at the EEP summit at CMU on May 6th and 7th, because one has to start somewhere. That is the pilot group. But it will grow. (And LearnSphere is only part of CMU’s total contribution.)

    With this kind of an ecosystem, we can create an environment in which practically useful innovations can spread much more quickly (and cheaply) which vendors regardless of size or marketing budget can be rewarded in the marketplace based on their willingness to make practical contributions of educational tools and methods that can be useful to customers and non-customers alike. Lumen Learning has made a contribution with the RISE research. They now want to make a further contribution to make that research more practically useful to customers and non-customers alike. CMU’s contributed infrastructure and the EEP network will give us an opportunity reward that kind of behavior with credit and attention.

    That is the kind of world I want to live in.

  • EEP, EDwhy, and Seeds

    So the news broke today about the Empirical Educator Project’s (EEP’s) year two experimental design, which we’re calling EDwhy. The “ED” stands for Educational Design,” so the full name means, basically, “Why is your educational design the way that it is?” It invites educators to interrogate their own designs and aspires to give them the tools to do so. Here is the press release.

    We have some good coverage to start you off from Inside Higher Ed and EdSurge. At IHE, Lindsay McKenzie goes broad. She starts with some good shoe leather work at Carnegie Mellon with some interviews. Pay close attention to the interview with Ken Koedinger, as he talks about (but does not name) a research finding called the doer effect, which I’m going to use as an example later in this blog post. She also provides a good refresher of the open source versus proprietary question that universities often face with substantial software intellectual property that they develop, and then touches lightly on EEP’s role with the EDwhy announcement at the end (although with a clutch statement from Duke’s Matthew Rascoff, who always seems to say the right thing with a lot of intellectual and moral clarity in very few words). If you’re looking to find a way into this story from the beginning in a compact way, Linday’s story one good route in.

    Meanwhile, Jeff Young at EdSurge has dug a little deeper into significance behind the EDwhy idea and mechanics. I think the question that is on everyone’s minds is, “OK, $100 million dollars, lots of software, cool learning science-y things, but really, how is this going to be made useful?” Jeff begins to explore that question, and I’m going to take a deeper dive in this post. He also has some commentary from me about why we chose the name we did. You’ll have to go read it on EdSurge to get those details, but I’ll say this much here: On e-Literate, where one of our major roles is to critique hype and protect against the dangers of  bad actors, we have an ethical obligation to throw some sharp elbows. With EEP, where we are not watching from the sidelines but actually entering the fray, we are mindful that our obligation shifts as our role shifts. We take the e-Literate lessons to heart while also attempting to be humble both about the accomplishments of those before us and how easy it is for us to fall into the same traps that very smart people before us have fallen victim to.

    But I don’t want to write about the naming decision too much here. Instead, I want to write about how we are going to attempt to live up to the humbling confidence that Carnegie Mellon expressed in us when they chose us as a partner in their grand project. Obviously, when they offered to make their enormous contribution through our fledgling organization, it both forced and empowered us to rethink how we would go about the project in Year 2. We had always planned to stop, evaluate, and iterate on the design after the first year, but this opportunity demanded a pretty dramatic rethink in approach which, to be honest, is still ongoing. We have an idea that I’m going to share with you now that I believe makes sense in concept but does not yet have a fine-grained implementation plan. We are working hard with our Carnegie Mellon friends to have a foundation in place by the time of the summit. We will also workshop the idea at the summit with the cohort to refine our approach. This is going to be a year-long project. So we expect to spend some time after the summit continuing to put pieces in place and fine-tuning as we go. At the end of the year, we will do a progress check, evaluate, and iterate.

    The Hackathon

    I am always mindful about appropriating terms from Silicon Valley culture because I think it tends to be reflexively idealized. That said, there is a lot to like about the educational value of a hackathon. It is a social, time-bounded, self-organizing, problem-based learning exercise. A group of people will get together to solve a defined problem over a period of time. That group is often cross-functional. They might have software engineers, user experience designers, end users, and so on. Hackathons have a tangible and several intangible goals. The tangible goal in the canonical case is a piece of software, but we can think of it more broadly as an artifact that has been tested and demonstrated to solve the problem that was the goal set out at the beginning of the exercise. The intangible goals often include learning how to work in a cross-functional team, learning how to solve difficult problems with unexpected wrinkles, and learning particular craft-related skills necessary to solve the problem (e.g., programming tricks or software testing techniques).

    This is a good model for the kind of culture building that EEP has always aspired to achieve and, I believe that inspired Carnegie Mellon to see us as a good fit for their own ambitions. While I want to be clear that I do not speak for them, my understanding of their goals from our conversations thus far is that it would be a mistake to interpret their primary goal to be broader adoption of their software and other tools. Sure, they want to see that happen. But my read is that they see that as a second-order effect, or maybe as means to an end. What I hear from them in our conversations is that they really want to make their approach to improving education broadly accessible and meaningfully useful. They call that approach “learning engineering,” which they seem comfortable with me characterizing as one flavor or methodology within a broader developing family that we call “empirical education.” The hackathon works to support this goal because it creates an environment in which people habitually self-organize in cross-functional groups to improve educational design in ways that empower greater student success. It brings together the right people around the right kinds of goals and conversations. If we can then empower them with the right tools and methods, we are on your way to promoting learning engineering. If we can achieve that,  we can unlock the real power of the big release, which is to help democratize the science of education.

    While I said I didn’t want to dwell on our name choice here, it’s probably worth spending a little time on the word “design” in the way we are using it in EDwhy. A number of different overlapping but distinct stakeholder groups in academia tend to compete for mindshare around this word—Design Thinking practitioners, Instructional Designers, Learning Designers, User Experience Designers, and others. Making sense of how these all connect yet are distinct from each other is non-obvious even before we get to culturally local differences in usage. To give one example, Herb Simon, in addition to being the father of Learning Engineering, is considered by some to be the grandfather of Design Thinking. These are two compatible but distinct and non-interchangeable disciplines. In most places outside of Carnegie Mellon, their practitioners tend to be either completely ignorant of each other or find themselves cast as rivals in educational solution design.

    “Design” in the EDwhy context is a holistic and colloquial term meaning, simply, the way you decided to put something together. A cross-functional EDwhy hackathon team might include people with knowledge of Design Thinking, Instructional Design, Learning Design, User Experience Design, and/or Learning Engineering. Who is at the table will depend on the specific nature of the challenge being tackled and the kinds of expertise needed to take it on.

    At any rate, as we started thinking about how to help our network digest Carnegie Mellon’s $100 million contribution—never mind the sum of all possible contributions from all current and future EEP participants—we started thinking about both the digestive process and coming up with a form that is digestible. Verbs and nouns.

    The hackathon is the verb. Theoretically, the hackathon is flexible enough to allow for projects of different sizes and ambitions, whether inter- or intra-institutional. We still very much want to encourage inter-institutional collaboration, but one lesson we learned last year is that inter-institutional collaboration is incredibly hard, even with a lot of work done by third parties to lower barriers. We have to build a gentle slope toward that level of collaboration. The hackathon is a form that lets people start small and grow in ambition. At some point, they will outgrow the form and need to form something more like a traditional project with more formal management structures.

    We aspire to reach the point where we have that problem. For now, we are focused on culture-building, and we hypothesize that the hackathon is a good ritual for accomplishing that while also delivering immediate educational utility.

    The Seeds

    The hackathon idea is simple enough to grasp in the abstract. The hard part is putting it together with the right packages that help people identify and solve new problems using the contributions from Carnegie Mellon or other participants. For this, we’ve developed the concept of an EDwhy “seed.” This is one of the pieces I will want to workshop with the EEP cohort, but there’s enough here conceptually that the general idea should be clear.

    We start with a general area of interest where some research has been done but where there are more questions to be answered. For example (and as I mentioned earlier, Ken Koedinger and his CMU colleagues have done some research into something called “the doer effect.” It means pretty much what it sounds like. The researchers were able to demonstrate, using solid, quantitative methods that learning by doing is, for example, about six times more effective than learning by watching a video.

    (Side note for all you liberal arts folks out there who are suspicious of this data stuff: This study more or less just made the case for constructivism. Using numbers and computers and statistics and stuff.)

    That’s an interesting finding, if not a shocking one, but it also highlights a lot that we don’t know. For example, is doing always better than watching a video (or reading) for learning? Should we throw out all books and videos? If not, then how much watching or reading is good? In what order? Does the subject matter make a difference? The expertise of the learner? Other characteristics of the learner? Other characteristics of the overall course design? Or course goals?

    Let’s make this more concrete. One of my favorite course designs is Habitable Worlds by ASU’s Ariel Anbar. There is a lot of learning by doing in that problem-based course, but also liberal use of video. It would be interesting to do some testing and experimentation to find out how to make the most out of the doer effect and find the optimal balance of the course elements.

    As it turns out, Carnegie Mellon’s contributions include the software that was used to conduct the original doer effect research. (The IHE article mentions LearnSphere. Spend a little time exploring that site if you’re curious.) That software includes a data repository with access to (appropriately anonymized) data that could be used to replicate the results (or try to run different analyses on the data), a visual workflow that makes the study easily repeatable with different data, and access to the underlying R packages (for those who can understand them) to make the research methods completely transparent. If you put together the original studies, the software, the workflows, the data to practice reproducing the results, the transparency of the methods, and wrap in some documentation, some training, and a number of suggested starter questions for investigation, you have a seed. A self-organizing community could take up that seed and develop a hackathon project. If there were also a community forum where the hackathon group could ask questions of statisticians, cognitive psychologists, and psychometricians, as well as some technical support folks, as well as share lessons learned with each other, then you could really have something.

    I’m guessing the net result might turn out to be what would call an “intermediate” seed. Not every team would have the capability to self-organize around something this complex. We’d like to develop beginner, intermediate, and advanced level seeds, where beginner seeds are approachable by non-technical groups, intermediate seeds might require some technical skill and some knowledge of experimental design, and advanced seeds are really for folks who have some serious specialist expertise in their groups. The I’ll defer on the final difficulty ratings of each seed, including the one I just described, to the creators and the early adopters. One skill set we will be learning in the EDwhy experiment is how to package up a seed to make it accessible and useful to different sorts of audiences. Eventually, we may develop profiles of hackathon teams that are richer than just beginner/intermediate/advanced.

    At any rate, our goal for the year is to prove out and refine the approach through some pilot seeds and hackathons. We don’t imagine that we will be able to address the entire surface area of Carnegie Mellon’s $100 Million contribution in the one-year time frame, but we do aspire to prove out a novel and sustainable support and diffusion mechanism, not only for the software but for the methods and the culture. And during this time, we will also invite other EEP members to develop and contribute their own seeds, some of which will be less technical or tackle entirely different types of educational problems than Carnegie Mellon’s seeds will. This is a general mechanism we will be trying out. Interestingly, another arrow that CMU has in its quiver is the Open Learning Initiative (OLI) authoring and delivery platforms. So we may very well find their contributions to seed development goes well beyond the open source software code, which I think is the way in which people are naturally tending to think about the contribution at this early stage in the process.

    Both learning and science—or any path to enlightenment, really—starts with a simple admission: “There is so much that I don’t know, and so much that I would like to understand better.” Big announcements like this generally run against the grain of that admission. We have an ingrained cultural notion that, after spending a $100 million, you are supposed know all the answers. After spending 7 years in graduate school, you are supposed to know all the answers. After getting all the press and all the buzz, you are supposed to know all the answers.

    Nope. Sorry. It doesn’t work that way.

    There is so much that we don’t know, and so much that we would like to understand better. If you keep repeating that mantra to yourself every time you hear something new about Carnegie Mellon’s contribution or about EEP or the EDwhy initiative, each new piece of information will make a lot more sense to you.

  • Empirical Educator Project/CMU Updates

    Empirical Educator Project/CMU Updates

    As promised, I’m returning to Carnegie Mellon University’s announcement of the $100 million contribution of tools, software, and content to support the democratization of learning science, as well as the relationship of that contribution to the Empirical Educator Project.

    Just that first sentence shows how much meat there is yet to be put on the bones. “Contribution?” “Democratize learning science?” What does all that mean? These are fair questions. Jeff Young has some good reporting up at EdSurge that connects some dots on what’s public so far. I’m going to do my best to explain why the answers are not yet as clear as they might be, when and how they will be come clearer, and then give some reminders of past conversations here on e-Literate that provide some breadcrumbs leading in the direction of where all this is going. There will be more news in the press that will drop on Wednesday, followed by some more analysis by me on Thursday. From there, the story will build in pieces, through the Empirical Educator summit on May 6th and 7th, and afterward.

    Let’s start with the big picture. Carnegie Mellon has been a pioneer in various types of theoretical and applied educational and cognitive science research—which is now fashionable to roll up into a ball and call “learning science”—for decades. While their work often doesn’t get the publicity it deserves, the breadth and depth is astounding. I’ve had the privilege of visiting a few times and taking deep dives. ((When I say deep, I mean deep. See, for example, https://mfeldstein.wpengine.com/can-there-be-microscope-of-mind/.)) There are very few institutions in the world that are in their league in terms of the scope of what they do.

    The university made a decision to take a lot of their work product—over $100 million worth, in fact—and make it broadly accessible to the academic community. Honoring this commitment in a real and practical way is…hard. Very, very hard. It’s about more than just releasing the source code to software, which is hard enough to do right in and of itself. This is about making science—not just the output but the practice—accessible and useful to non-scientist educators. On top of that, the components of the contribution were not all designed together as a single software platform. They consist of many research projects, developed by different teams and that may be more loosely or tightly related to each other. Imagine if a top research laboratory decided to turn itself into the Smithsonian Institution, making itself a hands-on science museum accessible to everyone without dumbing itself down. That’s not exactly the goal that Carnegie Mellon’s Simon Initiative has set for itself, but its’ the same spirit. It’s incredibly ambitious and not easily imagined, much less described in a single press release.

    In fact, doing this well is itself going to take some empirical experimentation. And that’s where the partnership with the Empirical Educator Project comes in. Carnegie Mellon had already offered to host this year’s summit on May 6th and 7th. I’m honestly not sure how long they’ve been mulling over this idea of the giant release, but at some point they came to the conclusion that we would be good pilot partners for their effort. This week, we will be describing our high-level approach to our pilot design for helping to make CMU’s contribution accessible. And not just CMU’s contribution, either, since EEP is based on the premise that many academic institutions have innovations to contribute if only we can get better at sharing them. The scale of CMU’s contribution has given us an opportunity to challenge our assumptions about how we should be going about this work. It’s a once-in-a-lifetime opportunity.

    As I said, more details while emerge in the coming days, weeks, and months, first as we learn how to tell such a big story and later as we learn from the grand experiment as it unfolds. For now, I want to leave you with two video playlists. The first is a set of interviews of Carnegie Mellon faculty that I recorded while visiting a few years ago. At the time, I had no agenda other than to capture their individual and collective views on “learning science” and its relationship to classroom teaching. The second is a set of interviews from participants after the first day of the first EEP summit. Taken together, I think you will see why there is a good fit.

    CMU interview YouTube playlist.

    EEP interview YouTube playlist.

  • Carnegie Mellon’s $100 Million Announcement

    This is going to be a short post, in part because I’m traveling, but I need to call your attention to a developing story, both because it’s huge in its own right and for its importance to the Empirical Educator Project (which, by the way, has a new website).

    Carnegie Mellon has announced a $100 million contribution in tools, software, and content that “that is intended to catalyze a new era of progress in educational effectiveness that is equal to the challenge of rapid change and growth in 21st century educational needs. The suite of tools is the product of over $100 million of research and development from a wide variety of funders.”

    The suite of tools will be released in stages over the next year and represents a major departure from the “silver bullet” or “moon shot” efforts to revolutionize education with technology in recent years. Instead, the contribution is intended to democratize the science of learning and empower educators across the world to become citizen scientists. Carnegie Mellon’s goal is to provide knowledge of how to conduct applied educational research that classroom educators, researchers and educational technology companies can learn, apply, extend and share with the global educational community.

    “We live in a moment when our educational institutions are in danger of a catastrophic failure that we cannot afford. College and university closures are becoming regular occurrences, even as tuition and student debt rise to record levels,” said Norman Bier, executive director of the Simon Initiative.

    “This is happening at the same time that even highly educated people need to continue learning in order to keep their skills up-to-date, and when people all over the globe need increasing access to high quality educational opportunities through technology. In the face of these institutional and structural challenges, demonstrably improving outcomes and learning for students must be our foremost concern,” Bier said.

    I have lots of good things to say about this approach, but for a preview, you can go back and review my post about ed tech hype being in remission. This announcement is a little hard to parse because it’s just a down payment on a complex story, and because it’s a big price tag thing from a big engineering school, but trust me: this is not the same old thing.

    The full list of what is being released has yet to be announced, but I’ve seen it, and it is mind-blowing. The breadth and depth are pretty astonishing. In fact, one of Carnegie Mellon’s  biggest challenges will be explaining all of what’s in it. This isn’t a tool or a platform. It’s a collection that’s in the process of being knitted together into an ecosystem. And the way that people inhabit that ecosystem is what will really matter.

    There will be a lot more to say on that in the near future. The university is going to be revealing a lot of the details of their contribution at our second annual Empirical Educator Project summit, which they are graciously hosting on May 6th and 7th. We will have some announcements between now and then, likely a flurry of announcements (from us and from other parties) around the time of the summit itself, and will be releasing video of many of the talks after the summit afterward. There are also some reporters working this story, so I will keep running updates of those stories as they come out over at the new Empirical Educator Project site and periodically collect them in my updates here as well.

    Watch this space.

  • EEP News: Carnegie Mellon and Duke Lower Barriers to Conducting Educational Research

    EEP News: Carnegie Mellon and Duke Lower Barriers to Conducting Educational Research

    I’m thrilled to announce our first Empirical Educator Project contribution. From the press release:

    Carnegie Mellon University and Duke University have shared newly available free tools that will significantly lower the barriers to conducting ethical educational research. The two universities contributed the tools through e-Literate’s Empirical Educator Project (EEP), an effort to promote broader adoption of evidence-based teaching practices and foster a culture of empirical education across higher education.

    As with all academic research involving human subjects, educational researchers must have their experimental designs approved by their university’s Institutional Review Board (IRB). If a researcher wants to study students or their work, they must explain how they will get the students’ informed consent to participate.

    This can be a major barrier that often prevents research from being undertaken. Teaching faculty who may be interested in conducting a study may decide that the bureaucratic burden is more than they can take on. Multiple universities that want to collaborate on cross-institutional studies will have to get approval from each institution’s IRB in an environment where there are no widely adopted standards for reviewing and approving educational research by these bodies. Educational technology companies that want to be more transparent and collaborative with universities about their own research into product efficacy can find the IRB process impractically time-consuming. As a result, far less educational research gets conducted in ways that are both reviewed for ethical practices and shared as credible research that contributes to the state of the art in learning science.

    Through e-Literate’s EEP, learning science researchers at Carnegie Mellon and Duke Universities discovered that each institution had developed a solution for part of this problem. Carnegie Mellon University has developed templates approved by their IRB that they estimate will accommodate approximately 80% of classroom research use cases. Meanwhile, Duke University has developed language and a process approved by their IRB for requesting and tracking informed consent from students.

    The two universities have released the tools under a Creative Commons Attribution (CC-BY) license and provided “train the trainer” support for the use of their templates and protocols. Together, these contributions could enable many of the educators and product designers who are already conducting informal educational research all over the world to participate in the same sort of social fabric that has enabled communities of researchers in other human sciences to tackle problems from cancer to Alzheimer’s disease.

    Jeff Young has a great piece up about this release at EdSurge, and I believe we will see something from The Chronicle in their teaching newsletter on Thursday. I’d like to give you my own take on the reasons why this an important milestone.

    It’s an example of untapped inter-institutional opportunity

    Carnegie Mellon and Duke are theoretically peers. To use one very imperfect measure, they are both ranked in the top 25 national universities by US News and World Report. They are both more specifically ranked in the top 15 such schools for undergraduate teaching. Both are seriously concerned with improving undergraduate education through research-based practices. The two institutions have been working on similar yet complementary efforts to make that research easier. You would think that they would have had opportunities to share their work with each other, or at least know about what the other is doing.

    Before EEP, they didn’t.

    On closer examination, the complementarity of the two efforts suggests the kinds of opportunities that higher education is missing. Carnegie Mellon University has one of the broadest, deepest, most impressive, and most historic learning science research programs on the planet. There are only a handful of universities that are even in their league, and Stanford may be the only university that rivals them in depth and breadth. Further, the university has made a commitment in the form of the Simon Initiative to “[p]rovide accessible tools and methods with which any person or institution can adopt and advance CMU’s approach to learning engineering, improving outcomes for their own learners,” globally, in addition to improving teaching and learning at Carnegie Mellon University itself. (Not that this is distinct from but complementary to their Eberly Center for Teaching Excellence, which is similar in purpose to the centers of teaching and learning at many universities.) To get a flavor for who they are and how they think, here’s a playlist of three e-Literate TV videos that highlight a few of their faculty members:

    Now, Carnegie Mellon is known more generally for its engineering prowess, so it’s no surprise that the folks at the Simon Initiative talk about “learning engineering.” (The initiative is named after the late Herb Simon, a cognitive scientist and CMU luminary who coined that term.) That mentality, as well as the orientation of an institution known for its learning science research, shows in their contribution. They studied the IRB applications submitted for educational research by their faculty—not just their learning science faculty, but all faculty interested in publishing their research projects, identified common traits, and developed a template that they estimate covers somewhere in the neighborhood 80% of those projects. They then had their IRB review and approve the template. Now, any educator at CMU who wants to do publishable educational research and can use the template gets their IRB application fast-tracked.

    Without taking anything away from Duke’s own learning science research capabilities, one of the things that Duke is among the very best in the world at is making their undergraduates’ educational experience life-changing. As one example, the university has a large number of “professors of practice.” In many big research universities, a tenure track is a kind of death trap. Faculty are worked to the bone for three to five years in the hopes of achieving tenure when, in reality, most of them will be sent packing in the end. And while they are on that treadmill, they have a disincentive to invest too heavily in their teaching lest they neglect the publications and grants that will increase their odds of not being shown the door. That’s not how Duke rolls. Rather than exploiting young faculty by dangling a carrot that will forever be out of reach, they offer many a professor of practice position. Such faculty get basically everything except tenure, including long-term employment, a decent salary scale, and a say in shared governance. But they must not only show that they are excellent educators but also conduct research in education. In effect, rather than “physics” or “art history” being their discipline, it’s “physics education” or “art history education.” I don’t know whether Duke or their faculty would put it quite this way, but I intend it to be a compliment. The point is that they incentivize faculty to become disciplinary experts in supporting their students through evidence-based practices. I have been to one of their teaching and learning conferences and interacted with these faculty members. It was heavily attended and the atmosphere was electric. I have been to many of these kinds of events; yet I have rarely seen faculty who were more actively engaged in asking good, probing questions about teaching practice than I did at Duke.

    This different context from CMU’s has resulted in a slightly different approach to the same problem. Like everyone else, Duke’s educators who want to publish their research have to go through IRB approval. And given that Duke has a world-class medical research program, their IRB is both very tough and very focused on privacy concerns. So their Learning Innovation center developed an informed consent tool called WALTer, where WALT stands for “we are learning too.” WALTer sits right inside the LMS for any course in which research is being conducted. Faculty who want to conduct such research are led through a decision tree that produces IRB-approved language for informed consent, based on the conditions of the experiment. Having this form in place helps the instructor get faster approval of the project. And once that approval is in place, WALTer helps ensure that students are given the opportunity to provide their informed consent (or not). Since WALTer has been launched, the Learning Innovation team has seen an increase in the number of faculty expressing interest in conducting educational research and a decrease in the time it takes educational research applications to be approved by Duke’s IRB.

    Peanut butter, meet chocolate. CMU’s IRB template and Duke’s informed consent template are completely complementary. But before EEP, the peanut butter was in the fridge and the chocolate was in the drawer. They just didn’t meet.

    It’s an example of untapped intra-institutional opportunity

    The Duke feedback that they are getting more educational research interest and getting that research approved faster is illustrative of a larger point. The current institutional structures and processes of colleges and universities are not designed to facilitate the development, testing, sharing, and adoption of evidence-backed practices which improve student success. The IRB process is just one (very painful) example. The disincentive for faculty on the tenure treadmill is another. There are many more.

    The question that I have heard asked ad nauseam for years and years now, from a wide range of people, is “How can we make faculty care more about teaching?” That is the wrong question on every level. A better question would be, “How can we design universities such that focusing on improving excellence at supporting student success is less painful and more rewarding?”

    Think about just about every hot ed tech-related trend you can think of. Retention early warning analytics. Adaptive learning. Competency-based education. Stackable credentials. MOOCs. All this activity (and money) is swirling around the problem of making it easier for students to learn and succeed at school. Now think about all the hot trends that focus on making it practical and rewarding for faculty to focus their energy and considerable intellectual talents on solving the same problem. Can you name one?

    An IRB form for educational research may sound like a small and boring thing, but it is a piece of cultural and institutional infrastructure that makes it a little more practical for motivated faculty to focus energy and intellectual talent on learning how to better support student success. “We Are Learning Too” indeed. This is the kind of work that will make college education better and, in the process, make all of those ed tech tools more useful. A laser scalpel doesn’t do much to promote health unless it’s wielded by a physician who knows how, when, and why to use it. ((Is a “laser scalpel” a thing? I may have just made that up. Anyway, you get the point.)) Without that knowledge, it risks doing more harm than good.

    It’s an example of untapped multi-institutional and institution/vendor collaboration opportunity

    So yay for Carnegie Mellon and Duke. What about everyone else?

    Well, for starters, they have both contributed their language under a Creative Commons license. (Duke’s WALTer tool is built using a third-party proprietary platform, so contributing the source code wasn’t an option. But somebody else could easily build a tool that supports the workflow in the release document.) So there’s that.

    IRBs are notoriously idiosyncratic. (Some might say arbitrary. I’m not saying that. But some might.) So you could see an IRB at an institution that’s very different from CMU or Duke having something like the following argument:

    IRB Member #1: Hey, CMU and Duke are super-rigorous research institutions that are way more focused on these sorts of things than we are. If it’s good enough for them, it should be good enough for us.

    IRB Member #2: Actually, exactly because CMU and Duke are super-rigorous research institutions that are way more focused on these sorts of things than we are, what makes you think that what makes sense for them will also make sense for us?

    These are both reasonable starting positions. The conversation that should flow from this is an examination of the templates in which an IRB member who wants to change a part of it will need to provide a justification for doing so. This is exactly the next step we want to foster for this project, preferably at multiple institutions. Where we’d like to end up is at a toolkit in which an institution of any type can look at variations—and justifications for those variations—provided by peer institutions so that they adapt and adopt them. We’d also like to supplement what we already have with some more fleshed out student data privacy guidelines that are, once again, education-appropriate. Data privacy in this sort of research is just as important as it is in, say, medical research. But the specific concerns and methods for dealing with them aren’t necessarily identical. We should be developing a sector-wide consensus on ethical practices that can augment what is already in the Duke and CMU IRB contributions.

    This would hopefully help to lower one barrier to conducting educational research everywhere. But it could potentially do much more than that. Under the law, if researchers want to conduct a multi-institutional research project, then each institution’s IRB must approve the project. And since no two IRBs use the same standards and most don’t have particular guidelines for educational research, doing research across two institutions is more than twice as hard. Three is more than three times as hard. Doing large-scale, multi-institution research quickly becomes impossible in most scenarios. Unless you’re a vendor, in which case it is trivial, because you are not required to go through IRB for your own research—as long as you don’t publish it in a journal. So vendors can conduct research on students in multiple institutions for their own proprietary purposes easily, but if they want to do the right thing by going through IRB approval and sharing what they’re learning through peer-reviewed journals, it’s way, way harder.

    Imagine if it were easier for everyone to do the right thing and submit an application for IRB approval, knowing that they will be going through a streamlined but academically validated process. Imagine the kind of research opportunities that could open up. Now imagine further if there were some technology infrastructure behind this. Imagine if we could track IRB approval and informed consent across institutions, gather appropriately anonymized student data, and share it among researchers in a repository that is designed to respect the student privacy constraints dictated by the approved IRB applications while giving more researchers access to more research-relevant data—including data that are gathered through student interactions inside vendor tools. Technologically, this is quite practical. The hard part is getting the policy infrastructure solid and widely adopted.

    This is what EEP is about. The problem isn’t that higher education is failing to innovate or that professors don’t care about teaching well. The problem is that we are flushing 99% of the existing efforts, potential opportunities, and good intentions down the toilet because we don’t have the cultural institutions and social infrastructure to support and sustain them. But that can change.

    Kudos and thanks to the good folks at CMU and Duke. We have more folks working in EEP—from a diverse range of institutions—on a wide range of projects. And we’re still learning how to work together effectively. Expect more to come.

  • Toward a Methodology of Change in Higher Ed

    Toward a Methodology of Change in Higher Ed

    For a couple of years now, we’ve been saying that higher education is at the beginning stages of a long transition from a philosophical commitment to student success toward an operational commitment to it. In other words, colleges and universities are beginning to grapple in earnest with how to rewire themselves so that their culture and processes are deliberately optimized and continuously tuned to support their students in getting the best education possible. This is a profound shift. It will require major changes to the ways in which academia works and the ways in which ed tech designs and markets its products. It will be very hard and take a long time. But the drivers of this change are in place.

    Recently, I wrote about how our concept of Empirical Education has built into it a theory of change. The implication is that it has the backbone for a methodology of change. Our work as both analysts and consultants shown us that the increasingly aligned strategic priorities throughout the sector, when combined with the knowledge that is scattered across it, can be distilled down into a powerful yet flexible methodology for system change in education analogous to Design Thinking or one of the Agile software development methodologies. It can be a set of processes, built on a fairly small set of fundamental principles but supported by a lot of detailed craft knowledge and a rich ecosystem of supporting tools. It can be owned by no-one, although there would likely be some premiere practitioners of it. Colleges and universities could use it to redesign themselves to be more student-centric and, in the process, also more educator-centric. Product and service companies could design their offerings around it and compete based on their ability to help their academic customers better implement it.

    This sense of possibility has been the animating impulse behind the Empirical Educator Project (EEP). We started with only a hazy idea of what we were building. Over the last twelve months of working with academics and ed tech product people, some aspects have become clearer. I have grown more confident in the potential of the idea even as I have grown more overwhelmed with clearer understanding of the size of the undertaking.

    I am going to articulate my latest thinking about it in this post.

    The time is now

    There is a saying among consultants that potential clients won’t hire a consultant until and unless they both realize that they have a serious problem and come to accept that it is not a problem they can solve on their own. That holds equally true for a wide range of difficult changes that require help or cooperation, from coping with an addiction to building a functioning government to changing an institution. Higher education has been an incredibly stable system. As in, remarkably consistent over a period of about a thousand years. Historians of education tend to write about changes that take place over decades or half-centuries. There has been a looming question of whether such a slow-changing institution can adapt to such fast-changing times. Unsurprisingly, this debate has been raging for a few decades now, with relatively little sector-wide change to show for it. Is the system terminally rigid, or is it in a state of punctuated equilibrium that will shift in an appropriately dramatic amplitude once it reaches an inflection point?

    I believe the latter is the case, and I believe that we are at that inflection point. Access-oriented institutions—particularly publicly funded ones—have already been under pressure for some time now to show better outcomes for students in terms of rough measures like graduation rates and time to graduation, as well as some more meaningful but difficult measures popping up on the margins such as employment and career success. On the other end of the spectrum, the elite institutions whose brands have popularly defined excellence in education for the last century or more are starting to realize that they need to adjust to changing student expectations if they are going to continue to be considered the gold standard for the next century or more. The MOOC craze was complex and problematic, but it woke the elites up to the potential for use of technology-enabled approaches to enhance their teaching practices rather than detract from it, even as their students show up on campus with increasingly high expectations for the kinds of access to knowledge, interactive experiences, and high-touch communication that technology can enable. And in the middle, private universities with decent regional reputations and tuitions that approach those of Ivy League schools are increasingly under pressure to justify their tuition with something of more permanent value to students than climbing walls and dining halls. More and more, the buzz is about innovative partnerships with employers, or about learning analytics, or about student success systems supporting better guidance counseling. In other words, we are seeing colleges and universities grope toward approaches that enable them to more reliably support student success. And they are looking for help to do it.

    The focus of that previous paragraph is primarily on undergraduate education, but it also increasingly applies to graduate education. We sometimes see this problem manifest itself in financial terms, where it gets somewhat obscured by the current conversations around Online Program Management (OPM) companies. Universities often launch career-oriented graduate programs such as MBAs and MSWs because (a) they are looking for more revenue to make their institutions more sustainable, (b) online programs can scale without scaling costs like real estate and physical classrooms, and (c) they know there is a market of people who are inclined to sign up for online graduate programs that can fit with their work and family schedules while also giving them credentials that will help them advance in their career ambitions. But as the online MBA market gets saturated, universities increasingly have to find differentiators. And they can’t use climbing walls or dining halls. In the end, the only effective and durable differentiator for an online career-oriented graduate degree program is its effectiveness at helping the students achieve their goals. In this space, the immediate university driver is revenue and the immediate student goal is career advancement. So the sector tends to view this change narrowly. But if you zoom out a little, it becomes clear that the trend with graduate programs and OPMs is just one particularly clear example of where the academic institution’s financial sustainability issues are driving it toward a sharper operational focus on its mission.

    Let’s turn now to the educational vendors, who are also at an inflection point across product categories. All of these companies—curricular materials providers, LMS vendors, SIS vendors, analytics vendors, and so on—they are all looking to move up the value chain and argue that their products can directly, meaningfully, and provably impact student outcomes. ((I am use phrases like “student outcomes” and “student success” interchangeably and broadly for the purposes of this post, even though I know that they can have different connotations.)) They have to, because most of the major ed tech product categories are either in danger of commodifying or in danger of failing (in the case of established product categories) to achieve meaningful market penetration (in the case of new ones).

    The textbook companies hit the wall first. As students increasingly found ways to avoid buying new books (or any books), the textbook publishers raised their prices, which started a vicious cycle of reduced sell-through followed by price increases followed by further reduced sell-through followed by further price increases. This was ultimately unsustainable, particularly since the internet has made obtaining basic factual information and focused educational supplements—think YouTube—easily obtainable and free. Increasingly, publishers had to make the case that their content is somehow better than the commodity content. But better how? For a long time, the “better” publishers worked on was instructor convenience. But there’s only so far that slides, extra problem sets, and auto-graded homework can compensate for the vicious pricing cycle, particularly since the commodity materials get more organized and feature-rich over time. Eventually, the major publishers came to the conclusion that the only sustainable “better” they could shoot for is more educationally effective.

    Pearson was the first out of the gate with a massive push for “efficacy.” ((Disclosure: Pearson is a sponsor of EEP.)) They have bet and are still betting the company on that strategy. But as I have written about here before, the fundamental problem is that products can’t really be “efficacious” in and of themselves unless the educators in whose class the materials are being used (a) agree with the efficacy goals that have been defined by the product developers and (b) change their teaching to work with the educational strategies designed into the products. More fundamentally, the educators have to trust the research claims of the vendors in order to even think about the product-defined efficacy goals, much less adjust their teaching strategies. Pearson’s original articulation of efficacy failed to account for any of this. They have since adjusted their course, and other curricular materials developers—most notably McGraw-Hill Education and Macmillan among the larger players—have followed suit by also focusing more on encouraging faculty to buy into research-backed teaching practices and then, having obtained that buy-in, show how their products support and implement those practices. ((Disclosure: McGraw-Hill Education is a sponsor of EEP and subscriber to our Trusted Advisor market analysis service. Macmillan is a sponsor of EEP.)) But for all their good efforts—and they are generally, good, honest efforts—these vendors are pushing string. Most academics will never take them seriously as a source of advice for considering deep and scary changes to their teaching practice.

    Meanwhile in the LMS space, the developed markets have saturated and are stabilizing. New adoptions appear to be down. There are many developing markets to plumb, but they are slow and expensive to develop. So LMS vendors too have been trying to move up the value chain by talking more and more about student success. D2L has focused for some time now on the course design process and has been adding tools to its portfolio like LeaP, which is a tool for recommending personalized supplemental curricular materials. ((Disclosure: D2L is a sponsor of EEP and a subscriber to our LMS market analysis service.)) Blackboard has gone so far as to promote themselves as “your partner in change,” to the point of deprecating their flagship LMS project as “not enough.” ((Disclosure: Blackboard is a sponsor of EEP and a subscriber to our LMS market analysis service.))

    And yet, the LMS companies face the same uphill battle with credibility that the textbook publishers do. By and large, academics are not going to look to their LMS providers for guidance on how to change their teaching practices. The same goes for the upstart product categories like learning analytics. Vendors will struggle to convince academics to change their teaching practices, but their products will mostly fail to demonstrate meaningful learning impact until the academics adopt practices that take full advantage of the products. All these vendors need to climb a wall of credibility with academics, but they can’t do it unless somebody throws them a rope. (Companies with significant faculty-facing service components have the best chance of swimming upstream, but that’s another post for another time.)

    All the institutions in the sector—all types of colleges and universities, all types of ed tech vendors—have realized that they have a problem and are starting to realize that they can’t solve it on their own. They recognize that the core problem is that colleges and universities need to get much better at supporting student success, however their particular students may define it. They all want to get there and are starting to look to each other for help. But they don’t know how, and most of them can’t do it alone.

    There’s only one stakeholder group in this picture that has not gone through the process of seeing that they have a deep problem and accepting that they need help solving it yet. Have you spotted who they are?

    The people who can actually solve the problem

    While the shift in incentives has reached a tipping point for the institutions, the same cannot be said for the faculty. Their graduate training is largely unchanged. Their tenure and promotion criteria are largely unchanged. The rewards and accoutrements of professional accomplishment are largely unchanged. Faculty have been given no reason to change; therefore, they don’t. Everybody knows this is true.

    Or not. There are several vital aspects of this story which everybody “knows” that are either misleading or flat out wrong.

    First, faculty do change. Anybody who has significant experience with the development of online learning programs or other course redesign efforts has seen it happen. They have faculty say that their experience in the redesigned class has changed the way they teach in other classes. They have watched skeptical faculty turn into preachers of the gospel. There are converts. Despite a dearth of incentives and a plethora of disincentives, despite uneven support, despite the fact that most will earn no glory for it on the other side of the closed doors of their respective classrooms, faculty do embrace pedagogical change when they have the right sorts of experiences that enable them to see the benefits.

    Where are these amazing faculty members? They are everywhere and nowhere. They tend to be invisible on their home campuses, although if you ask around in different departments, you might be lucky enough to catch sight of one or three. (Or a dozen.) They have often learned the hard way that there is little benefit and significant pain involved with preaching on their home campuses, so many of them keep quiet and quietly work their magic in their own classrooms. If you want to see them in numbers, you usually have to go to one of the conferences where they congregate. I am going to one this week. One of the main activities of the participants will be crying on each other’s shoulders about how under-appreciated and under-resourced their efforts are on their respective home campuses.

    It is also untrue that incentives for faculty to excel in their teaching craft remain rare. It’s still early days, but there are green shoots everywhere. Most of the time, we only hear about a small number of schools that are doing remarkable things. Arizona State University, Southern New Hampshire University, and Western Governors University, over and over again. If you’re a little more knowledgeable, you might have heard about work at University of Central Florida or Georgia State University. And if you’re paying attention to formal scholarship, you might a little about work coming out of places like Carnegie Mellon University, Duke, and Stanford. We could look a little further down the publicity pyramid at places like the University of Maryland Baltimore County. You very likely haven’t heard about the amazing work happening at diverse schools ranging from James Madison University to Coppin State University. I wouldn’t have known anything about the accomplishments of either of these institutions if I hadn’t stumbled upon them through my various travels in this very odd job of mine.

    And because the news tends to focus on a few exceptional institutions, it also focuses on three contributors to success that are among the hardest to change: leadership, governance, and money. It is simply not true that the only institutions making real change have once-in-a-generation presidents, an iron grip on the faculty, and/or tons of funding. We see innovation everywhere. And everywhere it happens, it happens because institutions are finding new ways to draw on their most precious yet plentiful resource: their faculty.

    There is an old term of art that deserves reviving and refreshing: the scholarship of teaching and learning (SoTL). SoTL is often seen as a grassroots effort by faculty who care about teaching to wrap it in the cloak of academic validity. If the only way that excellence in teaching will be valued by the institution is to get it into peer-reviewed journals, then let’s find a way to get it into peer-reviewed journals. In the past, institutions generally didn’t take the bait. Many treated SoTL as a pat on the head to faculty who were slaving away carrying the heaviest teaching and advising loads. “Here, you care about this teaching stuff. Have a workshop. You can pretend what you’re doing is scholarship for a while. And we’ll give you a certificate!”

    That is changing. More and more institutions are realizing that faculty aren’t the problem; they are the solution. But that grassroots energy that comes from SoTL and other faculty empowerment efforts must be aligned with institutional efforts through support, incentives, and research. More and more institutions are making that connection. For example, here’s a graphic illustration of the dynamic, taken directly from Georgetown University’s Designing Our Future(s) web site:

    Here are some lessons learned from Georgetown’s white paper about the progress the initiative has made so far:

    A few core rules for this innovation work have emerged. First, every project has to push against some structural constraint (the 15-week semester, the credit hour, the nine-month calendar, etc.) and test variations of it. Second, projects cannot be idiosyncratic or depend on the particular interests of one talented faculty member; they have to be pilots from which we can generalize and which we might apply to other scenarios or problems. Lastly, we only fund a Red House project for one year (or the equivalent); after that, if a project is to survive, it has to be absorbed into the curriculum and faculty workload.

    Beyond these basic rules we have also learned some valuable lessons about the viability of experimental and creative curricular work in a culture designed for deliberative shared governance and slow change:

    • We developed strong stakeholder involvement as part of our iterative design process—one that frequently included associate deans, the registrar, compliance officers, and financial aid representatives—early in each project’s development. Likewise, we communicated well and regularly with our board, alumni, and donors
    • We do not give ourselves as good a grade on continuous communications with faculty. Early on there were many open invitations and speaker events, and a drumbeat of updates. As the work became more intense and demanding, we focused inward, and neglected to continue to reach back out to this important community. We learned it is absolutely critical to spiral communications outward, and to be as inclusive and open as possible, especially as the work takes specific shape within a core group.
    • Very early on we should have established a formal faculty review and approval process for Red House pilots. We assumed we would work within the Curriculum Committee approval structures, long established for important reasons, but which do not in the end benefit a research and development initiative. Last year, a Designing the Future(s) Advisory Committee was created, with the sole mission of approving and monitoring innovation projects. This system is now working very well; it might have accelerated progress if it had been instituted earlier.

    These are very early lessons, and they are somewhat Georgetown-specific. But it’s easy to see some more general principles emerge that could be useful across a wide range of educational and cultural contexts. And some of the most fascinating and remarkable changes are happening at institutions that you never read about, including some that have traditional faculty governance, few financial resources, and leaders who are extraordinary in the “normal” sense that many committed, hard-working, people-oriented academic leaders are in colleges and universities of all shapes and sizes.

    I am going to write about some specific examples of this sort of organizational alignment in upcoming posts. For now, I want to spend a little time on the characteristics of a good methodology.

    Toward a methodology of Empirical Education

    When I think about general methodology that can be adopted and adapted across a wide range of contexts, the two models that come to mind immediately are Agile software development and Design Thinking. I’ll focus on Agile (and particularly Scrum) for the moment because I know it better, but as far as I can tell, the same basic principles apply to Design Thinking.

    First, the methodology should be designed to unleash the creativity of the knowledge workers involved in the critical processes. All too often, we take really smart people and put them in a strait jacket of process. We tend to design our mission-critical processes to get us predictable results, often by controlling the human element through various management techniques. The problem arises when we ask for predictable results in an unpredictable environment, having handicapped the very smart people who are best able to minimize the problems that arise out of unforeseen circumstances while maximizing the benefits of unforeseen opportunities. There is no knowledge work I know of that has more frequent and dramatic unforeseeable challenges and opportunities than education. We wrap a lot of process around education, but it’s not the right kind of process to promote excellence by getting the most out of talented educators, just as using Gantt charts was not the right sort of process to promote excellence in software development by getting the most out of talented engineers.

    At the same time, empowering knowledge workers is not the same thing as letting them do whatever they want. I have been in an Agile software development environment where the engineers interpreted Agile to mean that they decide everything. The results were not good. All Agile methods that I am familiar with have multiple roles, with each role having certain authority and responsibilities. These roles are designed to be mutually supportive, and the success or failure is very much a success or failure of the entire team and its teamwork. This is a big cultural change for many institutions, where “academic freedom” has come to be used reflexively as a shield from any demands, sometimes because some of those demands are unreasonable or unwise. There has to be a well-defined process by which student success is understood to be the collaborative responsibility of the academic team, working together as an ensemble.

    These two basic principles—empowering individuals and working as teams—can generally be captured in a fairly small number of rules and roles, regardless of the flavor of Agile being practiced. And most Agile teams that get them right will function adequately while getting more satisfaction from their work—under relatively unchallenging circumstances. They may even feel that they are doing Agile well. But then there is a whole world of craft that is all about handling context-specific challenges. How do you balance functional versus non-functional requirements? How do you prioritize aging aspects of the software, a.k.a. “technical debt”? How do you manage large projects that require many Agile teams? How do you deal with extrinsic constraints on release timing (like the start of an academic term)? Agile practitioners can always improve their craft, both as individuals and as teams. Entire industries of tools and consulting have grown up around supporting excellence in that craft.

    Which is utterly unlike the way in which the industries that surround education function (or fail to function) today. There is a reason for that. An industry designed to promote operational excellence of knowledge workers cannot succeed in absence of a shared understanding among the knowledge workers about what operational excellence looks like. Agile software development is a craft with a lot of consensus around the principles, a track record of results, and enough expert practitioners that knotty problems, along with their solutions, can be shared fairly efficiently across a very large and loosely organized profession. There is a lot of debate too, which is the sign of a healthy ecosystem of knowledge workers advancing the leading edge of their craft. But that debate occurs within the context of a common understanding that is woven into the culture. Practitioners in those debates are rewarded with recognition of their expertise and contribution to the field. And their employers love having these experts and reward them appropriately because their excellence at creatively applying and innovating with the methodology advances institutional goals. 

    With that cultural substrate in place, a tool or service vendor can come in and say, “We help you solve X sort of problem in your Empirical Education process,” and the prospective customers will, understand what is being offered, be capable of evaluating its utility, and place (monetarily quantifiable) value on that utility. That’s what we need for learning analytics, adaptive learning, or just about any whizzy, trendy ed tech thingamabob you can think of or will be thought of.

    Most or all of the elements for a methodology of operational excellence in education exist in the world today. They need to be gathered, distilled, and refined into a learnable, repeatable, and adaptable practice. That is the outcome we aspire to achieve in collaboration with the participants in the EEP, not to mention support from the collective wisdom and will of higher education writ large.

    Moving forward

    As I wrote earlier, I will be blogging about relevant examples we are seeing, on both the institutional side and the vendor side, in the coming days. And EEP will soon be announcing the first release of some tools that can help form a foundational layer of the institutional infrastructure for Empirical Education. In the meantime, if you are going to be at the Online Learning Consortium Accelerate conference, I will be moderating an EEP-relevant panel discussion of SoTL on Thursday at 11:15 AM in Oceanic 1. From there, some of us will head to the exhibition hall, where we will have an EEP meet-up at the Soomo booth (#226) at 12:15 PM. You don’t have to be a member of the current EEP cohort to join us; the EEP-curious are welcome.