Just that first sentence shows how much meat there is yet to be put on the bones. “Contribution?” “Democratize learning science?” What does all that mean? These are fair questions. Jeff Young has some good reporting up at EdSurge that connects some dots on what’s public so far. I’m going to do my best to explain why the answers are not yet as clear as they might be, when and how they will be come clearer, and then give some reminders of past conversations here on e-Literate that provide some breadcrumbs leading in the direction of where all this is going. There will be more news in the press that will drop on Wednesday, followed by some more analysis by me on Thursday. From there, the story will build in pieces, through the Empirical Educator summit on May 6th and 7th, and afterward.
Let’s start with the big picture. Carnegie Mellon has been a pioneer in various types of theoretical and applied educational and cognitive science research—which is now fashionable to roll up into a ball and call “learning science”—for decades. While their work often doesn’t get the publicity it deserves, the breadth and depth is astounding. I’ve had the privilege of visiting a few times and taking deep dives. ((When I say deep, I mean deep. See, for example, https://mfeldstein.wpengine.com/can-there-be-microscope-of-mind/.)) There are very few institutions in the world that are in their league in terms of the scope of what they do.
The university made a decision to take a lot of their work product—over $100 million worth, in fact—and make it broadly accessible to the academic community. Honoring this commitment in a real and practical way is…hard. Very, very hard. It’s about more than just releasing the source code to software, which is hard enough to do right in and of itself. This is about making science—not just the output but the practice—accessible and useful to non-scientist educators. On top of that, the components of the contribution were not all designed together as a single software platform. They consist of many research projects, developed by different teams and that may be more loosely or tightly related to each other. Imagine if a top research laboratory decided to turn itself into the Smithsonian Institution, making itself a hands-on science museum accessible to everyone without dumbing itself down. That’s not exactly the goal that Carnegie Mellon’s Simon Initiative has set for itself, but its’ the same spirit. It’s incredibly ambitious and not easily imagined, much less described in a single press release.
In fact, doing this well is itself going to take some empirical experimentation. And that’s where the partnership with the Empirical Educator Project comes in. Carnegie Mellon had already offered to host this year’s summit on May 6th and 7th. I’m honestly not sure how long they’ve been mulling over this idea of the giant release, but at some point they came to the conclusion that we would be good pilot partners for their effort. This week, we will be describing our high-level approach to our pilot design for helping to make CMU’s contribution accessible. And not just CMU’s contribution, either, since EEP is based on the premise that many academic institutions have innovations to contribute if only we can get better at sharing them. The scale of CMU’s contribution has given us an opportunity to challenge our assumptions about how we should be going about this work. It’s a once-in-a-lifetime opportunity.
As I said, more details while emerge in the coming days, weeks, and months, first as we learn how to tell such a big story and later as we learn from the grand experiment as it unfolds. For now, I want to leave you with two video playlists. The first is a set of interviews of Carnegie Mellon faculty that I recorded while visiting a few years ago. At the time, I had no agenda other than to capture their individual and collective views on “learning science” and its relationship to classroom teaching. The second is a set of interviews from participants after the first day of the first EEP summit. Taken together, I think you will see why there is a good fit.
This is going to be a short post, in part because I’m traveling, but I need to call your attention to a developing story, both because it’s huge in its own right and for its importance to the Empirical Educator Project (which, by the way, has a new website).
Carnegie Mellon has announced a $100 million contribution in tools, software, and content that “that is intended to catalyze a new era of progress in educational effectiveness that is equal to the challenge of rapid change and growth in 21st century educational needs. The suite of tools is the product of over $100 million of research and development from a wide variety of funders.”
The suite of tools will be released in stages over the next year and represents a major departure from the “silver bullet” or “moon shot” efforts to revolutionize education with technology in recent years. Instead, the contribution is intended to democratize the science of learning and empower educators across the world to become citizen scientists. Carnegie Mellon’s goal is to provide knowledge of how to conduct applied educational research that classroom educators, researchers and educational technology companies can learn, apply, extend and share with the global educational community.
“We live in a moment when our educational institutions are in danger of a catastrophic failure that we cannot afford. College and university closures are becoming regular occurrences, even as tuition and student debt rise to record levels,” said Norman Bier, executive director of the Simon Initiative.
“This is happening at the same time that even highly educated people need to continue learning in order to keep their skills up-to-date, and when people all over the globe need increasing access to high quality educational opportunities through technology. In the face of these institutional and structural challenges, demonstrably improving outcomes and learning for students must be our foremost concern,” Bier said.
I have lots of good things to say about this approach, but for a preview, you can go back and review my post about ed tech hype being in remission. This announcement is a little hard to parse because it’s just a down payment on a complex story, and because it’s a big price tag thing from a big engineering school, but trust me: this is not the same old thing.
The full list of what is being released has yet to be announced, but I’ve seen it, and it is mind-blowing. The breadth and depth are pretty astonishing. In fact, one of Carnegie Mellon’s biggest challenges will be explaining all of what’s in it. This isn’t a tool or a platform. It’s a collection that’s in the process of being knitted together into an ecosystem. And the way that people inhabit that ecosystem is what will really matter.
There will be a lot more to say on that in the near future. The university is going to be revealing a lot of the details of their contribution at our second annual Empirical Educator Project summit, which they are graciously hosting on May 6th and 7th. We will have some announcements between now and then, likely a flurry of announcements (from us and from other parties) around the time of the summit itself, and will be releasing video of many of the talks after the summit afterward. There are also some reporters working this story, so I will keep running updates of those stories as they come out over at the new Empirical Educator Project site and periodically collect them in my updates here as well.
Carnegie Mellon University and Duke University have shared newly available free tools that will significantly lower the barriers to conducting ethical educational research. The two universities contributed the tools through e-Literate’s Empirical Educator Project (EEP), an effort to promote broader adoption of evidence-based teaching practices and foster a culture of empirical education across higher education.
As with all academic research involving human subjects, educational researchers must have their experimental designs approved by their university’s Institutional Review Board (IRB). If a researcher wants to study students or their work, they must explain how they will get the students’ informed consent to participate.
This can be a major barrier that often prevents research from being undertaken. Teaching faculty who may be interested in conducting a study may decide that the bureaucratic burden is more than they can take on. Multiple universities that want to collaborate on cross-institutional studies will have to get approval from each institution’s IRB in an environment where there are no widely adopted standards for reviewing and approving educational research by these bodies. Educational technology companies that want to be more transparent and collaborative with universities about their own research into product efficacy can find the IRB process impractically time-consuming. As a result, far less educational research gets conducted in ways that are both reviewed for ethical practices and shared as credible research that contributes to the state of the art in learning science.
Through e-Literate’s EEP, learning science researchers at Carnegie Mellon and Duke Universities discovered that each institution had developed a solution for part of this problem. Carnegie Mellon University has developed templates approved by their IRB that they estimate will accommodate approximately 80% of classroom research use cases. Meanwhile, Duke University has developed language and a process approved by their IRB for requesting and tracking informed consent from students.
The two universities have released the tools under a Creative Commons Attribution (CC-BY) license and provided “train the trainer” support for the use of their templates and protocols. Together, these contributions could enable many of the educators and product designers who are already conducting informal educational research all over the world to participate in the same sort of social fabric that has enabled communities of researchers in other human sciences to tackle problems from cancer to Alzheimer’s disease.
Jeff Young has a great piece up about this release at EdSurge, and I believe we will see something from The Chronicle in their teaching newsletter on Thursday. I’d like to give you my own take on the reasons why this an important milestone.
It’s an example of untapped inter-institutional opportunity
Carnegie Mellon and Duke are theoretically peers. To use one very imperfect measure, they are both ranked in the top 25 national universities by US News and World Report. They are both more specifically ranked in the top 15 such schools for undergraduate teaching. Both are seriously concerned with improving undergraduate education through research-based practices. The two institutions have been working on similar yet complementary efforts to make that research easier. You would think that they would have had opportunities to share their work with each other, or at least know about what the other is doing.
Before EEP, they didn’t.
On closer examination, the complementarity of the two efforts suggests the kinds of opportunities that higher education is missing. Carnegie Mellon University has one of the broadest, deepest, most impressive, and most historic learning science research programs on the planet. There are only a handful of universities that are even in their league, and Stanford may be the only university that rivals them in depth and breadth. Further, the university has made a commitment in the form of the Simon Initiative to “[p]rovide accessible tools and methods with which any person or institution can adopt and advance CMU’s approach to learning engineering, improving outcomes for their own learners,” globally, in addition to improving teaching and learning at Carnegie Mellon University itself. (Not that this is distinct from but complementary to their Eberly Center for Teaching Excellence, which is similar in purpose to the centers of teaching and learning at many universities.) To get a flavor for who they are and how they think, here’s a playlist of three e-Literate TV videos that highlight a few of their faculty members:
Now, Carnegie Mellon is known more generally for its engineering prowess, so it’s no surprise that the folks at the Simon Initiative talk about “learning engineering.” (The initiative is named after the late Herb Simon, a cognitive scientist and CMU luminary who coined that term.) That mentality, as well as the orientation of an institution known for its learning science research, shows in their contribution. They studied the IRB applications submitted for educational research by their faculty—not just their learning science faculty, but all faculty interested in publishing their research projects, identified common traits, and developed a template that they estimate covers somewhere in the neighborhood 80% of those projects. They then had their IRB review and approve the template. Now, any educator at CMU who wants to do publishable educational research and can use the template gets their IRB application fast-tracked.
Without taking anything away from Duke’s own learning science research capabilities, one of the things that Duke is among the very best in the world at is making their undergraduates’ educational experience life-changing. As one example, the university has a large number of “professors of practice.” In many big research universities, a tenure track is a kind of death trap. Faculty are worked to the bone for three to five years in the hopes of achieving tenure when, in reality, most of them will be sent packing in the end. And while they are on that treadmill, they have a disincentive to invest too heavily in their teaching lest they neglect the publications and grants that will increase their odds of not being shown the door. That’s not how Duke rolls. Rather than exploiting young faculty by dangling a carrot that will forever be out of reach, they offer many a professor of practice position. Such faculty get basically everything except tenure, including long-term employment, a decent salary scale, and a say in shared governance. But they must not only show that they are excellent educators but also conduct research in education. In effect, rather than “physics” or “art history” being their discipline, it’s “physics education” or “art history education.” I don’t know whether Duke or their faculty would put it quite this way, but I intend it to be a compliment. The point is that they incentivize faculty to become disciplinary experts in supporting their students through evidence-based practices. I have been to one of their teaching and learning conferences and interacted with these faculty members. It was heavily attended and the atmosphere was electric. I have been to many of these kinds of events; yet I have rarely seen faculty who were more actively engaged in asking good, probing questions about teaching practice than I did at Duke.
This different context from CMU’s has resulted in a slightly different approach to the same problem. Like everyone else, Duke’s educators who want to publish their research have to go through IRB approval. And given that Duke has a world-class medical research program, their IRB is both very tough and very focused on privacy concerns. So their Learning Innovation center developed an informed consent tool called WALTer, where WALT stands for “we are learning too.” WALTer sits right inside the LMS for any course in which research is being conducted. Faculty who want to conduct such research are led through a decision tree that produces IRB-approved language for informed consent, based on the conditions of the experiment. Having this form in place helps the instructor get faster approval of the project. And once that approval is in place, WALTer helps ensure that students are given the opportunity to provide their informed consent (or not). Since WALTer has been launched, the Learning Innovation team has seen an increase in the number of faculty expressing interest in conducting educational research and a decrease in the time it takes educational research applications to be approved by Duke’s IRB.
Peanut butter, meet chocolate. CMU’s IRB template and Duke’s informed consent template are completely complementary. But before EEP, the peanut butter was in the fridge and the chocolate was in the drawer. They just didn’t meet.
It’s an example of untapped intra-institutional opportunity
The Duke feedback that they are getting more educational research interest and getting that research approved faster is illustrative of a larger point. The current institutional structures and processes of colleges and universities are not designed to facilitate the development, testing, sharing, and adoption of evidence-backed practices which improve student success. The IRB process is just one (very painful) example. The disincentive for faculty on the tenure treadmill is another. There are many more.
The question that I have heard asked ad nauseam for years and years now, from a wide range of people, is “How can we make faculty care more about teaching?” That is the wrong question on every level. A better question would be, “How can we design universities such that focusing on improving excellence at supporting student success is less painful and more rewarding?”
Think about just about every hot ed tech-related trend you can think of. Retention early warning analytics. Adaptive learning. Competency-based education. Stackable credentials. MOOCs. All this activity (and money) is swirling around the problem of making it easier for students to learn and succeed at school. Now think about all the hot trends that focus on making it practical and rewarding for faculty to focus their energy and considerable intellectual talents on solving the same problem. Can you name one?
An IRB form for educational research may sound like a small and boring thing, but it is a piece of cultural and institutional infrastructure that makes it a little more practical for motivated faculty to focus energy and intellectual talent on learning how to better support student success. “We Are Learning Too” indeed. This is the kind of work that will make college education better and, in the process, make all of those ed tech tools more useful. A laser scalpel doesn’t do much to promote health unless it’s wielded by a physician who knows how, when, and why to use it. ((Is a “laser scalpel” a thing? I may have just made that up. Anyway, you get the point.)) Without that knowledge, it risks doing more harm than good.
It’s an example of untapped multi-institutional and institution/vendor collaboration opportunity
So yay for Carnegie Mellon and Duke. What about everyone else?
Well, for starters, they have both contributed their language under a Creative Commons license. (Duke’s WALTer tool is built using a third-party proprietary platform, so contributing the source code wasn’t an option. But somebody else could easily build a tool that supports the workflow in the release document.) So there’s that.
IRBs are notoriously idiosyncratic. (Some might say arbitrary. I’m not saying that. But some might.) So you could see an IRB at an institution that’s very different from CMU or Duke having something like the following argument:
IRB Member #1: Hey, CMU and Duke are super-rigorous research institutions that are way more focused on these sorts of things than we are. If it’s good enough for them, it should be good enough for us.
IRB Member #2: Actually, exactly because CMU and Duke are super-rigorous research institutions that are way more focused on these sorts of things than we are, what makes you think that what makes sense for them will also make sense for us?
These are both reasonable starting positions. The conversation that should flow from this is an examination of the templates in which an IRB member who wants to change a part of it will need to provide a justification for doing so. This is exactly the next step we want to foster for this project, preferably at multiple institutions. Where we’d like to end up is at a toolkit in which an institution of any type can look at variations—and justifications for those variations—provided by peer institutions so that they adapt and adopt them. We’d also like to supplement what we already have with some more fleshed out student data privacy guidelines that are, once again, education-appropriate. Data privacy in this sort of research is just as important as it is in, say, medical research. But the specific concerns and methods for dealing with them aren’t necessarily identical. We should be developing a sector-wide consensus on ethical practices that can augment what is already in the Duke and CMU IRB contributions.
This would hopefully help to lower one barrier to conducting educational research everywhere. But it could potentially do much more than that. Under the law, if researchers want to conduct a multi-institutional research project, then each institution’s IRB must approve the project. And since no two IRBs use the same standards and most don’t have particular guidelines for educational research, doing research across two institutions is more than twice as hard. Three is more than three times as hard. Doing large-scale, multi-institution research quickly becomes impossible in most scenarios. Unless you’re a vendor, in which case it is trivial, because you are not required to go through IRB for your own research—as long as you don’t publish it in a journal. So vendors can conduct research on students in multiple institutions for their own proprietary purposes easily, but if they want to do the right thing by going through IRB approval and sharing what they’re learning through peer-reviewed journals, it’s way, way harder.
Imagine if it were easier for everyone to do the right thing and submit an application for IRB approval, knowing that they will be going through a streamlined but academically validated process. Imagine the kind of research opportunities that could open up. Now imagine further if there were some technology infrastructure behind this. Imagine if we could track IRB approval and informed consent across institutions, gather appropriately anonymized student data, and share it among researchers in a repository that is designed to respect the student privacy constraints dictated by the approved IRB applications while giving more researchers access to more research-relevant data—including data that are gathered through student interactions inside vendor tools. Technologically, this is quite practical. The hard part is getting the policy infrastructure solid and widely adopted.
This is what EEP is about. The problem isn’t that higher education is failing to innovate or that professors don’t care about teaching well. The problem is that we are flushing 99% of the existing efforts, potential opportunities, and good intentions down the toilet because we don’t have the cultural institutions and social infrastructure to support and sustain them. But that can change.
Kudos and thanks to the good folks at CMU and Duke. We have more folks working in EEP—from a diverse range of institutions—on a wide range of projects. And we’re still learning how to work together effectively. Expect more to come.
If you’re a regular e-Literate reader, you know we have a macro thesis that the higher education sector is in the early stages of an evolution from having a philosophical commitment to student success toward having an operational commitment to student success. In other words, colleges and universities are starting to approach student success systematically, not as the natural by-product of hiring good faculty but as something that every student-facing aspect of the institution needs to be optimized for.
There is no road map for making this transformation and a number of formidable obstacles to it. First, academia was simply never designed for this purpose. The civilizational goal of empowering every human to live up to her or his potential via access to higher education is very new. Much newer than higher education system itself. In fact, it’s almost a thousand years newer. The University of Bologna in Italy, which is the world’s oldest university in continuous operation, was founded in 1088. The Morrill Land Grant Act, which created the first public universities in the United States, was passed in 1862. The G.I. Bill passed in 1944. Pell grants were created as part of the Higher Education Act in 1965. In 2018, achieving the as yet unrealized ambition of access to higher education regardless of income is very much a live political discussion. Just this month, the Sacramento Bee reported on a poll showing that 58% of Californians view college affordability as “a big problem,” with another 25% saying it is “somewhat of a problem.”
Even newer is the idea that we should not only be giving universal access to higher education but also taking responsibility to ensure that, once students have access, the institution is maximizing their chances of success (as opposed, for example, to the much older and still much more common idea of elitist “weeder” programs that filter for the “best” by failing out most). The deep structure of academia, from its governance to its professional training to its funding structure to its culture, is the evolutionary product of serving different missions than the one which we are now asking it to serve.
Second, even if we agree to embrace the mission of universal access and affirmative responsibility for student success in higher education writ large, how that plays out is very different at, say, Stanford, Loyola Marymount, UC Berkeley, Cal State Chico, and Los Angeles City College. The requirements for access are different. The definitions of and requirements for success are different.
And then their are the students, each of whom comes with her own definition of success, life goals, strengths, needs, and life context.
This is a hard problem. One that drives a lot of our work and our thinking. In a series of posts, I’m going to try to lay out what that shift looks like in a variety of academic contexts, how a successful shift across the sector would impact the future various ed tech product categories, and how the Empirical Educator Project (EEP) is intended to foster a methodology for empowering that shift.
In this first case study, I decided to start with the California Community Colleges Online Education Initiative (OEI). ((Disclosure: CCC OEI is a consulting client of ours.)) In fact, much of the structure of this post is drawn from an analysis we wrote on their behalf for the California State Legislature. I’m interested in extracting some generalizable lessons from OEI’s design. It’s important to be clear that the story I’m telling here is compatible with but not quite the same as OEI’s official position as represented in the report that they submitted to the legislature. OEI is also an interesting place to start this post series because, as we will see, it operates in an extreme environment that makes it particularly instructive.
This is a story about the whole being greater than the sum of its parts. OEI has put together a number of pieces that other institutions also have put in place, either individually or in various combinations. But they have done so with larger strategic vision for the future of California Community Colleges firmly and consistently in mind. It is also the story of a work in progress. California OEI has some impressive early successes under its belt. But it is also a hugely ambitious effort with much still to achieve. (Its in-process merger and rebranding with California Virtual Campus (CVC) is one example of forward-looking plans that I will touch on later in this post.)
The California Community Colleges is the largest system of higher education in the nation, with 2.1 million students attending 115 colleges. Our colleges provide students with the knowledge and background necessary to compete in today’s economy. With a wide range of educational offerings, the colleges provide workforce training, basic courses in English and math, certificate and degree programs and preparation for transfer to four-year institutions.
That’s a lot of students and a lot of colleges. One more college than last year, in fact. The legislature just approved the creation of a 115th campus (which will be virtual). California Community Colleges cover a lot of ground—literally as well as metaphorically. If you were to drive from College of the Siskiyous, which is about an hour south of the Oregon border, to Imperial Valley College which is about 20 minutes from the Mexico border, you would have to travel over 820 miles. They serve the top 100% of students. A lot of ground indeed.
There’s one word you won’t find in that rather dramatic description of California Community Colleges: “system.” Many state college and university systems are pretty big on local control, but California Community Colleges takes that principle to an extreme. For example, despite being a program intended to serve the entire system, OEI is run out of the Foothill-De Anza Community College District (after winning a competitive grant) because the Chancellor’s Office of California Community Colleges is more or less forbidden the legislature from running it centrally. California’s legislators are fiercely protective of the autonomy of their home districts. And as far as I know, OEI has negligible power to compel campuses to do anything.
In that environment, how do you help the entire 2.1-million-student, 115-campus, 820-mile-long “system” move together toward better operational excellence in enabling student success?
Academics tend to bristle at terms like “business drivers” and “business processes” when applied to academia, and there are good reasons to be cautious about using them. I’m applying the terms narrowly here because terms like “sustainability” are less effective at focusing people’s thinking about the machinery of balancing budgets. At the end of the day, colleges and universities need to take in as much money as they spend in order to keep fulfilling their mission. If you want to think clearly about how the sustainability machine works, then business is not a terrible metaphor. People have a basic, intuitive sense of what kind of machine a business is. The same sort of machinery is obscured the moment you start using words like “university” or even “institution” (or “sustainability”).
What do businesses (or sustainability machines) need? Money. There are a number of ways to have more money. One is to spend less of it. So one of OEI’s first moves was to offer to pay for the campus’ LMS, thus relieving each campus of the need to spend that money. LMS licensing may be an insignificant expense for an R1 university with a big endowment, but for a community college, it matters. There is no wiggle room in the budget. Hard choices have to be made—choices that impact student access and student success. We found anecdotal evidence that campuses have been using the money freed up by OEI’s LMS subsidy to invest in student success.
Our courses are much improved. In one year we have had 75% of current online instructors are fully certified. Almost 80 additional faculty are in process of being certified. We have approved 46 online course sections and reviewed or are currently review this semester another 35-40 courses. Without the resources from OEI and @one, we could not have made this happen.
– Faculty Senate Curriculum Chair, College of the Desert
Funding that would be used for [the common course management system] can be redirected to training for faculty who need extra help learning HOW to teach online.
– Dean, Business, Technology, and Career Technical Education, Ohlone College
Part of the people/resources that Coastline was able to shift, include our new Faculty Success Center, whose staff was able to create a new online course template in Canvas that helps faculty design a quality course. In addition, we were able to devote trainers to help faculty learn to use [the common course management system]. In this environment, our Academic Senate then felt comfortable mandating training for online instructors, something we never had before. I believe all this would not have happened if we had to pay for the [the common course management system] license….
So, continued state/OEI support for the…license will be critical for us to continue to train/support faculty and disseminate the use of these [OEI support] apps and support services….
One thing we were able to do, due to the free license, is pay all District faculty a stipend for the completion of [course management system] training.
– Associate Dean, Distance Learning, Coastline Community College
So just saving the campuses more money, by itself, led to actions by at least some campuses invested in improving their operational excellence at enabling student success. But that was really just the beginning. First, to get the subsidy, the campuses had to agree to do certain things. One of which was to adopt the same LMS. There were reasons for this, which I’ll get to shortly. For now, consider the likelihood of getting that many resource-strapped community colleges to migrate LMSs. How hard would it be? How long would it take?
All 114 campuses signed the contract agreeing to move to the common LMS, and many moved quickly to implement. In fact, the migration proceeded so far ahead of schedule that OEI had to go back to the legislature and request additional funding to cover the unanticipated extra subsidies. Saving these campuses money was a powerful motivator. And, as we’ll see, OEI accomplished a lot more than meets the eye with this one seemingly prosaic move of subsidizing an important but work-a-day piece of enterprise software.
How else can businesses make more money? By doing a good job of aligning their investments with their business opportunities. A grocery store doesn’t want to overstock with produce that will spoil on the shelves. But it also doesn’t want to run out of that produce when there’s high demand. And demand is variable. Demand for produce the week before Thanksgiving is likely to be different than the week after.
Colleges have an inventory management problem too. Sometimes courses are under-enrolled; other times they are over-enrolled. Both represent money problems to the campuses. One of the reasons that OEI wanted all the colleges on the same LMS—not just the same brand, but the same instance—was to create a course exchange. Balancing course “inventory” in a single community college is tough. Room availability, instructor availability, changes in the job market and economy, and the unpredictability of part-time student enrollments all work against you. But balancing “inventory” across 114 community colleges is less hard (once you can figure out how to get it to work in the first place). One campus may be over-enrolled in macroeconomics, but chances are pretty good that one of the 113 other campuses is under-enrolled in the same course. If you can get enough campuses to put enough courses on the online course exchange, then you can solve a “business” problem for all of the campuses. And the more courses there are on the exchange, the more valuable it becomes to the campuses. Thus, colleges have incentives to create courses for the exchange, and the more courses that are created, the more incentive the colleges have to utilize the exchange.
You could tell this same story from a student access perspective. Over-enrolled courses prevent students from taking them in a timely way. If the course is required, this could force them to delay graduation (and a full-time or better paying job), take on additional unneeded courses in order to qualify for financial aid, take extra financial aid from the state and federal governments, take up an enrollment space that might have gone to other students, and increase the risk that they will not graduate. Under-enrolled courses risk cancellation, with many of the same knock-on effects. I don’t mean to neglect or downplay this portion of the story.
But the focus on business incentives lets us think more clearly about the machinery of the institution itself. Which, in turn, helps us to think clearly about how that machine works and how it can be tuned. OEI designed a machine to drive operational excellence at enabling student success across the largest community college system in the country. And it runs on only positive incentives because. This design constraint immediately rules out copying some of the most frequently cited examples of innovative universities which, through one mechanism or another, can exert varying degrees of top-down control. At ASU, President Michael Crow has an unusually strong hand to play within a reasonably traditional structure of faculty shared governance. (Ithaka S+R has some interesting and revealing interviews of some of ASU’s top leaders that give some hints about how that governance works.) Western Governors University is more extreme; there is no faculty senate and no shared governance. SNHU’s Paul LeBlanc has tried a combination of strategies, working with with the faculty senate on governance of the traditional college while separating out their College of Online and Continuing Education (COCE) and running it in a way that is only loosely coupled to the shared governance of the rest of the university. Like many universities and systems, OEI cannot redesign the machine from the top down. so thinking about the live-or-die campus sustainability incentives that could be used to drive collective action has been a central principle that influenced the rest of OEI’s design.
Creating the infrastructure
The desire to move all campuses in the system to one LMS wasn’t just for the sake of contracting convenience. It accomplished a variety of goals. First, it became a foundational layer of software infrastructure for rolling out other system-wide capabilities and services, from plagiarism detection to online tutoring to faculty training and help resources. Having everybody on the same instance of the same platform—cloud-hosted Instructure Canvas—made it much easier to do this. In the old world, where campuses were on a hodgepodge of different self-hosted and vendor-hosted LMSs, the best the system could have accomplished would have been common contracting. It would still be up to each campus to integrate and support the tools and services. After all, the way a tool looks and works in Moodle can be different than in Brightspace. Centralized support would have been a nightmare. And remember, these campuses are very tight on resources. Supporting add-on tools and services is costly to them.
In addition, sharing one LMS made it easier for OEI to create faculty training that could be shared across the system, and for campuses to do the same. As with system-wide licensing for LMS-connected tools and services, it’s not impossible to do this in a system with different LMSs. But the added friction makes it less likely to happen. I’m going to use the “B” word again: academia needs to think about business processes. Once again, stripping away the culture- and mission-inflected language lets us see the machinery more clearly. A business process is the way in which a business accomplishes something that is important for the business. For example, how does a business make sure that all its employees have up-to-date software, including critical ones like system updates and the latest anti-virus software? Sure, they could leave that to the individual employees to do. We’ve all updated our software on our personal computers; it can be done. But how likely is it that everyone will do so in a fashion that is timely, reliable, and consistently correct? And what work are all those employees not getting done while they are wrestling with software updates? It’s better to develop a business process for pushing out those updates from a central IT group so that employees don’t have to worry about them. Likewise, there are effiency benefits to centrally rolling out and support services for 115 campuses than to have each campus IT support person duplicate the effort. From a perspective of strengthening the business drivers that hold the group together, all of these benefits can be boiled down to saving money by providing additional capabilities with reduced cost to on-campus resources (in direct licensing fees, support staff time, or both). As we have already seen, the campuses tend to invest the money they’ve saved in enhancements that are specific to their local needs and that benefit their students.
The common LMS also helps with the over- and under-enrollment problem. Having all course exchange courses on a single instance of a common LMS made it easier both to provide more data to the campuses that would help them with their planning and to reduce friction in expanding the course exchange. If everybody is using the same system, that’s one less thing for faculty and students to learn, less help desk support, and more productive support (for both course delivery and course design) because the OEI staff don’t have to try to accommodate multiple flavors of learning environments.
Of course, there are trade-offs, the biggest one being autonomy. In OEI’s case, for example, all the campuses had to agree to use the same LMS rather than choosing their own. Anyone who has run a campus LMS selection process knows it can be an exercise in delicate diplomacy. Imagine doing the same with 114 campuses. As we’ll see, OEI turned this challenge into an opportunity. I’ll have more to say about that in the next section.
Anyway, once you start seeing infrastructure as the structure “underneath” (i.e., “infra-“) that supports business processes, two things immediately start to happen. First, your definition of success changes. You can no longer declare victory just because you successfully installed the software and got people to start using it. You have to start looking at whether it successfully enabled or improved the business processes you were intending to support.
Our Professional Development Coordinator is encouraging the creation of Pro Dev workshops in [common course management system] Canvas, such as health and wellness (“dealing with difficult people”), how to create Open Educational Resources, how to use [Student Learning Outcomes] for better teaching, and a lecture on science and its assumptions. What is developed at Butte can be instantly shared with other schools, and vice versa. I see a renaissance of Pro Dev opportunities!
– Technology Mediated Instruction coordinator, Butte College
The second thing that happens when you start thinking in terms of business processes is you identify new problems as well as rethinking and reprioritizing old ones. For example, OEI is in the process of revamping (and merging brands with) California Virtual Campus (CVC). Why? CVC exists today. It’s a web-based catalog of online courses students can across the California Community Colleges and California State University System, which is many more than the handful of OEI exchange courses. As of 2017, CVC included 23,445 online courses and 1,376 degree programs. So it’s big. If you think of infrastructure as a thing to have, then you might think of CVC as a massive success.
But if you think about CVC as the structure underneath that supports the critical business process of students finding, (wisely) selecting, and registering for online courses across the many campuses represented in the CVC course catalog, then you start to develop different metrics for success. And if you also think about the back-end process of making sure the right institutions get the registration information, tuition, and transcript information (respectively), then CVC becomes both a critical priority and a tough piece of infrastructure to build well, particularly across so many different campuses that are not all migrated to a single instance of common Student Information System (SIS) software. If students fail to register for online courses that they need because the process is too cumbersome, or if they don’t get properly credited by their home institutions after taking an exchange course, that is bad for the long-term health of both the students and their institutions. This is what it really means to say that some infrastructure is “mission-critical.” CVC is a big catalog with lots of courses, but it does not yet do a great job of fulfilling its mission-critical role of helping students find, register for, get credit for, and pay for their courses. Each of those is a business process that CVC should support. And the number of courses in the catalog tells us very little about how well the software is supporting those processes for the students and the campuses.
If you think about infrastructure in this way, then your communications to your stakeholders will also change. Here’s an explainer video they had us create for them in order to help communicate that message to the various campus folks:
(Source video: https://youtu.be/1DdlaIZYiDI)
Why was this so important to communicate? OEI could have gone to their constituents with a message of “Here’s a bunch of great free stuff for you!” Instead, they chose a much more challenging message to communicate; one about the ripple effects of having shared infrastructure. That wasn’t an obvious choice.
If you’ve read any one of a million articles on how to succeed with any major campus-wide initiative, you will have read the cliché about how important it is to “get buy-in.” Most of the time, “getting buy-in” is interpreted as “handling objections” or “reducing resistance.” It is an obstacle to get past. But the video above shows that the OEI leadership has interpreted the term differently. You only communicate to your stakeholders in this way if you believe that buy-in is infrastructure.
Fostering a culture
In our consulting work, we facilitate LMS selection processes reasonably often. The more forward-thinking institutions view these processes as opportunities. How often do you get to gather a group of faculty and other academic stakeholders from across your institution in one room and have them talk to each other about how they teach and what they need to serve their students well? A search for a product like an LMS can become a rare opportunity for focused, intensive, purpose-driven community-building. Yes, it lowers resistance, enabling people who might not be happy with the final decision to at least feel like they were heard. But it also begins to foster familiarity and dialog that can help foster a broader and more lasting community of purpose. When you’re trying to build such a community across 114 campuses as part of an ambitious and completely voluntary system-wide effort, taking that view of LMS selection is even more important. Needless to say, it wasn’t easy. Or seamless. That said, both during the selection process and and afterward when communicating the results, OEI worked toward building affirmative buy-in—not just lowering resistance but increasing the sense of goodwill and common purpose. For example, the selection committee was near unanimous in its selection, with the sole dissenter acknowledging the importance of what the committee was doing together and supporting the final decision.
The program design elements I’ve described so far helped OEI to foster increased organizational alignment at two levels across campuses. The campus executives who are responsible for financial health and sustainability of their campuses are aligned through infrastructure subsidization and the course exchange. In 2018, the state legislature decided to augment the initiative with an additional $35 million funding. California Community Colleges has chosen to invest that extra money capacity-building. In particular, the money will go toward grant programs intended to enable the campuses to launch more online courses on the OEI-CVC infrastructure, thus further strengthening this alignment while aiming to serve more students effectively. The common infrastructure and the culture-building process around it has helped to build a culture among the academic and technical support staff across campuses. The hoped-for consequence is that improvements on one campus will travel more quickly and easily to others:
I now know that I can ring up any other [Distance Education] Coordinator and we’ll be speaking the same “language” regarding the use, training, and administration of the [course management system]. I’m also really looking forward to faculty being able to share ideas and resources via Commons.
– Director of Distance Education, Santa Rosa Junior College
Building a similar sort of sharing network among the faculty in the system is an even larger challenge. The culture-building work has been ongoing work for years now, but one can acknowledge all the hard work and progress to-date while still also recognizing that this is a huge project that has barely begun. Certainly, having common resources and common platform supported by OEI has provided a boost, as has the inclusion of faculty voices in the OEI planning process. The augmentation grants, which will likely include instructional design support, represent another opportunity. But to me, one of the most interesting vectors for culture-building is the course exchange course quality rubric. Every course on the exchange has to be evaluated against a rubric of evidence-backed effective online teaching practices. As the pace at which exchange courses are developed increases, OEI will not be able to keep up with demand to evaluate these courses using central staff. So they are creating a peer reviewer mechanism in which faculty on the campuses are trained on the rubric and presumably compensated to review courses that are candidates for the exchange.
This opportunity fascinates me. We know that faculty who go through an expert-supported course redesign process often experience intellectually deep and emotionally moving shifts in their teaching strategies. Is the same true when faculty are trained reviewers of their colleagues’ redesigned courses? What effect will simply exposing faculty to more and different course designs have? How will their role as reviewers and critiquers shape or enhance that effect? Can a continuously improved and updated rubric become a vector for sharing new research-supported processes across the system on an ongoing basis? Will the impact be broad and deep enough to foster new kinds of intra- and inter-campus faculty dialogs about the scholarship of teaching and learning (SoTL)? Will these cultural changes help to foster alignment around continuous operational improvement for enabling student success? This is the last mile problem of higher education. Operational excellence at student success cannot be achieved unless it is infused in the daily operations in individual classrooms. That requires affirmative faculty buy-in, support, training, and embedding in a culture that invites them into the larger conversation.
This is highly reminiscent of the cultural transition that doctors had to make from the mid-Nineteenth through the mid-Twentieth Century. In the 1840s, one could begin practicing as a physician with no medical training at all, just as one can start practicing as professor with no pedagogical training today. Doctors learned medicine from whomever they happened to train with and whatever they read in the newspaper ads about cures and treatments ranging from early antiseptics to leeches and literal snake oil, with no easy way to distinguish them. There were no major conferences or respected, peer-reviewed journals. There were no standards for quality research or convincing evidence. There were a handful of teaching hospitals and medical colleges of wildly varying quality that touched only a small minority of practicing physicians. All of these institutions, all of this social infrastructure, needed to be built and bought into by physicians before antiseptics could be differentiated from snake oil, the signal separated from the noise, regarding “progress” or “innovations” that might help their patients’ welfare.
OEI has accomplished some remarkable early successes in an extremely challenging context. But the degree to which they are able to move 114 California community colleges toward better support of student success as a group may well depend on the ability of the social infrastructure they are creating to reach faculty, be embraced by them, and and foster a culture in which academics collaborate differently and more intensively in their day-to-day work of helping students to succeed, one student at a time.
It is almost exactly the six-month anniversary of the Empirical Educator Project’s (EEP’s) first annual summit at Stanford University. The thesis behind EEP is simple: We believe that one of the biggest barriers to increasing student access and success in US higher education is a failure to communicate. We see substantial innovation and progress happening across the sector, from individual classrooms to entire university systems, from access-oriented institutions to research universities to vendors of all shapes and sizes. But for a variety of reasons, the lessons being learned rarely travel far. We believed that if we could just get the right people talking to each other and sharing their work with each other, then good things would happen.
That’s it. It was that simple.
Six months ago, we tried it for the first time with about 50 hand-picked people for one and a half days. Most people there knew less than half the people in the room. That was one of our goals; we wanted to get people talking across the normal boundaries of their peer (or customer, or competitor) groups.
Early indications are that we succeeded beyond our wildest hopes. The group generated somewhere between 15 and 20 project collaboration ideas, many of which have been active even over the summer months. We haven’t written much about the ongoing work because we’ve been waiting for the participants to feel they have made enough progress on their respective projects that they are ready to talk about them publicly.
Today, I am happy to tell you about the first EEP collaboration that has crossed that threshold. Here’s the text of an announcement from James Madison University’s (JMU’s) department of Psychology:
Since that time, JMU faculty member Dr. Jeanne Horst traveled to CMU to engage in training offered by the Eberly Center for Teaching Excellence & Educational Innovation in June 2018. During the 2.5 day Teaching As Research Institute, Dr. Horst engaged in conversation about evidence-based teaching and learning practices. She and the CMU professionals shared experiences regarding the process of developing and implementing intentional curriculum and instruction in higher education and assessing its effectiveness.
In July 2018, JMU hosted our CMU colleagues Mike Melville and Elizabeth Whiteman for training offered during JMU’s week-long national award-winning Assessment 101. During this training, JMU faculty and graduate students offer practice-orientated materials and build skills regarding student learning outcomes assessment.
In August 2018, Dr. Marsha Lovett traveled to JMU to observe JMU’s large-scale Assessment Day. In addition to learning about how to plan, proctor, and engage students during institutional accountability testing, Dr. Lovett also observed a focus group to elicit students’ perceptions of general education courses. She also met with JMU faculty and graduate students to plan the next steps in our strategic partnership. JMU faculty look forward to sharing our expertise in assessment and measurement with CMU faculty while gaining additional knowledge and skills associated with creating and implementing evidenced-based programming from the experts (and our friends) at CMU.
JMU and CMU are about half a day’s drive from each other. They are both (underrated) top-tier universities. Each has particular areas of strength in applied learning science research that complements the other’s. And yet, they didn’t know about each other until they met at EEP. That one meeting was all it took to trigger three rather substantial reciprocal campus visits—one in June, one in July, and one in August—to learn from each other and to develop what JMU is now publicly characterizing as a “strategic partnership.”
The EEP network has quite a few other such pots simmering on the stove that you’ll be hearing more about in the coming months. Not all ideas for joint efforts will bear fruit, and not all EEP participants are engaged in successfully progressing projects yet. We at e-Literate are still learning how to reduce the friction that inhibits successful collaboration. But we are off to a fantastic start, thanks in large part to the amazing folks in our first EEP cohort.
Here’s an e-Literate TV video of several CMU Simon Initiative and Eberly Center faculty talking about their view of learning science:
And here’s JMU’s Sara Finney talking about their approach to course and program effectiveness assessment in her lightning talk at the EEP summit:
Peanut butter, meet chocolate.
Stay tuned. We’ll have more good stories to share soon.
In my last post, I wrote that the Empirical Educator Project (EEP) is e-Literate’s way of putting our money where our mouth is, so to speak, regarding our own theory of change for improving student success and closing achievement gaps. What is that theory? I articulated it in some detail last February, before our inaugural summit.
But it was just a theory then. After one day at the summit, we asked participants to volunteer their own views about what EEP is and why it might be valuable. We created a four-part interview series, which can be found here. But I want to highlight a few points from the interviews that have convinced us that we are onto something.
Let me start by articulating that theory as simply as I can:
Many, many educators are already working hard on meeting student success challenges, but they are often doing it in isolation. If we can introduce the right people to each other and get them talking to each other about the student success challenges they are working on, then good stuff will happen.
That’s it. That’s the whole enchilada.
Here it is in a little more detail:
Many educators at many institutions—including both universities and the companies that serve them—feel personally responsible for improving student success and are actively testing ideas and learning lessons about how to do so.
Social barriers often prevent these lessons learned from being shared, tested, and built upon.
One of the biggest barriers is the difficulty of finding collaborators who are motivated to work on the same problem and have complementary capabilities for doing so.
One major reason that potential collaborators don’t find each other is because they live in different peer networks and don’t naturally cross paths.
If we can make it easier for those people to find each other, then they will do great things together, even without a lot of external funding or organizational support.
Once people from different peer networks collaborate on a project, the results will tend to propagate in all of the participants’ respective networks.
If we do this enough, then lasting bridges will begin to form among currently isolated networks and the pace of sector-wide progress will accelerate.
Here’s how we are testing the idea in EEP’s first year:
We made a list of people across the higher education sector who we know are doing great work and are good collaborators.
We trimmed the list down to a manageably small group that was spread across different peer networks. (Most of the summit participants told us they knew fewer than half of the other attendees.)
We raised enough money to get the group into the same room for a day and a half, and to provide some light follow-up.
We spent our time together at the summit encouraging folks to get to know each other, share what they are working on, and look for project collaborators.
Post-summit, we are watching to see whether collaborative projects happen and are lightly facilitating where we can.
From the beginning, we viewed this very much as an experiment and honestly didn’t know how much we would be able to accomplish with a group of strangers in less than two days. We were absolutely blown away by what happened.
Do yourself a favor. Take a little under eight minutes and listen to what these participants had to say about our theory of change after just one day of talking to each other:
It’s fairly easy to get a group of smart people together in a room and have them walk away feeling like they had a good, satisfying intellectual conversation about what might or could be done. It’s a lot harder to get them excited about a common, complex goal, with grand ambitions and a sense of how those ambitions might be met. If you didn’t watch the whole video above, then just listen to this bit from Duke University’s Bridgette Martin Hard about what she described as “one of the biggest, loftiest goals that I see being possible [for EEP]:
Culture change. For the entire sector. There it is. The whole enchilada. After one day of discussion.
OK, discussion, is nice, but is anything actually going to happen? Well, here is a sampling of participants talking about some of the project ideas that emerged from the summit—again, after just one day of being together:
Every person in that room said they had at least one collaborative project idea they had seriously discussed and thought had a realistic chance of actively happening. Some had as many as three projects.
That was after one day. What about now, three months later? I am in the process of compiling the first project update for the group. (We’ll keep these updates private to the group until each group of collaborators decides they have made enough collective progress to share their work more broadly. So far, I have about a dozen substantive updates from project teams, and I know for a fact that I have not received updates from all the groups that have engaged in post-summit work on their projects. This all sprung out of one brief gathering with about fifty people who largely didn’t know each other and have decided to move ahead with collaborative work despite the fact that they are getting zero grant money for the projects. The problem isn’t lack of motivation for change; it’s the barriers. In a variety of ways that I’ll explore in future posts, the match-making that is happening in EEP lowers various barriers—yes, the social barrier I talked about up front, but resource barriers as well. (There are hints of this in the videos above.)
There is no longer any question for us about whether we will continue EEP after this first year. We believe that some of these projects will be ready to come out of stealth mode by the fall, with more going public by the end of the calendar year. We are slowly and carefully adding new members to the cohort and expect to expand it more substantially in the spring of 2019, when we have our next summit. We are also exploring ideas for broader outreach, along the lines of Bridgette’s thinking above. We’ll have more to say about this in the fall.
In my last post, I wrote about the goal of the Empirical Educator Project (EEP) as getting a critical mass of educators to be “socially empirical”:
Socially empirical educators view effective teaching not as an individual art but as a shared pool of knowledge and experience that everyone can learn from and contribute to. They seek out common vocabulary, methods, and standards of proof so that they can learn with their colleagues and raise the collective bar. This is the beginning of disciplinarity.
One way of thinking about the goal of EEP is to create a critical mass of socially empirical educators. They may not all arrive at one shared vocabulary, set of methods, and standard of proof. We will likely need a family of subdisciplines and empirical approaches to handle all the many meaningful contextual differences in education. But if a majority of educators see their work as embedded in a collective effort to learn together about how we can be more effective educators, then we will progress much more rapidly as a sector.
What is the value of doing this? Here again, the EEP founding cohort participants weigh in:
Too often, our conversations about education are weirdly bifurcated. On the one hand, we talk about big problems—helping massive numbers of people improve their economic status, educating a sophisticated citizenry, solving societal equity problems, and so on. On the other hand, we talk about the day-to-day work—picking tools and content, solving a developmental math program at a single college…maybe at most we have consortia working on shared problems within the group. But we only rarely and sporadically talk about the link between solving the local problems and tackling the grand challenges. We don’t have a plan for thinking globally and acting locally.
We at e-Literate contend that higher education is too focused on immediate peer groups, convinced that the most value will come from the people whose context is highly similar. There is some logic to that, given that effective education heavily dependent on many contextual details. But we seem to need a statement of common purpose around the educational part of the mission that is at once broad enough to accommodate differences and focused enough to bring together different types of institutions and individual stakeholders that have different strengths and resources to offer.
I won’t go so far as to say that EEP can offer that, but we aspire to create a collaborative environment out of which that shared vision can emerge by solving specific problems—acting locally—through collaborations that cross peer network boundaries.
