Announcing the Empirical Educator Project

Phil and I are excited to share the news that e-Literate is launching a new initiative called the Empirical Educator Project (EEP). This is something we’ve been working for about half a year now with an increasing number of collaborators. We are far enough along now that we feel we have something to share.

The short version is that we are starting a year-long sponsor-funded experiment intended to test the proposition that we can accelerate improvements in educational access and outcomes by fostering collaborations between organizations that have either similar or complementary needs and goals but don’t know about each other or have trouble figuring out how to work together. If the experiment is successful, then we will continue and expand the effort.

The details matter. I’d like to outline the project for you here and address some of the common questions we’ve gotten as we’ve described the project to various friends in the education sector.

What’s the Big Problem to be Solved?

Higher education is starting to move from talking about improving student outcomes to trying to improve student outcomes. We’d like to focus some of that new momentum in ways that get us to the part where we actually start improving student outcomes a little faster

There has always been a philosophical commitment to the “education” part of higher education in America. But compared to the commitment to disciplinary scholarship, there has not been anything close to the same level of collective focus and institutional support for excellence. In recent years, a number of different factors have created new motivations to start addressing this imbalance:

• The rising awareness among Americans about growing economic inequality has caused more people to ask how well our current educational system is serving those that are being left behind and to experiment with various alternative educational strategies.
• At the same time, good jobs and the expertise that they require are changing so quickly that even people who graduate with what is traditionally considered a “good” degree lack skills that employers need, creating a problem for both employers and job seekers.
• The cost of a college education, which has been rising quickly for some time, has motivated an increasing focus on the economic value of a degree by both students and policymakers, which has raised the profile of the first two problems.
• Elite institutions, which have been able to resist the economic forces for the most part, are starting to realize that today’s students expect more engagement and better use of technology than they get from even the best traditional lecture classes.
• Meanwhile, vendors of educational products ranging from textbooks to learning management systems are starting to see their products commoditize. Nothing less than direct improvement of student outcomes will justify the prices they want to charge. Companies in new product categories that are still trying to break through—like learning analytics—are also dependent on their ability to convince prospective customers that they can directly impact student learning.

As a result of these and other forces, we have seen an increase in activity around improving educational access and outcomes. New faces are showing up at conferences, and many of these people bring a sense of urgency with them. We see more symposia, more pilot projects, and more grants focused on “efficacy” or “outcomes.” Vendors are hiring researchers, publishing peer-reviewed papers, and risking the appearance of presumptuousness by joining in academic conversations about effective teaching practices.

And yet, for all this increased activity, we don’t have much to show for it yet in terms of results. This is going to be a tough nut to crack. Colleges and universities do not have the institutional knowledge, practices, and support structures in place to drive this transformation. Faculty do not have the training, incentives, and support they need. Vendors have neither the credibility they need or the means for establishing it in order to make contributions.

We believe this nascent movement toward a more systemic effort at educational excellence will take at least a generation to come to fruition—and that’s if all goes well. As the saying goes, to change everything, you need everyone.

EEP is our attempt to make a meaningful contribution.

What’s the Big Idea?

We believe one major barrier to progress is social. Work is being done, but it is fragmented. New approaches are being pioneered, but they aren’t shared, tested in different educational contexts, or framed in a way that makes them easy for other individuals and organizations to adopt them. An innovation often doesn’t make it outside the classroom or institution where it was created. When it does, it usually travels to peer groups—other members of the same department, other colleagues in the same discipline, or other universities of the same type.

Colleges that have educational challenges they need to solve often lack researchers to help them, while research universities often have researchers who are looking for good problems to solve. Despite the obvious collaboration opportunity that this resource imbalance presents, classroom educators aren’t aware of relevant educational researchers, and researchers rarely make their work product accessible to and adoptable by classroom educators. And if those researchers happen to work for vendors rather than universities, the problem is even worse. Vendors have no good way to convince the average academic that their research is valid, even when they have real and significant contributions to make. Academics who want to engage with vendors on research projects have no clear avenues to do so.

At e-Literate, we are in the rare and lucky position to be able to see into and across all these various silos. Given that we are seeing both increasing motivation and increasing activity in pockets across the higher education sector, we want to focus on increasing propagation and collaborative refinement of this work by lowering the barriers and increasing the opportunities and incentives for collaborative work. But part we believe that creating incentives for action is not the biggest challenge. Many people and organizations already have enough incentive that they are taking actions now, on their own, with whatever resources they can muster. In many cases, we will just need to create enough additional incentive to make collaboration worth the effort over and above whatever energy they are already putting into their projects.

We believe the bigger challenge will be in the matchmaking.

Why “Empirical Educator Project”?

The first task in fostering collaboration across a diverse group is framing the problem broadly enough to be relevant to everyone yet specific enough to clarify which sorts of problems are in or out of scope for collective action. In this case, we want anyone who is directly responsible for helping students succeed to think of themselves as educators in some sense, and for all educators to think of themselves as being empirical in some sense. The details will vary from educator to educator and context to context. Our project title is intended to define these unifying goals while allowing for flexibility within that shared purpose.

We chose “empirical” rather than “evidence-based,” “efficacy,” “learning science-based,” or any other more popular term because people and organizations differ in both their developmental arcs and the specific kinds of empiricism that make the most sense for their educational contexts. By “empirical,” we simply mean creating some sort of a feedback loop, which can be boiled down to a basic four-step method. First, start with an idea of how you can improve results from some existing baseline. Second, try your idea. Third, examine your results and compare them to the baseline. And finally, adjust your beliefs and actions accordingly.

This method can be applied across a wide range of educational processes. For example, (1) design a lesson that you believe will improve the students’ performance relative to how they do with the current lesson, (2) try the lesson, (3) compare the students’ performance to your baseline, and (4) decide what, if anything, you will do differently next time. Or (1) design a student orientation strategy that you think will improve first-year retention over previous years, (2) try the strategy, (3) check the retention numbers, and (4) decide what to do next year. Or (1) design a new tenure and promotion system that you think will incentivize more effective teaching by some measure you’ve identified, (2) try the new incentive system, (3) see if your effectiveness measurement changes, and (4) adjust accordingly.

Feedback loops can be more or less formal, frequent, and elaborate. Some classes have no feedback loops at all. It’s not uncommon for the only assessment in a law school course to be the final exam. Students have no tools for testing and improving their study strategies throughout the course. In cases where there are no feedback loops, a goal might be to add some. In other cases, goals might be to improve the frequency or quality of the feedback.

There can be many different kinds of feedback loops, as scholars know from their research work. A debate about experimental design, or about the best method for analyzing literature, is a debate about the best way to construct a feedback loop. It is expected that scholarly methods will and should vary dramatically from discipline to discipline, or even between different research problems within a discipline. Likewise, pedagogical approaches and methods ranging from personalized learning to competency-based education to social constructivism all have their own methods of constructing educational feedback loops. We take no position on which is the best kind. We simply want to foster a culture in which educators think of themselves as empirical in the sense that they continually develop and refine feedback loops for themselves and their students.

That brings us to “educator.” We believe that anybody who has some sort of direct responsibility for students’ education should think of themselves as educators. They don’t have to think of themselves as solely or even primarily educators; they can also think of themselves as scholars, researchers, deans, academic technology administrators, instructional designers, product managers, Mets fans, Episcopalians, and so on. But somebody who teaches a class is, among other things, an educator. This may seem obvious, but we have found that it isn’t to all academics. Somebody who sets tenure and promotion policies for teaching faculty is also an educator, as is someone who designs curricular materials or educational technology platforms. To the degree that all these people think of themselves as educators, it will be easier for them to find common ground. We expect the people who participate in the Empirical Educator Project to do so as educators.

Finally, “project.” We intend to practice what we preach by creating our own feedback loop. EEP is a (roughly) one-year experiment. If you’ve read this far, then you know our basic hypothesis. On February 28th, we will be bringing together about 50 people with different roles from a wide range of organizations to peer review that hypothesis, workshop our experimental design, act as an initial cohort to identify a handful collaborative projects that can serve as models, and help us figure out how to open up the experiment to broader community participation.

In the year that follows, we will foster collaborative projects among pairs or small groups of EEP participants. The scope is wide open, as long as the projects create or improve educational feedback loops at the participating colleges and universities. In fact, we are specifically looking to find out which sorts of projects participants are sufficiently motivated to take on without external incentives like big grants.

At the end of the year, we will evaluate whether or not the level of project activity and quality of outcomes support the EEP hypothesis. If our feedback loop tells us we are onto something, we will adjust the design and expand the project. If it doesn’t, we will publish the results of the experiment and think about designing a different one.

How, Exactly, Will This Work?

We will finalize our detailed plan based on the peer review from the summit at the end of this month. But the our core work after the summit will be matchmaking. We will help organizations that are doing similar projects find each other and compare lessons learned. We will help researchers and educators in need of research support find each other. We will help vendors and schools figure out how to collaborate with each other.

Some of this will be directly facilitated by us and our staff, occasionally supported by small grants out of the pool of sponsorship money. A lot of it will happen in an online community space. We have already set up a private conversation area for the summit group. Sometime shortly after the summit—it might take us a little while to figure out the best way to do it—we will open up that space as broadly as is practical to the academic community, possibly in several iterations. If you are reading this and you work at a college or university, we want to make it possible for you to participate.

Vendors and other non-university actors (like foundations, for example) can participate if they are accepted as sponsors.

You Keep Mentioning Sponsors…

Yes. We have sponsors. We don’t need a lot of money to make this pilot work, but we need some. More importantly, we need vendors who have skin in the game.

For the first round of sponsors, we reached out only to a subset of the people that we personally know and trust who are also in positions to foster project participation and support from their companies. We limited the number of sponsors we invited in advance of the summit, in part because we have limited space. Mitchell Stevens and Candace Thille kindly offered to lend us the use of the Lytics lab at Stanford, which is a great space but not a huge one. We had to ensure that we have the right mix of people, that the sponsors don’t outnumber the academics, and so on.

We also made a conscious decision not to privilege sponsors based on their size and ability to pay larger fees than their smaller competitors. There are no gold, silver, and bronze sponsors. We created three standard fee levels that we assign to sponsors based on their valuation. Billion-dollar companies paying the highest fee and small start-ups paying the lowest fee get exactly the same benefits, which are primarily opportunities to participate in the conversation and support the project work.

Since we will be writing about the projects, and since we will be working to help EEP projects get some form of formal or informal peer review whenever possible, sponsors that make real contributions will get credit that would be hard for them to earn otherwise. In addition to supporting projects started by academics, sponsors can propose projects, as long as (a) the projects investigate educational improvements that are not product- or vendor-specific, (b) the sponsoring company agrees to some sort of community or informal peer review of the work, (c) the results are contributed to the educational community under a Creative Commons license or some equivalent contribution mechanism, and (d) there is at least one college or university helping to drive the project.

Here are the current EEP sponsors, in alphabetical order:

• 2U
• Blackboard
• D2L
• Hobsons
• iDesign
• Instructure
• Lumen Learning
• McGraw-Hill Education
• RealizeIT
• Pearson
• Unicon

While we are not actively soliciting additional sponsors, we do anticipate that others will be interested in participating in the project work. (We have already been approached by a couple.) We will vet them and include them using the same criteria we have used so far, including that (a) we only want sponsors who have something to contribute beyond money, and (b) sponsors of all shapes and sizes will be treated fairly and equally. We feel very good about the spirit that our current sponsors are bringing to the work so far and want to maintain a collaborative environment free of sales and marketing pitches.

What About the Academics?

Since we haven’t yet asked the participants’ permission to share their names, I can’t share them here (although I invite participants to announce themselves in the comments if they feel so inclined). While we have encouraged pairs of people to come from institutions in order to increase the likelihood of project work getting initiated, academics participating in the summit are not officially representing their universities. (That includes our host.) So we are being very careful not to put them in awkward positions with their home institutions.

We expect to get permission to share more about who they are after the summit. For now, I will say that we worked hard to get representation from a wide range of institutional types and to include researchers, practitioners, and leaders.

How is This Different from X? And Does This Compete with Y?

There are many projects with somewhat overlapping goals in different corners of academia. That’s healthy. We believe that EEP is unique in its focus on barriers to collaboration, emphasis on participant projects with sound experimental design, and diverse representation across the higher education sector.

We do not intend to compete with any existing efforts. On the contrary, our primary goal is to develop connective tissue between and among them. While we made a decision not to invite any representatives of existing network organizations to the summit because of the space limitations, we were delighted to publicly announce EEP to our friends at the recent EDUCAUSE ELI conference. We look forward to collaborating more with them, as well our friends in other network organizations, as we move forward.

Where Does the Money Go?

The majority of the money covers the cost of the summit, including travel expenses for academics, as well as our staff time for the work involved. What’s left will go toward facilitating the match-making, any additional related staff support time, and likely some small project grants (like paying a graduate assistant, for example). If we get more money, we will reinvest it into project-related activities.

How Does EEP Fit With e-Literate and MindWires?

We’ve heard two versions of this question. One addresses underlying confusion about the relationship between e-Literate, a 14-year-old public service blog, and MindWires, a consulting company that has sprouted up around the blog in recent years, run by the same two guys but seemingly only loosely related. The other asks about the long-term sustainability of EEP under our care. It turns out the answers to both are related.

Almost all of our consulting business comes from readers of e-Literate, and our clients often hire us to help them apply our analysis to their local contexts. But because our paid work usually takes the form of traditional consulting, legal and ethical barriers prevent us from writing about it. As a result, we are not able to share many of the valuable lessons we learn. On top of that, potential conflicts of interest that we are ethically bound to disclose in our writing can appear worse than they really are because we are unable to disclose the specific nature of the engagements.

We haven’t decided how we will fund EEP going forward because any plans for Year 2 depend upon the results from Year 1. But we’re not worried about figuring that out because EEP can potentially solve these problems for us, which would make the endeavor worthwhile for us to sustain even if it just breaks even. To the degree that our clients begin to hire us to support and extend EEP-related projects, the support and publicity they will get from working under the EEP umbrella should incentivize them to share valuable lessons learned. Sharing more information about the work we do for our clients will help us clarify, and in some cases reduce or eliminate, potential conflicts of interest in our writing.

And if EEP can change market incentives such that companies and universities alike will get rewarded for making genuine contributions to a community and culture of educational improvement, then we’re confident we can find ethical ways to get paid to facilitate that change. Our clients may recognize the benefits of engaging us to solve some of their harder problems through the matchmaking and peer support that we can facilitate. But we’re a long way from that. For this year, we are focused on helping EEP participants to develop good collaborative projects.

When and How Can I Join the Fun?

If you work at an academic institution, just sit tight for a little while. We intend to open up community participation within a few weeks of the summit. We will publish the specifics as soon as we have worked them out.

If you work at a potential sponsoring organization, please have the appropriate person contact my colleague Jeanette Wiseman for an initial conversation.