e-Literate

Present is Prologue

Tag: Empirical Educator Project

  • Climbing the Ladder of Empirical Education

    A while back, I wrote a post about the four levels of Empirical Education. To recap, they are as follows:

    1. Intuitively empirical: Intuitively empirical educators are curious about their students and try to figure out how to help them when they see them struggling. They try different things and pick up tricks in their teaching as they become more experienced in the classroom.
    2. Mindfully empirical: This means that the educators try to create as many opportunities as possible to evaluate how their students are doing and make little (or big) adjustments to her teaching strategies constantly as they get to know their students better. Intuitively empirical educators are empirical in the moment. Mindfully empirical educators are empirical by design.
    3. Metacognitively empirical: Metacognitively empirical educators have made the leap from assessing their students to assessing themselves. When a class struggles with a concept semester after semester, metacognitively empirical educators don’t just accept that the topic is hard. They ask whether their teaching strategies might be part of the problem. They challenge their own beliefs about good teaching.
    4. Socially empirical: When educators reach the level of being socially empirical, it means that they have begun to see that testing teaching strategies and learning to improve can be a shared endeavor. It can be a discipline, with common language and standards of evidence for effectiveness. Socially empirical educators see teaching not just as an art that is personal and ineffable but as a craft that can be taught and learned, and maybe even as a science that can be advanced through shared research and peer review.

    This isn’t just a taxonomy. It’s a theory of change. Anyone who has participated in course redesign efforts likely recognizes the milestones of this progression. That’s not an accident. Course redesign, particularly when properly facilitated by experts who are themselves practitioners of Empirical Education, can move the instructors who are going through the redesign process at least partway up the ladder. In fact, it often happens spontaneously.

    Let’s take the example of launching a distance learning program. In most cases, these programs are going to attract intuitively empirical educators. People who aren’t curious about the process of educating students usually don’t volunteer to teach using a method that they’ve never tried before. (And when they are forced to do so, it is often a disaster.) So distance learning programs are often lucky to get a preponderance of instructors who are already on the first rung of our ladder.

    Teaching online with any degree of quality almost always forces a course redesign. Educators moving from a physical classroom to a virtual one lose some senses while gaining others. Likewise, certain strategies they have come to rely on in the classroom won’t work online and have to be translated to or replaced by strategies that will. And the educator that is new to online teaching isn’t always going to know what will work and what won’t. They are almost required to become mindfully empirical in order to figure out how to replace the bandwidth they lost in the transition.

    Sometimes, the process of trying new strategies because of the redesign provokes the empirical educator to rethink some of her basic assumptions about teaching. Anyone who has helped educators get courses online for more than a couple of years has likely heard at least a couple of them say that the experience of teaching online has caused them to change the way they teach in the classroom. That’s the beginnings of metacognitively empirical education.

    And this is where things start to get hard, for several reasons. First, it’s very difficult to get faculty to make the leap to metacognitively empirical education without a conversion experience that happens when they have to teach their course differently. Both experience and research tell us that many educators—and probably most humans—have deeply held beliefs about what makes for effective teaching based on their own formative experience as learners. No matter how otherwise rational and open-minded they are, they are not likely to be persuaded by research studies. They have to live the change. Outside of distance learning programs, it’s hard to programmatically create new opportunities for formative experiences.

    Second, even when they do live the change and bring new ideas back to their other classes, that doesn’t mean that they have reached the point where they are mindfully or systematically examining their own teaching strategies. Doing the latter requires both another step of self-reflection and some new skills. So a key challenge for Empirical Education is to make it easier to reliably give educators the right experiences, prompts for further reflection, and tools for investigation, in the proper order. Universities aren’t currently set up to do this. But they could be.

    The third challenge in climbing the ladder is moving from metacognitively empirical to socially empirical. Educators who have made the metacognitive leap tend to get excited. It can be a life-changing experience. So they often want to evangelize. But where? If their home institution isn’t already creating a fertile environment for them to share teaching strategies with their peers—which it usually isn’t—then the metacognitively empirical educators tend to go to discipline-specific conferences, where they end up swapping ideas with a small circle of people who are already converted and largely share their background and training. This is far from ideal for cross-fertilization. So another great challenge of Empirical Education is to create new opportunities for metacognitively empirical educators to find each other and learn from each other across normal silos in higher education, like discipline or school type.

    Phil and I believe that colleges and universities in all segments across the sector are beginning to make the transition from a philosophical commitment to student success toward an operational commitment toward student success. What that looks like in the end, and how long it takes, is up for grabs. Empirical Education is, among other things, a theory of change that is intended to accelerate the progress of that transition while strengthening the institutions of higher learning that have served civilization so well by empowering and motivating educators to be change agents.

  • The Empirical Educator Project’s First Public Collaboration

    It is almost exactly the six-month anniversary of the Empirical Educator Project’s (EEP’s) first annual summit at Stanford University. The thesis behind EEP is simple: We believe that one of the biggest barriers to increasing student access and success in US higher education is a failure to communicate. We see substantial innovation and progress happening across the sector, from individual classrooms to entire university systems, from access-oriented institutions to research universities to vendors of all shapes and sizes. But for a variety of reasons, the lessons being learned rarely travel far. We believed that if we could just get the right people talking to each other and sharing their work with each other, then good things would happen.

    That’s it. It was that simple.

    Six months ago, we tried it for the first time with about 50 hand-picked people for one and a half days. Most people there knew less than half the people in the room. That was one of our goals; we wanted to get people talking across the normal boundaries of their peer (or customer, or competitor) groups.

    Early indications are that we succeeded beyond our wildest hopes. The group generated somewhere between 15 and 20 project collaboration ideas, many of which have been active even over the summer months. We haven’t written much about the ongoing work because we’ve been waiting for the participants to feel they have made enough progress on their respective projects that they are ready to talk about them publicly.

    Today, I am happy to tell you about the first EEP collaboration that has crossed that threshold. Here’s the text of an announcement from James Madison University’s (JMU’s) department of Psychology:

    JMU faculty associated with the Assessment & Measurement PhD program and the Center for Assessment and Research Studies have formed a strategic partnership with colleagues at Carnegie Mellon University to assess educational effectiveness. Drs Sara Finney and Keston Fulcher began this partnership with CMU when meeting Dr. Marsha Lovett at the Empirical Educators Project meeting at Stanford in February 2018.

    Since that time, JMU faculty member Dr. Jeanne Horst traveled to CMU to engage in training offered by the Eberly Center for Teaching Excellence & Educational Innovation in June 2018. During the 2.5 day Teaching As Research Institute, Dr. Horst engaged in conversation about evidence-based teaching and learning practices. She and the CMU professionals shared experiences regarding the process of developing and implementing intentional curriculum and instruction in higher education and assessing its effectiveness.

    In July 2018, JMU hosted our CMU colleagues Mike Melville and Elizabeth Whiteman for training offered during JMU’s week-long national award-winning Assessment 101. During this training, JMU faculty and graduate students offer practice-orientated materials and build skills regarding student learning outcomes assessment.

    In August 2018, Dr. Marsha Lovett traveled to JMU to observe JMU’s large-scale Assessment Day. In addition to learning about how to plan, proctor, and engage students during institutional accountability testing, Dr. Lovett also observed a focus group to elicit students’ perceptions of general education courses. She also met with JMU faculty and graduate students to plan the next steps in our strategic partnership. JMU faculty look forward to sharing our expertise in assessment and measurement with CMU faculty while gaining additional knowledge and skills associated with creating and implementing evidenced-based programming from the experts (and our friends) at CMU.

    JMU and CMU are about half a day’s drive from each other. They are both (underrated) top-tier universities. Each has particular areas of strength in applied learning science research that complements the other’s. And yet, they didn’t know about each other until they met at EEP. That one meeting was all it took to trigger three rather substantial reciprocal campus visits—one in June, one in July, and one in August—to learn from each other and to develop what JMU is now publicly characterizing as a “strategic partnership.”

    The EEP network has quite a few other such pots simmering on the stove that you’ll be hearing more about in the coming months. Not all ideas for joint efforts will bear fruit, and not all EEP participants are engaged in successfully progressing projects yet. We at e-Literate are still learning how to reduce the friction that inhibits successful collaboration. But we are off to a fantastic start, thanks in large part to the amazing folks in our first EEP cohort.

    Here’s an e-Literate TV video of several CMU Simon Initiative and Eberly Center faculty talking about their view of learning science:

    And here’s JMU’s Sara Finney talking about their approach to course and program effectiveness assessment in her lightning talk at the EEP summit:

    Peanut butter, meet chocolate.

    Stay tuned. We’ll have more good stories to share soon.

  • Empirical Educators Define the Project

    In my last post, I wrote that the Empirical Educator Project (EEP) is e-Literate’s way of putting our money where our mouth is, so to speak, regarding our own theory of change for improving student success and closing achievement gaps. What is that theory? I articulated it in some detail last February, before our inaugural summit.

    But it was just a theory then. After one day at the summit, we asked participants to volunteer their own views about what EEP is and why it might be valuable. We created a four-part interview series, which can be found here. But I want to highlight a few points from the interviews that have convinced us that we are onto something.

    Let me start by articulating that theory as simply as I can:

    Many, many educators are already working hard on meeting student success challenges, but they are often doing it in isolation. If we can introduce the right people to each other and get them talking to each other about the student success challenges they are working on, then good stuff will happen.

    That’s it. That’s the whole enchilada.

    Here it is in a little more detail:

    1. Many educators at many institutions—including both universities and the companies that serve them—feel personally responsible for improving student success and are actively testing ideas and learning lessons about how to do so.
    2. Social barriers often prevent these lessons learned from being shared, tested, and built upon.
    3. One of the biggest barriers is the difficulty of finding collaborators who are motivated to work on the same problem and have complementary capabilities for doing so.
    4. One major reason that potential collaborators don’t find each other is because they live in different peer networks and don’t naturally cross paths.
    5. If we can make it easier for those people to find each other, then they will do great things together, even without a lot of external funding or organizational support.
    6. Once people from different peer networks collaborate on a project, the results will tend to propagate in all of the participants’ respective networks.
    7. If we do this enough, then lasting bridges will begin to form among currently isolated networks and the pace of sector-wide progress will accelerate.

    Here’s how we are testing the idea in EEP’s first year:

    1. We made a list of people across the higher education sector who we know are doing great work and are good collaborators.
    2. We trimmed the list down to a manageably small group that was spread across different peer networks. (Most of the summit participants told us they knew fewer than half of the other attendees.)
    3. We raised enough money to get the group into the same room for a day and a half, and to provide some light follow-up.
    4. We spent our time together at the summit encouraging folks to get to know each other, share what they are working on, and look for project collaborators.
    5. Post-summit, we are watching to see whether collaborative projects happen and are lightly facilitating where we can.

    From the beginning, we viewed this very much as an experiment and honestly didn’t know how much we would be able to accomplish with a group of strangers in less than two days. We were absolutely blown away by what happened.

    Do yourself a favor. Take a little under eight minutes and listen to what these participants had to say about our theory of change after just one day of talking to each other:

    It’s fairly easy to get a group of smart people together in a room and have them walk away feeling like they had a good, satisfying intellectual conversation about what might or could be done. It’s a lot harder to get them excited about a common, complex goal, with grand ambitions and a sense of how those ambitions might be met. If you didn’t watch the whole video above, then just listen to this bit from Duke University’s Bridgette Martin Hard about what she described as “one of the biggest, loftiest goals that I see being possible [for EEP]:

    Culture change. For the entire sector. There it is. The whole enchilada. After one day of discussion.

    OK, discussion, is nice, but is anything actually going to happen? Well, here is a sampling of participants talking about some of the project ideas that emerged from the summit—again, after just one day of being together:

    Every person in that room said they had at least one collaborative project idea they had seriously discussed and thought had a realistic chance of actively happening. Some had as many as three projects.

    That was after one day. What about now, three months later? I am in the process of compiling the first project update for the group. (We’ll keep these updates private to the group until each group of collaborators decides they have made enough collective progress to share their work more broadly. So far, I have about a dozen substantive updates from project teams, and I know for a fact that I have not received updates from all the groups that have engaged in post-summit work on their projects. This all sprung out of one brief gathering with about fifty people who largely didn’t know each other and have decided to move ahead with collaborative work despite the fact that they are getting zero grant money for the projects. The problem isn’t lack of motivation for change; it’s the barriers. In a variety of ways that I’ll explore in future posts, the match-making that is happening in EEP lowers various barriers—yes, the social barrier I talked about up front, but resource barriers as well. (There are hints of this in the videos above.)

    There is no longer any question for us about whether we will continue EEP after this first year. We believe that some of these projects will be ready to come out of stealth mode by the fall, with more going public by the end of the calendar year. We are slowly and carefully adding new members to the cohort and expect to expand it more substantially in the spring of 2019, when we have our next summit. We are also exploring ideas for broader outreach, along the lines of Bridgette’s thinking above. We’ll have more to say about this in the fall.

     

  • The ASU + GSV Conference was More GSV than Ever—And That’s Good

    The ASU + GSV Conference was More GSV than Ever—And That’s Good

    I’ve been struggling with how to characterize this year’s ASU + GSV Summit. Part of the problem is that it’s just become harder to sum up, for three reasons. First, the conference has gotten so big that it’s hard to see enough of it to get a representative sample. Second, as it has become a “must attend” conference for certain circles, it becomes harder for me to clear time away from meetings with people I know—both planned and unplanned—to be a fly on the wall. I can partly deal with that problem by bringing e-Literate colleagues who can be extra pairs of eyes and ears.

    But the third problem is harder. As the conference and the culture around it mature, the trends become harder to identify because they’re no longer fashion. We’re past the Year of the MOOC and the Year of Personalized Learning and all of that. Some of the less patient money has left, and some of it has refocused on corporate talent management. (Good luck with that; I don’t think it’s any easier than edtech.) A lot of the hype has faded into the background. The conversations seem more real and more focused on tackling important educational problems in realistic ways.

    Don’t get me wrong; there are still 12-year-olds in $300 jeans talking to 22-year-olds in $3,000 suits about how their math app is going disrupt education, and a few random dudes—and they are still often dudes, despite a sincere and concerted effort to make the conference more diverse—walking around wearing tinfoil headbands and loudly declaring, “I CAN READ STUDENTS’ MINDS!”

    But it’s easy to get distracted by the spectacle and miss the big picture. One way to look at the summit is as a giant peer review machine. Yes, with so many submissions, the quality will be uneven. And yes, sometimes a bad product will make it through the peer review process. And yes, you may find the process and the culture to be alien and unsettling if you’re not from that world. But from what I can see, the peer review machine is getting better in the sense that the pitches from startups to investors increasingly seem to focus on things that actually make sense in terms of developing products that are educationally useful.

    Coincidentally, I was reviewing some of the videos of the lightning talks from our Empirical Educator Project (EEP) summit this morning as I was thinking about this post. I was struck by how the presentation from RealizeIT, which is the type and size of company that would typically pitch at ASU + GSV, was not hugely different from the best of the pitches at that summit:

    The main difference is that, for the EEP crowd, the presentation focused on the collaborative research with University of Central Florida and others to improve student impact, while a good GSV pitch might pivot off the impact visualizations to talk about how their evidence of impact has led to increased sales. Having the latter conversation between companies and funders is an important piece in the ecosystem that was missing before the ASU + GSV conference. Trying to have the former type of conversation doesn’t work easily in the same forum with the latter type, partly for reasons of cultural differences. In the early years of ASU + GSV, the conference organizers—again, with the best intentions—worked very hard to try to host both conversations, with the result often being strong negative reviews from the academics. In more recent years at the summit, the academic needs seem to be coming in more organically via the companies as intermediaries and self-selecting academics who are, for lack of a better term, bi-cultural.

    The one trend that did stand out at ASU + GSV, which I think is related to the above, is a lot of activity around Online Program Management (OPM) companies. Something fairly deep is happening in this space. Right now, most of the buzz is around the debate over whether the future of OPM business models is in revenue share or fee-for-service. I think that’s a red herring, created in part by the bad blood between some individuals at Noodle Partners—one of the leading proponents of the fee-for-service model—and 2U—the darling of the revenue share crowd. The truth is that, while there is some overlap in competition, what we are seeing increasingly is a range of different companies, many but not all of which are currently labeled as “OPMs,” that offer different bundles of services and different financing models that are best suited for solving different kinds of problems. The common thread is that there is a growing range of service businesses that help colleges and universities develop strong, student-centered degree and certificate programs, either by creating new ones or strengthening existing ones. We don’t have a good term or set of subcategories for this class of businesses, of which traditional OPM is one subcategory. (And some of these business are pretending that they are not service businesses for a variety of reasons, but that’s another story.)

    To me, this squishiness in the OPM market is a sign that more and more schools are serious enough about learning to be systematically better at supporting student success that they are increasingly willing to pay for help. And that change in prioritization is potentially good for everyone. And ASU + GSV has become a good vantage point for spotting that kind of change—if you look hard enough.

  • What is the Value of Empirical Education?

    In my last post, I wrote about the goal of the Empirical Educator Project (EEP) as getting a critical mass of educators to be “socially empirical”:

    Socially empirical educators view effective teaching not as an individual art but as a shared pool of knowledge and experience that everyone can learn from and contribute to. They seek out common vocabulary, methods, and standards of proof so that they can learn with their colleagues and raise the collective bar. This is the beginning of disciplinarity.

    One way of thinking about the goal of EEP is to create a critical mass of socially empirical educators. They may not all arrive at one shared vocabulary, set of methods, and standard of proof. We will likely need a family of subdisciplines and empirical approaches to handle all the many meaningful contextual differences in education. But if a majority of educators see their work as embedded in a collective effort to learn together about how we can be more effective educators, then we will progress much more rapidly as a sector.

    What is the value of doing this? Here again, the EEP founding cohort participants weigh in:

    (https://youtu.be/rCqPzCRn0ZU)

    Too often, our conversations about education are weirdly bifurcated. On the one hand, we talk about big problems—helping massive numbers of people improve their economic status, educating a sophisticated citizenry, solving societal equity problems, and so on. On the other hand, we talk about the day-to-day work—picking tools and content, solving a developmental math program at a single college…maybe at most we have consortia working on shared problems within the group. But we only rarely and sporadically talk about the link between solving the local problems and tackling the grand challenges. We don’t have a plan for thinking globally and acting locally.

    We at e-Literate contend that higher education is too focused on immediate peer groups, convinced that the most value will come from the people whose context is highly similar. There is some logic to that, given that effective education heavily dependent on many contextual details. But we seem to need a statement of common purpose around the educational part of the mission that is at once broad enough to accommodate differences and focused enough to bring together different types of institutions and individual stakeholders that have different strengths and resources to offer.

    I won’t go so far as to say that EEP can offer that, but we aspire to create a collaborative environment out of which that shared vision can emerge by solving specific problems—acting locally—through collaborations that cross peer network boundaries.

  • What is an Empirical Educator?

    We have the first batch of Empirical Educator Project (EEP) summit videos available in the form of a four-episode interview case study. You can see the full set here. I’ll be blogging about the series over the next week or so, starting today with the first episode on the definition of an empirical educator.

    A lot of the currently popular terms of art that bear some relation to “empirical educator”—efficacy, evidence-based, research-backed, data-informed, etc.—tend to produce discipline-inflected responses (e.g., “It makes no sense to talk about ‘data” when you’re teaching English composition”) or suspicion due to vendor, funder, and media hype (e.g., “‘Efficacy’ is just the latest marketing buzz phrase”). And many of these terms tend to emphasize products or implementations. We wanted to put the focus on people, and we wanted the term to be inclusive of a wide range of practices that are discipline- and context-appropriate. So we introduced a new term, which always requires some norming to get people rallied around a common understanding of it.

    Here’s what a sampling of summit participants had to say about the meaning of “empirical educator” after a day of work together:

    (https://youtu.be/-h5-OB4WvwQ)

    The definition becomes increasingly inclusive as you listen to the complementary perspectives over the course of the video.

    I think we’ll develop a more rigorous, evidence-based understanding of what it means to be an empirical educator over time—while hopefully maintaining the inclusive spirit. For now, I’ll put out my own taxonomy of levels of empirical education:

    1. Intuitively empirical: This comes down to whether you pay attention to your students and do something differently with them based on what you observe. Do you always do the same thing, or do you have a bag of tricks that you can draw from when you see students struggle? I believe that the substantial majority of educators are empirical in this sense. They may not think of it as empiricism, but they are observing student behaviors and are adjusting their strategies based on what they see, guided by some sort of rationale for choosing which strategy to employ in different circumstances.
    2. Mindfully empirical: Mindfully empirical educators think about how they can get the maximum amount of useful diagnostic information from day-to-day course work. They design their courses with a goal of creating many feedback loops that enable them to be adjust their teaching to the needs of the students.
    3. Meta-cognitively empirical: Meta-cognitively empirical educators are empirical not only about how they use their existing bag of tricks but also about which tricks they should have and how effective those tricks really are. They consciously and regularly test their own assumptions about effective teaching, and they are open to trying new appropaches. My read of Lauren Herckis’ research is that the barrier of moving from mindfully empirical to meta-cognitively empirical (and to the next level, socially empirical) is where a lot of the difficult work needs to be done. Lots of educators are intuitively empirical, and the transition from there to mindfully empirical is not a huge leap. Getting them to test and challenge their deeply held beliefs about what constitutes effective teaching is a lot harder.
    4. Socially empirical: Socially empirical educators view effective teaching not as an individual art but as a shared pool of knowledge and experience that everyone can learn from and contribute to. They seek out common vocabulary, methods, and standards of proof so that they can learn with their colleagues and raise the collective bar. This is the beginning of disciplinarity.

    One way of thinking about the goal of EEP is to create a critical mass of socially empirical educators. They may not all arrive at one shared vocabulary, set of methods, and standard of proof. We will likely need a family of subdisciplines and empirical approaches to handle all the many meaningful contextual differences in education. But if a majority of educators see their work as embedded in a collective effort to learn together about how we can be more effective educators, then we will progress much more rapidly as a sector.

    I think it’s also important to note that the term “educator” is intended to be more inclusive than just people who work in classrooms. Although instructors will be central change agents in any empirical education movement, anyone who is directly involved with designing or delivering a meaningful educational intervention is included in our definition of an educator. This means program administrators, instructional designers, and product designers, among others, can all be empirical educators.

  • The CWiC Framework: Context around Courseware

    Update: I got feedback that it wasn’t clear the embedded video was a three-video playlist, so I have broken them out individually.

    We have a new case study up in which we interview members of the Courseware in Context (CWiC) framework advisory committee about their work:

     

    The focus isn’t so much on the framework itself as on the project goals and, somewhat less directly, it’s sustainability model. We’ve seen repeated failures in the market to create selection tools for curricular materials or edtech products. (Some of us are old enough to remember EduTools.) The committee members are very thoughtful about the missing element, which is context. There are, in CWiC advisory committee member Tanya Joosten’s words, “a million” different contextual variables, from classroom implementation to support to individual student needs. CWiC is an attempt to take a traditional product selection tool model and enrich it enough to account for all these contextual variables.

    The committee members seem acutely aware of how big of a challenge this could be, which is part of what makes these interviews so interesting. At our recent Empirical Educator Project summit, René Kizilcec (whose research we have featured here on e-Literate) made a fascinating comment, which is that we education “desperately needs” a “science of context.” The CWiC folks are trying infuse the output we currently have from the beginnings of that science into what is essentially a tool for selection and procurement. Will it work? I don’t know. The committee members are pretty clear about the size of the challenge. But it’s the right direction.

    The Empirical Educator Project (EEP) attempts to take on the other side of René’s phrase. What would it mean to have a science of context? Who would the scientists be? What cultural institutions would we need to form a consensus around what a science of context would look like such that people who care about it could cooperate on it more effectively, and such that it would attract the funding it needs to be carried out? What business process support tools, including but not limited to procurement tools, do we need to implement a science of context, and how would implementation provide input back to the scientific research? These are the questions that EEP attempts to answer. Will it work? I don’t know. But I feel confident that it’s the right direction.