I have some announcements and updates to share regarding the Standard of Proof webinars. All good stuff, which I’m sure we could all use more of these days. As a general reminder, Standard of Proof webinars are always about contributions that vendors have made to our general knowledge or capabilities for supporting students—i.e., contributions that don’t just help their own customers—and they always include academics who were meaningful partners in the work. Companies featured in the Standard of Proof webinars are all sponsors of the Empirical Educator Project (EEP), but sponsorship does not guarantee a webinar. Only a qualifying contribution to the educational commons, as judged by me, earns them a webinar. This is not just promotional fluff. So I’m pleased to share the latest with you, which you’ll see represents an uptick in our tempo.
First, the video archive of the webinar on the LXPathways learning architect and instructional technologist competency programs from iDesign is now live on YouTube. Or you can just watch it here, if you have the hour:
iDesign LX Pathways Standard of Proof webinar
Second, a reminder that we have a webinar tomorrow with Macmillan on their method for studying the effectiveness of their products early in the product development cycle. I serve as a (paid) member of the Macmillan Impact Research Advisory Council (IRAC), so I have had both a ring-side seat and some input on this process. The transparency they are attempting to provide regarding what they know about product impact, how they know it, and how confident they are in the results at any given moment is commendable. I’m looking forward to that conversation tomorrow, i.e., Wednesday, March 10th at 2 PM ET. Register for the Macmillan efficacy methodology webinar here.
Third, you may have seen an announcement go out earlier today about our webinar for next Wednesday, March 18th at 2 PM ET for our Driving Toward a Degree webinar with Tyton Partners. We still have a lot to learn about increasing graduation rates, particularly among first-generation, post-traditional, and underrepresented students. Tyton has made an important contribution with their finding of the value of a coordinated response that far too few colleges and universities even attempt to achieve for this sort of effort. Their methodological approach to studying the impact of different approaches is also really interesting. Register for the Driving Toward a Degree webinar here.
And finally, I have a hold-the-date pre-announcement for a follow-up webinar on the research AdmitHub has been doing. (You can see the video of our webinar with them on summer melt here.) This research has broader implications for helping students get to graduation. I’ll hold the details until we have the description nailed down with the participants—this is new research, so I always want to get the characterization right—but for now, save the date for a webinar on Thursday, April 9th at 2 PM.
I am just thrilled to post the archive of our inaugural Standard of Proof webinar:
Summer Melt Standard of Proof webinar
https://youtu.be/7NoYUa_VHfs
It is a near-perfect encapsulation of everything that the Empirical Educator Project aspires to promote and foster: front-line educators, academic researchers, and commercial vendors working together to measurably increase student success and to do so in a way that is both persuasive in terms of the rigor of evidence and repeatable by other institutions.
As a reminder, Standard of Proof highlights academic collaborations by the Empirical Educator Project’s commercial sponsors with academic researchers and academic practitioners. All projects must either contribute new knowledge to the commons or promote the diffusion of established evidence-backed practices.
In this case, we have a story of a small EdTech start-up—AdmitHub—that has a total of nine randomized controlled trials either completed or in process with academic partner institutions.
Nine.
How many of your EdTech vendors have nine randomized controlled trials either done or in the works? How many have one? How many have you even asked this question? (If you want better answers on the first two questions, then try to improve your answer to the last one.)
They were able to do this, in part, because institutions like Georgia State University had real leaders, like GSU’s Tim Renick, who refused to accept the proposition that disadvantaged students were already beyond help by the time they got to college. Instead, the GSU folks (and their colleagues at other academic institutions conducting research with AdmitHub) set out to study their own practices and identify areas that they were failing students in ways that could be fixed. Using data, with the help of their vendor and an academic researcher—in this case, University of Pittsburgh Professor and summer melt expert Lindsay Page—GSU demonstrated that we can make education better and more equitable. It is the perfect counter-example to the sad situation I described in my post on what I didn’t see at EDUCAUSE.
This is the future that we all should be reaching for, folks. And it is reachable, as all three of my guests made clear. The video is well worth an hour of your time. Standard of Proof webinar production values will improve as we learn our craft, but the conversation quality just doesn’t get much better than this.
We’re actively working on the spring webinar line-up and will have some juicy conversations scheduled by the time you return from your holiday break.
This post is partly a nudge for you to sign up for the inaugural e-Literate Standard of Proof webinar coming up this Monday at 2 PM and partly a post to tell you why I’m so excited to be kicking off the series with this particular story.
My macro thesis for a while now has been that colleges and universities are in the early stages of a transformation from having a philosophical commitment to student success toward being operationally excellent at supporting and enabling student success. That proposition has been a little abstract for some. If you want to understand what that looks like in the real world, I can’t think of a better example than Georgia State University (GSU) under Tim Renick’s leadership. And this webinar will tell the story of one of his seminal achievements.
Summer Melt is the classic example of the kind of problem that is traditionally invisible to universities that think of student success as a philosophical commitment rather than a core operational responsibility. It’s the phenomenon where students graduate high school, apply to college, get admitted, say they’re coming, fully intend to come, and then never show up. It disproportionately hits first-generation students, students of color, and economically disadvantaged students. Why? Because getting from admission to the first day of classes is a lot harder than many of us remember. You have to fill out a FAFSA form, which as Renick put it yesterday in his IMS presentation, is basically a tax return. I don’t know about you, but I didn’t fill out my FAFSA. My dad did. Not every seventeen-year-old is lucky enough to be able to hand off that responsibility. Then there are inoculations, forms to be filled out and signed by relatives with whom you may or may not have contact, places to get to, fees to pay, and so on, and so on. In their eponymous book on the topic, researchers Benjamin Castleman and Lindsay Page tell us,
In some school districts where as many as 40 percent of college-intending students fail to matriculate, it would be more appropriate to refer to this as a “summer flood.”
Castleman and Page, Summer Melt: Supporting Low-Income Students Through the Transition to College
Forty. Percent.
And some of these problems are very solvable—if you know about them. For example, once the GSU folks realized that immunizations were a problem that was preventing students from getting to the first day of classes, they started parking free immunization trucks outside during times when those soon-to-be students would be visiting campus.
The trick is knowing. So Tim Renick and his team partnered with an AI chatbot company called AdmitHub. It turns out that their CEO, Drew Magliozzi, had read the summer melt book too. And he thought his tool could do something about it. GSU brought in AdmitHub to try it out. But they also brought in Lindsay Page to conduct a randomized controlled trial. It’s one thing to say that you think your intervention improved a problem. It’s quite another to gather credible evidence.
To help these students, the university identified the common obstacles to enrollment that students face between graduating high school and the start of college, including financial aid applications and documents, immunization records, placement exams and class registration, among others. Georgia State developed an approach that would help at-risk students through these obstacles by instituting a combination of a new student portal to guide students through the steps needed to be ready for the first day of classes and an artificial-intelligence-enhanced chatbot, “Pounce,” to answer thousands of questions from incoming students 24/7 via text messages on their smart devices.
In 2016, during the first summer of implementation, Pounce delivered more than 200,000 answers to questions asked by incoming freshmen, and the university reduced summer melt by 22 percent. This translated into an additional 324 students sitting in their seats for the first day of classes at Georgia State rather than sitting out the college experience.
The “Standard of Proof” webinar series is designed to tell stories like this one: Universities working with credible vendor partners to learn and share something new and important about supporting students that is a benefit to the entire sector.
In a recent post, I wrote about my experience on the EDUCAUSE exhibition hall looking for vendors with proof that their products actually help students and how this is an example of the more general problem:
Right now, higher education has very poor signals to quickly distinguish between those vendors who can prove that their products are effective and those that can’t. Between those vendors who are contributing to the general state of knowledge about how to improve student success and those that aren’t. Until we can improve the signal-to-noise ratio, we are doomed to ride the roller coaster from hell that is the hype cycle until we pass out from exhaustion.
I’m going to do something about improving the signal-to-noise ratio, and I will be announcing what that something is within the next week.
Today I am pleased to announce the Standard of Proof ongoing webinar series, which attempts to do just that. The first is scheduled for November. But the plan is to have at least one a month starting in late January, after folks have returned from winter break.
How webinar topics are selected
For many years now, I have turned down requests to facilitate paid webinars because it’s very hard to do them without sliding down a slippery slope and either appearing to be a shill or actually crossing over the line and becoming one. And yet, I would like to give webinars that highlight genuinely good work, and I would like to get paid for my time so that I can afford to do this work.
The Empirical Educator Project (EEP) has given me a novel way to thread the needle. Vendors don’t pay me directly for a webinar. Instead, they sponsor EEP. I vet prospective EEP sponsors—they have to have something to contribute beyond just money—and I also price sponsorship to incentivize active participation and contribution. Standard of Proof webinars are now another incentive for vendors to contribute.
A Standard of Proof webinar is not an automatic benefit of EEP sponsorship. Each webinar is about not a company but a contribution to the work of empirical education. It can be, and often is, novel research conducted by the company that (a) contributes to our knowledge of how to help students succeed, and (b) adds to our knowledge in a way that is generalizable beyond their product or service. In other words, it is a contribution to the commons. But the contribution can also be a project that helps disseminate established research-backed practices or generally promote the culture of empirical education. A contribution should either be free if it has no maintenance costs or non-profit if it has does have a cost to maintain.
Contributions also have to have been developed with some meaningful collaboration with or peer review from academics. As with the definition of “contribution” above, I am more interested in the spirit of meaningful collaboration than in setting up narrow rules. But there has to be an academic who was involved enough in the project that she or he is willing to speak publicly about their participation and their views of the project.
While EEP sponsors are not guaranteed a minimum number of webinars in return for their sponsorship, neither are they limited to a maximum number. If they have made multiple contributions and I have time to cover them in the Standard of Proof schedule, I will. If that means running more than one webinar a month, I would be happy to have that problem.
The end goal here is to help vendors that meaningfully contribute to education differentiate based on those contributions. Providing credible evidence that their solutions support student success is part of that work, but the bar is higher than that. The webinar series title, “Standard of Proof,” refers not only to the products but to the vendors. The series highlights acts of good citizenship that promote empirical education. By raising the profile of these acts and the vendors who perform them, I hope to influence the criteria by which prospective vendors are selected and therefore their incentives for behavior.
Folks, this only works if you and your institutions decide that you will prioritize good behavior—including but not limited to credible proof that a product is effective—in your vendor selection. The most that I can do is help make that good behavior a little more visible.
The first webinar
As I wrote earlier in this post, the first webinar coming up later this month. I have two more lined up for early next year that I will be announcing at a later date and more in the pipeline.
Summer melt: A randomized controlled trial
We’ll be kicking off the series a randomized controlled trial by AdmitHub and its academic partners to advance our understanding of a critical phenomenon that influences the success of first-generation students.
Folks involved with admissions at access-oriented schools may be familiar with the term “summer melt.” It’s when students are admitted to a university, say they are coming, and then never show up. And it tends to be a first-generation student problem. There are all kinds of logistical and bureaucratic hurdles in getting from admissions to first day of classes that may be harder for first-generation students, either because the processes are not designed to accommodate their situations or because they don’t have anybody in their lives who can help them. Maybe they need an immunization certification and they don’t know how to get it. Maybe they need the signature of a parent who they don’t know how to contact. This is a problem that is often caused by a number of small contributing factors, and it is hard to diagnose because the people who know what kept them from coming to college…never came to college. So having a dialogue with them after the fact is difficult to arrange (not to mention unhelpful for the student, who has already missed an opportunity).
There are a number of products on the market, particularly machine learning-enhanced chatbots, that purport to address the summer melt problem. AdmitHub is one of them. But rather than just claim that they can help, they collaborated with their major customer, Georgia State University (GSU), and an academic researcher from the University of Pittsburgh who studies summer melt, to conduct a rigorous study of the impact of the AdmitHub-enabled interventions on summer melt at GSU. The results of the study? GSU was able to reduce summer melt by 20% using AdmitHub.
In addition to providing credible evidence for the product’s effectiveness, this study’s results and experimental design both contribute to our general knowledge of summer melt. We’ll be talking about that as part of the webinar, which will include the PI Lindsay Page from Pitt as well as Tim Renick from GSU, who happens to be a personal hero of mine.
