One aspect of the current crisis that I’ve not seen a lot written about is retention. Oh, there’s plenty from a hand-wringing financial perspective. Will students take a gap year? If so, will they come back? Will they move to different institutions with better online learning offerings?
What I haven’t seen a lot of yet is how to address the increased risk that traditionally successful students are more likely to drop out due to financial problems, the stress of working from home, other stresses in their lives, struggles to adapt to online education, and so on. If this list sounds familiar, it’s because students that we have thought of as “at risk” in the past—post-traditional students, including first-generation and older, working students—often face some or all of these challenges. But now, most or all students face them.
Every student should be considered an at-risk student in the current environment.
So we should be looking closely at the strategies that have been developed to address the needs of at-risk students and consider scaling them up to support every student. Particularly at higher end institutions whose populations have not been traditionally at-risk, it’s time to take a closer look at the supports you have in place for your non-traditional cohort and to learn from access-oriented colleges and universities that have been dealing with this problem at scale for a long time.
I recently ran a Standard of Proof webinar with Tyton Partners, which recently released the latest iteration of their Driving Towards a Degree study of this strategy. There’s a lot that’s of interest in their work, including their methodology, which is quite clever. But today I want to highlight one of the major findings of the study, which is that colleges that collaborate across departmental lines on their retention initiatives achieve better retention. We all understand what the phrase “falling through the cracks” means, but Tyton has actually found a way to quantify how many more students fall through the cracks when college departments let those cracks form by not working together to create a seamless experience:
Tyton’s findings on the impact of inter-departmental collaboration on retention
Collaboration is particularly hard right now. Everybody is just trying to stay afloat. But that includes the students. Tyton was very careful not to generalize their pre-COVID study results to the current COVID situation—as they should be—but I would be money that next year’s results will be more dramatic than this year’s because the stresses on students are higher and there are more students under stress. Universities need to work harder than ever to knit their student support efforts into a coherent safety net for their students, and to anticipate many more students needing that support.
The current pandemic is forcing us to rethink much that we used to take for granted. This is true in EdTech as much as anywhere else. For example, while privacy advocates have been shouting into the void about how we should all pay more attention to seemingly boring and definitely complicated legal and technical privacy protections in the EdTech products we use, the Zoom debacle has finally brought more attention to the issue. To reduce the chances of many equally bad privacy problems going unnoticed, we must improve both the tech privacy literacy of people purchasing or selecting the products and the craft of transparency on the part of the product providers. University folk need to become better at reading privacy assurances critically while the service providers need to become better at writing those assurances clearly.
The same goes for the teaching value and effectiveness of products. While this term has been a scramble just getting students online and doing the best that we can for them, next term there will be less excuse for slapdash use of random EdTech products. We must get better at evaluating the strength of vendor claims regarding the improvements in student impact that educators can achieve using their products and, likewise, the vendors much get better at communicating their evidence of impact. This has always been true, but now it is also more obviously urgent. We need to get better at talking about what works, under which circumstances, and how we know what we think we know.
EEP sponsor Macmillan Learning, my most recent guest on the e-Literate Standard of Proof webinar series, talked about how they both test what they know about the effectiveness of their products and communicate what they think they know to customers in the crucial early stages of product development, when there aren’t enough data to make strong claims based on mountains of evidence. This work must be a collaboration with educators in at least two senses. To begin with, EdTech products generally rarely have a significant impact on their own. They usually have the most effective when they are paired with appropriate and effective teaching practices. So early EdTech product development work must involve working with faculty to understand both what they do in the classroom and what they might be willing to do try doing differently if they had a product with affordances that supported a teaching practice that is novel to them. Second, as product developers making increasingly confident claims about their product, those claims—and the process by which they were derived—must be reviewed by academic experts who can provide peer review to both the experimental design before it is used with students and the interpretation of results as the data come in.
As a (paid) member of Macmillan’s Impact Research Advisory Council (IRAC), I have had the pleasure of both observing and participating in their process. Their model of early research and transparency in reporting is a contribution to the field.
Here’s the interview:
Macmillan Learning on e-Literate’s Standard of Proof webinar
I have some announcements and updates to share regarding the Standard of Proof webinars. All good stuff, which I’m sure we could all use more of these days. As a general reminder, Standard of Proof webinars are always about contributions that vendors have made to our general knowledge or capabilities for supporting students—i.e., contributions that don’t just help their own customers—and they always include academics who were meaningful partners in the work. Companies featured in the Standard of Proof webinars are all sponsors of the Empirical Educator Project (EEP), but sponsorship does not guarantee a webinar. Only a qualifying contribution to the educational commons, as judged by me, earns them a webinar. This is not just promotional fluff. So I’m pleased to share the latest with you, which you’ll see represents an uptick in our tempo.
First, the video archive of the webinar on the LXPathways learning architect and instructional technologist competency programs from iDesign is now live on YouTube. Or you can just watch it here, if you have the hour:
iDesign LX Pathways Standard of Proof webinar
Second, a reminder that we have a webinar tomorrow with Macmillan on their method for studying the effectiveness of their products early in the product development cycle. I serve as a (paid) member of the Macmillan Impact Research Advisory Council (IRAC), so I have had both a ring-side seat and some input on this process. The transparency they are attempting to provide regarding what they know about product impact, how they know it, and how confident they are in the results at any given moment is commendable. I’m looking forward to that conversation tomorrow, i.e., Wednesday, March 10th at 2 PM ET. Register for the Macmillan efficacy methodology webinar here.
Third, you may have seen an announcement go out earlier today about our webinar for next Wednesday, March 18th at 2 PM ET for our Driving Toward a Degree webinar with Tyton Partners. We still have a lot to learn about increasing graduation rates, particularly among first-generation, post-traditional, and underrepresented students. Tyton has made an important contribution with their finding of the value of a coordinated response that far too few colleges and universities even attempt to achieve for this sort of effort. Their methodological approach to studying the impact of different approaches is also really interesting. Register for the Driving Toward a Degree webinar here.
And finally, I have a hold-the-date pre-announcement for a follow-up webinar on the research AdmitHub has been doing. (You can see the video of our webinar with them on summer melt here.) This research has broader implications for helping students get to graduation. I’ll hold the details until we have the description nailed down with the participants—this is new research, so I always want to get the characterization right—but for now, save the date for a webinar on Thursday, April 9th at 2 PM.
This post is partly a nudge for you to sign up for the inaugural e-Literate Standard of Proof webinar coming up this Monday at 2 PM and partly a post to tell you why I’m so excited to be kicking off the series with this particular story.
My macro thesis for a while now has been that colleges and universities are in the early stages of a transformation from having a philosophical commitment to student success toward being operationally excellent at supporting and enabling student success. That proposition has been a little abstract for some. If you want to understand what that looks like in the real world, I can’t think of a better example than Georgia State University (GSU) under Tim Renick’s leadership. And this webinar will tell the story of one of his seminal achievements.
Summer Melt is the classic example of the kind of problem that is traditionally invisible to universities that think of student success as a philosophical commitment rather than a core operational responsibility. It’s the phenomenon where students graduate high school, apply to college, get admitted, say they’re coming, fully intend to come, and then never show up. It disproportionately hits first-generation students, students of color, and economically disadvantaged students. Why? Because getting from admission to the first day of classes is a lot harder than many of us remember. You have to fill out a FAFSA form, which as Renick put it yesterday in his IMS presentation, is basically a tax return. I don’t know about you, but I didn’t fill out my FAFSA. My dad did. Not every seventeen-year-old is lucky enough to be able to hand off that responsibility. Then there are inoculations, forms to be filled out and signed by relatives with whom you may or may not have contact, places to get to, fees to pay, and so on, and so on. In their eponymous book on the topic, researchers Benjamin Castleman and Lindsay Page tell us,
In some school districts where as many as 40 percent of college-intending students fail to matriculate, it would be more appropriate to refer to this as a “summer flood.”
Castleman and Page, Summer Melt: Supporting Low-Income Students Through the Transition to College
Forty. Percent.
And some of these problems are very solvable—if you know about them. For example, once the GSU folks realized that immunizations were a problem that was preventing students from getting to the first day of classes, they started parking free immunization trucks outside during times when those soon-to-be students would be visiting campus.
The trick is knowing. So Tim Renick and his team partnered with an AI chatbot company called AdmitHub. It turns out that their CEO, Drew Magliozzi, had read the summer melt book too. And he thought his tool could do something about it. GSU brought in AdmitHub to try it out. But they also brought in Lindsay Page to conduct a randomized controlled trial. It’s one thing to say that you think your intervention improved a problem. It’s quite another to gather credible evidence.
To help these students, the university identified the common obstacles to enrollment that students face between graduating high school and the start of college, including financial aid applications and documents, immunization records, placement exams and class registration, among others. Georgia State developed an approach that would help at-risk students through these obstacles by instituting a combination of a new student portal to guide students through the steps needed to be ready for the first day of classes and an artificial-intelligence-enhanced chatbot, “Pounce,” to answer thousands of questions from incoming students 24/7 via text messages on their smart devices.
In 2016, during the first summer of implementation, Pounce delivered more than 200,000 answers to questions asked by incoming freshmen, and the university reduced summer melt by 22 percent. This translated into an additional 324 students sitting in their seats for the first day of classes at Georgia State rather than sitting out the college experience.
The “Standard of Proof” webinar series is designed to tell stories like this one: Universities working with credible vendor partners to learn and share something new and important about supporting students that is a benefit to the entire sector.
In a recent post, I wrote about my experience on the EDUCAUSE exhibition hall looking for vendors with proof that their products actually help students and how this is an example of the more general problem:
Right now, higher education has very poor signals to quickly distinguish between those vendors who can prove that their products are effective and those that can’t. Between those vendors who are contributing to the general state of knowledge about how to improve student success and those that aren’t. Until we can improve the signal-to-noise ratio, we are doomed to ride the roller coaster from hell that is the hype cycle until we pass out from exhaustion.
I’m going to do something about improving the signal-to-noise ratio, and I will be announcing what that something is within the next week.
Today I am pleased to announce the Standard of Proof ongoing webinar series, which attempts to do just that. The first is scheduled for November. But the plan is to have at least one a month starting in late January, after folks have returned from winter break.
How webinar topics are selected
For many years now, I have turned down requests to facilitate paid webinars because it’s very hard to do them without sliding down a slippery slope and either appearing to be a shill or actually crossing over the line and becoming one. And yet, I would like to give webinars that highlight genuinely good work, and I would like to get paid for my time so that I can afford to do this work.
The Empirical Educator Project (EEP) has given me a novel way to thread the needle. Vendors don’t pay me directly for a webinar. Instead, they sponsor EEP. I vet prospective EEP sponsors—they have to have something to contribute beyond just money—and I also price sponsorship to incentivize active participation and contribution. Standard of Proof webinars are now another incentive for vendors to contribute.
A Standard of Proof webinar is not an automatic benefit of EEP sponsorship. Each webinar is about not a company but a contribution to the work of empirical education. It can be, and often is, novel research conducted by the company that (a) contributes to our knowledge of how to help students succeed, and (b) adds to our knowledge in a way that is generalizable beyond their product or service. In other words, it is a contribution to the commons. But the contribution can also be a project that helps disseminate established research-backed practices or generally promote the culture of empirical education. A contribution should either be free if it has no maintenance costs or non-profit if it has does have a cost to maintain.
Contributions also have to have been developed with some meaningful collaboration with or peer review from academics. As with the definition of “contribution” above, I am more interested in the spirit of meaningful collaboration than in setting up narrow rules. But there has to be an academic who was involved enough in the project that she or he is willing to speak publicly about their participation and their views of the project.
While EEP sponsors are not guaranteed a minimum number of webinars in return for their sponsorship, neither are they limited to a maximum number. If they have made multiple contributions and I have time to cover them in the Standard of Proof schedule, I will. If that means running more than one webinar a month, I would be happy to have that problem.
The end goal here is to help vendors that meaningfully contribute to education differentiate based on those contributions. Providing credible evidence that their solutions support student success is part of that work, but the bar is higher than that. The webinar series title, “Standard of Proof,” refers not only to the products but to the vendors. The series highlights acts of good citizenship that promote empirical education. By raising the profile of these acts and the vendors who perform them, I hope to influence the criteria by which prospective vendors are selected and therefore their incentives for behavior.
Folks, this only works if you and your institutions decide that you will prioritize good behavior—including but not limited to credible proof that a product is effective—in your vendor selection. The most that I can do is help make that good behavior a little more visible.
The first webinar
As I wrote earlier in this post, the first webinar coming up later this month. I have two more lined up for early next year that I will be announcing at a later date and more in the pipeline.
Summer melt: A randomized controlled trial
We’ll be kicking off the series a randomized controlled trial by AdmitHub and its academic partners to advance our understanding of a critical phenomenon that influences the success of first-generation students.
Folks involved with admissions at access-oriented schools may be familiar with the term “summer melt.” It’s when students are admitted to a university, say they are coming, and then never show up. And it tends to be a first-generation student problem. There are all kinds of logistical and bureaucratic hurdles in getting from admissions to first day of classes that may be harder for first-generation students, either because the processes are not designed to accommodate their situations or because they don’t have anybody in their lives who can help them. Maybe they need an immunization certification and they don’t know how to get it. Maybe they need the signature of a parent who they don’t know how to contact. This is a problem that is often caused by a number of small contributing factors, and it is hard to diagnose because the people who know what kept them from coming to college…never came to college. So having a dialogue with them after the fact is difficult to arrange (not to mention unhelpful for the student, who has already missed an opportunity).
There are a number of products on the market, particularly machine learning-enhanced chatbots, that purport to address the summer melt problem. AdmitHub is one of them. But rather than just claim that they can help, they collaborated with their major customer, Georgia State University (GSU), and an academic researcher from the University of Pittsburgh who studies summer melt, to conduct a rigorous study of the impact of the AdmitHub-enabled interventions on summer melt at GSU. The results of the study? GSU was able to reduce summer melt by 20% using AdmitHub.
In addition to providing credible evidence for the product’s effectiveness, this study’s results and experimental design both contribute to our general knowledge of summer melt. We’ll be talking about that as part of the webinar, which will include the PI Lindsay Page from Pitt as well as Tim Renick from GSU, who happens to be a personal hero of mine.
