e-Literate

Present is Prologue

Category: Outcomes

  • During #Covid19, Every Student is At-Risk

    One aspect of the current crisis that I’ve not seen a lot written about is retention. Oh, there’s plenty from a hand-wringing financial perspective. Will students take a gap year? If so, will they come back? Will they move to different institutions with better online learning offerings?

    What I haven’t seen a lot of yet is how to address the increased risk that traditionally successful students are more likely to drop out due to financial problems, the stress of working from home, other stresses in their lives, struggles to adapt to online education, and so on. If this list sounds familiar, it’s because students that we have thought of as “at risk” in the past—post-traditional students, including first-generation and older, working students—often face some or all of these challenges. But now, most or all students face them.

    Every student should be considered an at-risk student in the current environment.

    So we should be looking closely at the strategies that have been developed to address the needs of at-risk students and consider scaling them up to support every student. Particularly at higher end institutions whose populations have not been traditionally at-risk, it’s time to take a closer look at the supports you have in place for your non-traditional cohort and to learn from access-oriented colleges and universities that have been dealing with this problem at scale for a long time.

    I recently ran a Standard of Proof webinar with Tyton Partners, which recently released the latest iteration of their Driving Towards a Degree study of this strategy. There’s a lot that’s of interest in their work, including their methodology, which is quite clever. But today I want to highlight one of the major findings of the study, which is that colleges that collaborate across departmental lines on their retention initiatives achieve better retention. We all understand what the phrase “falling through the cracks” means, but Tyton has actually found a way to quantify how many more students fall through the cracks when college departments let those cracks form by not working together to create a seamless experience:

    Tyton’s findings on the impact of inter-departmental collaboration on retention

    [https://youtu.be/8At8Er_JXMY]

    Collaboration is particularly hard right now. Everybody is just trying to stay afloat. But that includes the students. Tyton was very careful not to generalize their pre-COVID study results to the current COVID situation—as they should be—but I would be money that next year’s results will be more dramatic than this year’s because the stresses on students are higher and there are more students under stress. Universities need to work harder than ever to knit their student support efforts into a coherent safety net for their students, and to anticipate many more students needing that support.

    Watch the whole webinar here.

  • Summer Melt

    Summer Melt: Read this book

    This post is partly a nudge for you to sign up for the inaugural e-Literate Standard of Proof webinar coming up this Monday at 2 PM and partly a post to tell you why I’m so excited to be kicking off the series with this particular story.

    My macro thesis for a while now has been that colleges and universities are in the early stages of a transformation from having a philosophical commitment to student success toward being operationally excellent at supporting and enabling student success. That proposition has been a little abstract for some. If you want to understand what that looks like in the real world, I can’t think of a better example than Georgia State University (GSU) under Tim Renick’s leadership. And this webinar will tell the story of one of his seminal achievements.

    Summer Melt is the classic example of the kind of problem that is traditionally invisible to universities that think of student success as a philosophical commitment rather than a core operational responsibility. It’s the phenomenon where students graduate high school, apply to college, get admitted, say they’re coming, fully intend to come, and then never show up. It disproportionately hits first-generation students, students of color, and economically disadvantaged students. Why? Because getting from admission to the first day of classes is a lot harder than many of us remember. You have to fill out a FAFSA form, which as Renick put it yesterday in his IMS presentation, is basically a tax return. I don’t know about you, but I didn’t fill out my FAFSA. My dad did. Not every seventeen-year-old is lucky enough to be able to hand off that responsibility. Then there are inoculations, forms to be filled out and signed by relatives with whom you may or may not have contact, places to get to, fees to pay, and so on, and so on. In their eponymous book on the topic, researchers Benjamin Castleman and Lindsay Page tell us,

    In some school districts where as many as 40 percent of college-intending students fail to matriculate, it would be more appropriate to refer to this as a “summer flood.”

    Castleman and Page, Summer Melt: Supporting Low-Income Students Through the Transition to College

    Forty. Percent.

    And some of these problems are very solvable—if you know about them. For example, once the GSU folks realized that immunizations were a problem that was preventing students from getting to the first day of classes, they started parking free immunization trucks outside during times when those soon-to-be students would be visiting campus.

    The trick is knowing. So Tim Renick and his team partnered with an AI chatbot company called AdmitHub. It turns out that their CEO, Drew Magliozzi, had read the summer melt book too. And he thought his tool could do something about it. GSU brought in AdmitHub to try it out. But they also brought in Lindsay Page to conduct a randomized controlled trial. It’s one thing to say that you think your intervention improved a problem. It’s quite another to gather credible evidence.

    And that’s what they did. According to GSU’s web page on the topic,

    To help these students, the university identified the common obstacles to enrollment that students face between graduating high school and the start of college, including financial aid applications and documents, immunization records, placement exams and class registration, among others. Georgia State developed an approach that would help at-risk students through these obstacles by instituting a combination of a new student portal to guide students through the steps needed to be ready for the first day of classes and an artificial-intelligence-enhanced chatbot, “Pounce,” to answer thousands of questions from incoming students 24/7 via text messages on their smart devices.

    In 2016, during the first summer of implementation, Pounce delivered more than 200,000 answers to questions asked by incoming freshmen, and the university reduced summer melt by 22 percent. This translated into an additional 324 students sitting in their seats for the first day of classes at Georgia State rather than sitting out the college experience.

    GSU, “Reduction of Summer Melt“

    The “Standard of Proof” webinar series is designed to tell stories like this one: Universities working with credible vendor partners to learn and share something new and important about supporting students that is a benefit to the entire sector.

    This is a canonical example of that, folks.

    Sign up for the webinar.

    The webinar: Come for the enlightenment. Bring your own food.

  • What I Didn’t See at EDUCAUSE

    I often get asked what themes I see coming out of EDUCAUSE. The last few years, the theme has been “there is no theme,” perhaps reflecting the fact that ed tech hype has been in remission. With the exception of the OPM product category, whose hype cycle from zero to peak to trough may have been the fastest we’ve seen yet, the market seems tired of the endlessly repeating roller coaster ride of hope and disappointment.

    This year was a little different. Overall, the conference was the healthiest that I’ve seen it in a while. Attendance seemed to be up. The balance between IT and learning presentations seemed good. There were more vendors on the floor, with more big booths (but no magicians, jugglers, or sword swallowers, thankfully). Overall, the organization seems to be undergoing a revitalization under John O’Brien.

    Within that big picture, there was some color. The major textbook publishers had very little presence, perhaps reflecting their current financial state or perhaps indicating that they’ve figured out the EDUCAUSE crowd isn’t really their target audience, for the most part. On the other hand, ERP/SIS vendors were back with bigger booths, including some names I didn’t recognize (like Unit4, which apparently has been around for a while but which hadn’t heard of before this year).

    There also was a proliferation of small vendors with various flavors of solutions aimed at improving student retention and graduation rates. Each one had its own take on the problem. One had an integration dashboard, pulling in data from the LMS, SIS, and other applications to tell advisors when students might be struggling and need encouragement. Another took a similar approach but focused on giving students nudges directly. Yet another focused on engaging students on social media. And so on.

    Whenever I saw one of these companies, I tried an experiment. I’d do a quick scan of their booth posters and video screens. Just enough to get a very rough sense of what they claimed to do. In other words, I looked about as carefully as the typical exhibition floor browser looks. Then I would approach the booth and say, “I have a rough sense of what your product does from what I see on your booth.” I have one question for you: How do you know that it works?”

    I got a range of different answers.

    “Our customers love it.”

    Bzzt. Wrong answer.

    “We have dashboards that show 67 points where students could be getting bogged down.”

    “OK, but how do you know that it works? I mean, have you tested it to see if it actually impacts outcomes?”

    “…Well…we have a white paper.”

    Pass.

    “We have a customer who improved their retention by 60%! But to be truthful, our product was one of a number of initiatives they put into place, so I don’t know how much of that improvement they would attribute to it.”

    Close, but no cigar. Points for honesty, though.

    “We have conducted multiple randomized controlled trials.”

    “Oh, really! What result were you testing against?”

    “Oh, right, of course. Year-to-year retention…”

    “Thanks, you can stop there for now. May I please have your business card?”

    I found one—just one—vendor who appears to be able to deliver the goods on evidence. I wouldn’t have picked up on that from my cursory scan of their booth. They didn’t look any different than any of the others I had spoken to. In fact, if you had asked me which of the products I looked at would have been likely to prove out as effective, I wouldn’t have ranked them near the top of the list. The trick that they advertise most heavily, which is that they repurpose existing advertising mechanisms on popular social media platforms to deliver productive nudges, seems clever but a little disconnected from problems specific to retention. While they mention “behavioral science” on their web site, they don’t provide a lot of detail. That said, if you read their marketing copy carefully, you’ll see that they do provide a rather stripped-down, easy-to-follow explanation of a randomized controlled trial that they conducted.

    The name of that company, by the way, is Motimatic. Have you heard of them? Because I haven’t. I’m going to look into them and find out more. Maybe you should too.

    Right now, higher education has very poor signals to quickly distinguish between those vendors who can prove that their products are effective and those that can’t. Between those vendors who are contributing to the general state of knowledge about how to improve student success and those that aren’t. Until we can improve the signal-to-noise ratio, we are doomed to ride the roller coaster from hell that is the hype cycle until we pass out from exhaustion.

    I’m going to do something about improving the signal-to-noise ratio, and I will be announcing what that something is within the next week.

    Stay tuned.

  • Empirical Educator Project/CMU Updates

    Empirical Educator Project/CMU Updates

    As promised, I’m returning to Carnegie Mellon University’s announcement of the $100 million contribution of tools, software, and content to support the democratization of learning science, as well as the relationship of that contribution to the Empirical Educator Project.

    Just that first sentence shows how much meat there is yet to be put on the bones. “Contribution?” “Democratize learning science?” What does all that mean? These are fair questions. Jeff Young has some good reporting up at EdSurge that connects some dots on what’s public so far. I’m going to do my best to explain why the answers are not yet as clear as they might be, when and how they will be come clearer, and then give some reminders of past conversations here on e-Literate that provide some breadcrumbs leading in the direction of where all this is going. There will be more news in the press that will drop on Wednesday, followed by some more analysis by me on Thursday. From there, the story will build in pieces, through the Empirical Educator summit on May 6th and 7th, and afterward.

    Let’s start with the big picture. Carnegie Mellon has been a pioneer in various types of theoretical and applied educational and cognitive science research—which is now fashionable to roll up into a ball and call “learning science”—for decades. While their work often doesn’t get the publicity it deserves, the breadth and depth is astounding. I’ve had the privilege of visiting a few times and taking deep dives. ((When I say deep, I mean deep. See, for example, https://mfeldstein.wpengine.com/can-there-be-microscope-of-mind/.)) There are very few institutions in the world that are in their league in terms of the scope of what they do.

    The university made a decision to take a lot of their work product—over $100 million worth, in fact—and make it broadly accessible to the academic community. Honoring this commitment in a real and practical way is…hard. Very, very hard. It’s about more than just releasing the source code to software, which is hard enough to do right in and of itself. This is about making science—not just the output but the practice—accessible and useful to non-scientist educators. On top of that, the components of the contribution were not all designed together as a single software platform. They consist of many research projects, developed by different teams and that may be more loosely or tightly related to each other. Imagine if a top research laboratory decided to turn itself into the Smithsonian Institution, making itself a hands-on science museum accessible to everyone without dumbing itself down. That’s not exactly the goal that Carnegie Mellon’s Simon Initiative has set for itself, but its’ the same spirit. It’s incredibly ambitious and not easily imagined, much less described in a single press release.

    In fact, doing this well is itself going to take some empirical experimentation. And that’s where the partnership with the Empirical Educator Project comes in. Carnegie Mellon had already offered to host this year’s summit on May 6th and 7th. I’m honestly not sure how long they’ve been mulling over this idea of the giant release, but at some point they came to the conclusion that we would be good pilot partners for their effort. This week, we will be describing our high-level approach to our pilot design for helping to make CMU’s contribution accessible. And not just CMU’s contribution, either, since EEP is based on the premise that many academic institutions have innovations to contribute if only we can get better at sharing them. The scale of CMU’s contribution has given us an opportunity to challenge our assumptions about how we should be going about this work. It’s a once-in-a-lifetime opportunity.

    As I said, more details while emerge in the coming days, weeks, and months, first as we learn how to tell such a big story and later as we learn from the grand experiment as it unfolds. For now, I want to leave you with two video playlists. The first is a set of interviews of Carnegie Mellon faculty that I recorded while visiting a few years ago. At the time, I had no agenda other than to capture their individual and collective views on “learning science” and its relationship to classroom teaching. The second is a set of interviews from participants after the first day of the first EEP summit. Taken together, I think you will see why there is a good fit.

    CMU interview YouTube playlist.

    EEP interview YouTube playlist.

  • Carnegie Mellon’s $100 Million Announcement

    This is going to be a short post, in part because I’m traveling, but I need to call your attention to a developing story, both because it’s huge in its own right and for its importance to the Empirical Educator Project (which, by the way, has a new website).

    Carnegie Mellon has announced a $100 million contribution in tools, software, and content that “that is intended to catalyze a new era of progress in educational effectiveness that is equal to the challenge of rapid change and growth in 21st century educational needs. The suite of tools is the product of over $100 million of research and development from a wide variety of funders.”

    The suite of tools will be released in stages over the next year and represents a major departure from the “silver bullet” or “moon shot” efforts to revolutionize education with technology in recent years. Instead, the contribution is intended to democratize the science of learning and empower educators across the world to become citizen scientists. Carnegie Mellon’s goal is to provide knowledge of how to conduct applied educational research that classroom educators, researchers and educational technology companies can learn, apply, extend and share with the global educational community.

    “We live in a moment when our educational institutions are in danger of a catastrophic failure that we cannot afford. College and university closures are becoming regular occurrences, even as tuition and student debt rise to record levels,” said Norman Bier, executive director of the Simon Initiative.

    “This is happening at the same time that even highly educated people need to continue learning in order to keep their skills up-to-date, and when people all over the globe need increasing access to high quality educational opportunities through technology. In the face of these institutional and structural challenges, demonstrably improving outcomes and learning for students must be our foremost concern,” Bier said.

    I have lots of good things to say about this approach, but for a preview, you can go back and review my post about ed tech hype being in remission. This announcement is a little hard to parse because it’s just a down payment on a complex story, and because it’s a big price tag thing from a big engineering school, but trust me: this is not the same old thing.

    The full list of what is being released has yet to be announced, but I’ve seen it, and it is mind-blowing. The breadth and depth are pretty astonishing. In fact, one of Carnegie Mellon’s  biggest challenges will be explaining all of what’s in it. This isn’t a tool or a platform. It’s a collection that’s in the process of being knitted together into an ecosystem. And the way that people inhabit that ecosystem is what will really matter.

    There will be a lot more to say on that in the near future. The university is going to be revealing a lot of the details of their contribution at our second annual Empirical Educator Project summit, which they are graciously hosting on May 6th and 7th. We will have some announcements between now and then, likely a flurry of announcements (from us and from other parties) around the time of the summit itself, and will be releasing video of many of the talks after the summit afterward. There are also some reporters working this story, so I will keep running updates of those stories as they come out over at the new Empirical Educator Project site and periodically collect them in my updates here as well.

    Watch this space.

  • Postscript on Rio Salado Coverage: Clarity about different outcome types

    Postscript on Rio Salado Coverage: Clarity about different outcome types

    After my two posts two weeks ago calling into question whether Rio Salado College has demonstrated clear student outcomes that justify its usage as an exemplar institution, college officials have responded through the WCET blog and in an article from Paul Fain at Inside Higher Ed. I recommend reading the IHE article in particular to get an understanding of the challenging an nuanced question of what is “good enough” in terms of college completion, and for even more detail read Rio’s written response. Unfortunately, the lengthy explanations on graduation rates from Rio end up obscuring a critical distinction between awarding degrees versus certificates of completion. The reason this matters is that Rio appears to be doing a better than average job supporting student transfers to four-year degree programs, a worse than average job of helping students complete two-year degrees, and an unknown job of providing students with non-degree stand-alone certificates.

    Before dealing with this issue, I’d like to point out how useful this dialog has been and how open Rio has been in sharing internal metrics. One example that I wish other institutions used, and that can help us clarify how to understand completion data, is based on student goals.

    Rio Student Intentions

    Fully 76% of Rio students have no intention to get a degree or certificate, and of this group 33% (11,628) indicated earning transferable credit as the key reason to enroll. As Sally Johnstone said in the IHE article, “It’s a feeder school for Arizona State University”. This transfer of credit mission can be good or bad – it depends on how well these students perform in getting an ASU degree. In the written response, but not in WCET or IHE articles, Rio shares an interesting but incomplete metric.

    Additionally, as part of the work RSC does related to accountability, we track the success of our transfer students to the three public universities in Arizona (The 2010/11 – 2015/16 5 year trend shows a 77.10% increase in bachelor degrees awarded to students that had completed a minimum of 12 credit hours (up to 60 or more hours) at Rio Salado College).

    Increase to what and over what? It is not clear if this is comparing to Arizona university students who do not transfer any credits from Rio, or those who transfer less than 12, or those who transfer from other colleges. Nevertheless, this is an indicator of the college helping students obtain four-year degrees through credit transfers.

    Once you get beyond non-degree / non-certificate seeking students, the numbers are murkier, and this is the area that needs clarity. One out of four Rio students are there to get an award – how successful are they? Rio recommends we look at the Voluntary Framework for Accountability (VFA) measures, instead of IPEDS, with the following results.

    • 2-year graduation rate is higher than peer institutions: Rio 13%; Peer Institutions 10%
    • 6-year graduation rate for credential-seeking cohort is 36.2%
    • The credential seeking cohort far outperformed sister institutions in completion: Rio 42%; Peer Institutions 16%

    I do not know why Rio uses the term graduation rate, as VFA instead describes Percent Students Completed a Formal Award, but from the numbers used we can see where the data comes from. The first and third bullets come from the VFA Two-Year Progress measures (blue column under Completed). ((The VFA definition of the cohorts: Main Cohort = fall entering, first time at reporting college; “all students”; Credential Seeking = earned 12 credits by end of year two.))

    What is key is that the Completed metric includes those who obtained an associate’s degree or official certificate. Unlike degrees, certificates of completion have no standards across schools, and their usage is very uneven. As an example of Rio’s usage:

    Notably, since 2010 Rio has defined and awarded more and more of these <1 year certificates, based on IPEDS data.

    We can see is that the large majority of completions, which Rio labels under graduation rate, comes from the nonstandard certificates rather than degrees.

    One of the strengths of VFA is that under the Six-Year Outcomes measures, you can separate out the various types of award. Below is the data for Rio – pay attention to the Credential-Seeking Cohort, as it removes the majority of non-degree / non-certificate seeking students at Rio only looking to transfer a small number of credits. ((For degree or certificate-seeking cohorts, I prefer the Rio definition based on student self-reporting, but the VFA definitions get at the same idea.)) Also note that VFA outcomes are mutually-exclusive and hierarchical in the order shown. If a student gets a certificate and a degree, they are listed under the degree outcome.

    For the Credential-Seeking Cohort, 22% of Rio students get a certificate and do not transfer to a four-year school, 7% get a certificate and transfer, 3% get an associate’s degree and do not transfer, and 4% get an associate’s degree and transfer. A large majority of Rio Salado completions are for certificates, which is consistent with the IPEDS data.

    How does Rio compare to other large community colleges? While we have not analyzed all 200+ colleges in VFA, spot checking with some peer schools indicates that Rio is well below others in awarding degrees but well above others in awarding certificates.

    The resulting data provide the basis of my comment in IHE:

    “This is not a bad situation, per se, as long as students are gaining value in the workplace for these official certificates,” he said via email. “But certificates are not useful in terms of comparing apples to apples, especially when one school uses them liberally, as does Rio, and most other schools do not.”

    Hill also noted that Rio Salado’s VFA completion rate for associate degree programs is a “troublesome” 7 percent.

    While I appreciate the valuable sharing of information from Rio Salado in this process, I’ll stick with my original conclusion:

    At best, this is a school with mixed results that should not simply be labeled a success without caveats or explanations.

    Rio has shown that they can cut costs and still be a successful transfer school, but not that they can be a successful degree school. This matters a great deal in terms of how transferrable the lessons are. It’s fine to focus on transfers when you’re right next door to ASU. But what if you’re Adirondack Community College? There are places where degree completion matter more or less than at Rio. And what about those students going to Rio who *are* seeking degrees? Does this model promote some educational outcomes at the expense of others? And if so, shouldn’t students know about that before enrolling?

  • In “EdTech”, “Ed” comes before “Tech”: A National Louis University/Acrobatiq Case Study

    A few weeks ago, I posted about our case study on the governance of the Courseware in Context (CWiC) Framework. CWiC is a laudable effort to take the traditional product evaluation matrix and get beyond “magic quadrant-ism” by adding some educational context to the selection process. Which product is “best” depends a lot on your goals, capabilities, and other contextual factors. The CWiC advisory committee members explain how hard it is to integrate these factors into a product evaluation matrix. A lot of the hoped-for value of CWiC is still aspirational. There’s nothing wrong with that; it’s good to have big goals and to recognize how close to (or far from) achieving them you really are.

    Sometimes just seeing your path to achieving those big aspirations is hard in and of itself. As this next case study shows, even if CWiC realizes its potential, it’s never going to be the right place to start if your goal is to meaningfully impact student outcomes and close achievement gaps. Our original goal with the National Louis University (NLU) and Acrobatiq interviews was to look at what it takes to implement courseware effectively with a vendor partner once the selection is made. In other words, what happens after you go through the process that the CWiC framework is meant to support. But we got so much more than that because NLU has been incredibly thoughtful about the program and students—the context—that impacts the use and value of the courseware. In other words, they also modeled what universities need to do before they select courseware, from designing a business/sustainability model that enables them to provide appropriate cost of an education to thinking about educational goals to figuring out where courseware does and doesn’t fit into that overall model. All of that prework has a profound impact on the working relationship with the vendor partner, the kinds of support identified as necessary to make the courseware work, and the future vision for how to continue to broaden and deepen the impact.

    In the end, I think this case study shows both that a tremendous amount of work is needed to integrate courseware if you truly want to drive improvement in student outcomes and that the significant amount of planning and implementation work involved is only a small percentage of the total necessary program planning and work required. NLU has effectively redesigned the entire undergraduate experience. Their partnership with Acrobatiq works in large part because Acrobatiq understands their role in NLU’s larger goals, and because NLU maintains their focus on those goals rather than getting caught up in the notion that buying some magic product will solve all their problems.

    (https://youtu.be/Iau4QaezReo)

    (https://youtu.be/rJtc53dXQbI)

    (https://youtu.be/R7xv7QRJ5R4)

    I’m proud to have this be the final case study in this series of grant-funded stories. It neatly reflects a lot of what we believe about the ways in which universities and vendors need to work together if we want to improve higher education.