Rio Salado College As Exemplar: A critical external view

In yesterday’s post I described how a review of two courses at Rio Salado College indicated reasons to question the use of this school in the ASU / BCG case study report on Digital Learning1. The report, titled “Making Digital Learning Work: Success Strategies From Six Leading Universities and Community Colleges”, has the following description [emphasis added]:

How can the use of digital technologies in postsecondary education impact students’ access to education, student outcomes, and the return on investment for students and institutions? What are the biggest challenges for an institution seeking to implement high-quality digital learning opportunities? What promising practices enable an institution to achieve impact at a larger scale? [snip]

The answers, at least in part, lie in case studies of six colleges and universities: Arizona State University, the University of Central Florida, Georgia State University, Houston Community College, Kentucky Community and Technical College System, and Rio Salado Community College. The first three institutions in this list are public research universities, representing different geographic populations and access missions. The other three institutions include two community colleges and a state-wide community college system.

These six institutions have a strong track record of using digital learning to serve large, socioeconomically diverse student populations, and each has been a pioneer in innovating to expand access to postsecondary education, improve student outcomes, and provide higher education at an affordable cost.

Let’s look at the track record of Rio Salado College in terms of aggregate academic student outcomes to see how appropriate it is to include them as an exemplar in such a case study-based report. For much of my analysis, I looked at the 862 public 2-year colleges fully reporting data in the Fall 2016 IPEDS data set.

Reduced Expenditures

The primary claim made in the ASU / BCG report is that Rio Salado is has reduced costs.

This claim does hold up to scrutiny, as Rio’s $89 of instructional delivery costs per student credit hour and $16 of student services costs are 10th and 9th lowest in the country for the 862 colleges I reviewed.

The primary outcomes claims made in the case study quickly brush off the standard metrics for Rio.

Rio Salado’s success is not defined solely by improvements in graduation and retention rates. In particular, the college has a high transfer-out rate (32% compared to an average of 19% for other MCCCD colleges [ed. – the other schools in the Maricopa County Community College District]), and the students who transfer to Arizona universities from Rio Salado have a 74% four-year graduation rate—3 percentage points higher than the average for other MCCCD transfer students. At Rio Salado itself, students’ course-level success rates have slowly been improving over time, to about 64% in 2016.

Graduation and Retention Rates

It is all well and good to point out the importance of transfer students, but we should not ignore graduation and retention rates. And we should explore whether the data supports the phrase “improvements in graduation and retention rates”.

The report describes Rio’s investment in a suite of advising tools.

Obtaining these advisory tools required an initial investment of $1 million, but the tools pay for themselves by increasing Rio Salado’s term-to-term retention by 7%, increasing ROI through improved student progress and the additional tuition dollars that the institution receives from retained students.

Looking at IPEDS data, we can see Year 1 to Year 2 retention rates (not quite the same as term-to-term), and it shows improvement since 2013. Unfortunately, Rio’s retention rates for full-time and part-time students are both in the bottom 10% of all community colleges at 33% and 27%, respectively. And the full-time rate is lower than it was prior to 2011.

While the ASU / BCG report does not mention graduation rates directly, there have been several claims made about Rio’s numbers in other publications.

• American Enterprise Institute, 2012: “Rio Salado boasts a 42 percent graduation rate, about twice the national average for community colleges.”
• The Hechinger Report, 2014: “Thanks to a barrage of support, however, Rio Salado boasts a graduation rate four times that of comparable schools, U.S. Department of Education figures show.”
• Community College Review, 2017: “The greatest success seen at Rio Salado thus far is their incredibly high graduation rate, which currently stands at four times that of comparable institutions!”
• Bill & Melinda Gates Foundation, 2017: “Rio Salado’s two-year graduation rate—that is, the percentage of students who finish an associate’s degree in the normal two years—is four times higher than the average for 100 peer institutions.”

The problem is that Rio Salado’s 4-year graduation rate for first-time full-time students as reported in IPEDS is 5% – the second lowest of any public 2-year college in the nation. How can we resolve this discrepancy?

The key to understanding the claims in these four articles is to follow the two links, which both point to a customized IPEDS Feedback Report from 2013 for Rio against their selection of 100 peer institutions.

This measure captures any award – degree or certificate – for a small cohort of entering full-time students at Rio from 2009. If you trace that data through IPEDS, you see that this measure is for a 150-student cohort (in 2012) out of the 45,000+ students at Rio.

Using this specific metric, you can find the 42% graduation rate in 2011, and the “four times greater” rate of 27% in 2012, both as the rate plummets from 64% in 2008 to 3% in 2016.

Using broader, up-to-date metrics for graduation and retention rates as well as the new IPEDS outcomes measures (which allows comparison outside of just full-time first-time cohorts), we see that Rio Salado College has some of the lowest student outcomes measures in the country.

For all but two measures (6-year First-Time Full-Time Awards and 6-year Non-First-Time Full-Time Awards), the performance is in the bottom 10% of all colleges in the study. For those two awards measures, the performance is in the bottom half of the country and represents just 11% of the student body.

Transfer Rates

What we are left with are reports about high transfer rates and subsequent success rates 3% higher than other transfers out of the same district. The IPEDS data set now includes transfer rates in its new Outcomes Measures section, and it is true that Rio Salado College does transfer out a significant portion of students. For all cohorts defined below, Rio is in top 20% of public 2-year institutions.

• First-Time Full-Time Students: 39% transfer rate, Rank 170 out of 862
• First-Time Part-Time Students: 47% transfer rate, Rank 112 out of 862
• Non-First-Time Full-Time Students: 46% transfer rate, Rank 196 out of 862
• Non-First-Time Part-Time Students: 59% transfer rate, Rank 121 out of 862

Note that this data indicates that students transferred to another institution and are still enrolled there. This data does not indicate what portion of the transfers were planned versus students deciding to move on for other reasons. Nevertheless, for a community college, especially one with very close ties to ASU (I have been told that a large percentage of Rio students are ASU students trying to fill out their schedule based on saving money and more convenient schedules), this is a favorable metric.

I cannot provide independent confirmation on the subsequent success rates, so we’ll accept that claim at face value.

Case Study Questions

I should note that Rio Salado College does provide accurate information on the metrics mentioned in this post, both through IPEDS reporting and through their Research & Planning section of the web site. The picture that emerges from the aggregate outcomes data is of a large school that:

• transfers out greater than average percentage of students;
• is just below average for awarding certificates for full-time students;
• is among the poorest performing in the country for retaining students;
• is among the poorest performing in the country for awarding degrees; and
• is among the poorest performing in the country for awarding certificates for part-time students.

At best, this is a school with mixed results that should not simply be labeled a success without caveats or explanations.

One question we should ask is whether it is appropriate to hold up a school with some of the lowest student outcomes measures in the country as an exemplar. Yes, Rio Salado has found a way to spend as little as possible on instruction and student support services, and yes, a lot of students transfer out, but that is not enough. We need greater evidence of student success if we are to use them as a case study for others to emulate. And we also need a more robust genre of a case study that looks across, at a minimum, a relatively standard set of publicly-available information, and deeper dives where appropriate, to understand educational practices and their impacts on students. These case studies should present information in context – the good and the bad – since education is complex and challenging, and a clear focus on evidence will benefit all parties in the end.