College Scorecard: With victories like these, who needs failures?

Goldie Blumenstyk had a fascinating interview with the Depart of Education’s Ted Mitchell on Friday that is well worth reading and / or watching (they have video of the interview along with full transcript). One of Mitchell’s key points jumped off the page for me.

I think it [College Scorecard] was one of the department and the administration’s greatest victories

Really? I can see the consolidation of student loans, a strong focus on college affordability, shifting of conversation away from just elite schools, and significant push on funding public schools as worthy victories to mention. But the College Scorecard – I don’t buy it.

Blumenstyk pushed back on Mitchell, which led to this interesting exchange [emphasis added].

GOLDIE BLUMENSTYK: I’m glad you mentioned the College Scorecard. I was thinking about that a little bit. It’s probably one of the places where the department had perhaps its biggest defeat, or maybe you might consider it a retreat. We were originally envisioning the Scorecard as a tool for accountability. Obviously, a lot of colleges and a lot of other people opposed that idea. And it became a complicated process even to create the effective scorecard. What did you learn from that process?

TED MITCHELL: So I guess I would have a slightly different interpretation.

GOLDIE BLUMENSTYK: I would imagine.

TED MITCHELL: I think it was one of the department and the administration’s greatest victories in two ways. First of all, we made an ambitious goal. We worked hard to get there. And as we got closer, we realized that the goal might be the wrong one and that the kind of accountability that we had talked about at the beginning of the process isn’t the kind of accountability that we think really, actually does matter on the ground. The kind of ability — excuse me. I get so excited talking about Scorecard. The kind of accountability that matters on the ground is the kind of accountability that allows an individual user to identify what’s important to them, to get reliable information about that, and then to make decisions about it.

We decided that it’s better for students and parents and counselors and helpers to be able to ask the queries that they want to ask rather than to hear what the government thinks is rated A, B, or C. So I think we listened. We learned. And we came up with a product that’s superior than what we started with.

Let’s take this argument and see where it takes us. Just under a year ago, Russ Poulin from WCET and I wrote a series of posts calling out major flaws in the College Scorecard, culminating in a joint article in the Washington Post. In these posts, we called out the following flaws in the Scorecard:

1. The data set is limited to Title IV schools, excluding those who don’t accept federal financial aid;
2. (At the time) the Scorecard excluded nearly one in three 2-year colleges;
3. Graduation rates are based on first-time full-time student cohorts that represent less than 20% of all students;
4. Transfer rates are not included; and
5. Average costs of attendance are based only on the roughly 50% of students taking federal financial aid.

Of these issues, nearly one year later, only problem #2 has been fixed (and kudos to ED for this fix). Problem #1 doesn’t affect that many schools, so we’ll ignore that for now.

Graduation and Transfer Rates

Earlier in the interview Mitchell acknowledged a very important point.

I think one of the things that has motivated the Obama administration from the beginning has been a recognition that in order to achieve our goals as a country and in order to support individuals in their striving for better lives, we have to understand that the college student of today is not the college student of my generation or your generation. The 18-year-old who got dropped off in the minivan at State U. is not a majority anymore. It’s the 24-year-old returning veteran. It’s the 36-year-old single mom. It’s the person who’s fully employed trying to up skill.

Then why does the ED continue to use first-time full-time student cohorts – “the 18-year-old who got dropped off in the minivan at State U” – to measure graduation rates? In this post last year I pointed out that University of Maryland University College (UMUC) has less than 2% of their students fitting this first-time full-time definition, yet the College Scorecard screams out at the user that the school has only a 4% graduation rate1.

UMUC has the data on different cohorts that give graduation rates from 18% – 56%, yet none of this matters for the Scorecard.

There is also no acknowledgement that “success” does not always equal graduation – it can include transferring from a 2-year to a 4-year school. This data is available in a data dump from the Scorecard, but it is nowhere to be found for the consumer web page that the vast majority of prospective students and parents will use. From my post:

Consider the harm done to prospective UMUC students by seeing the flawed, over-simplified ED College Scorecard data, and consider the harm done to UMUC as they have to play defense and explain why prospects should see a different situation. Given the estimate that non-traditional students – those who would not be covered at all in IPEDS graduation rates – comprise more than 70% of all students, you can see how UMUC is not alone. Community colleges face an even bigger problem with the lack of transfer rate reporting.

Average Costs

There are similar problems when considering the average cost of attendance, since those measurements come solely from students taking federal financial aid. Glendale Community College did a study on the effects of this measure on the group of community colleges in the Los Angeles area.

Enrollment fees are set by the state legislature, so all community colleges in the state charge the same amount there. Other fees are set by the colleges, but they shouldn’t be too different, should they? Cost of living varies widely across the state, so maybe that is factored in and would cause variations.

This is the reason that the study focused just one the Los Angeles area, to minimize (but not eliminate) variations in cost of living.

California community college fees are very low compared to fees in most states, and nearly all low-income students receive state aid such as Board of Governors (BOG) fee waivers. Students who receive no federal financial aid are not included in the Scorecard cost indicator, even if they receive BOG waivers and other state aid.

The result is an enormous variation in average costs measures that bears little resemblance to reality for the majority of students. I updated Glendale’s chart to include data on six colleges that were missing as of last fall.

Accountability Without Accuracy

I’ll go back to Mitchell’s definition of the great accountability achieved with the Scorecard.

The kind of accountability that matters on the ground is the kind of accountability that allows an individual user to identify what’s important to them, to get reliable information about that, and then to make decisions about it.

The data in the Scorecard is flawed, and I am stunned to see this listed as one of their greatest victories.

• For the vast majority of students who are not first-time full-time students, they have no access to realistic graduation rates and the ones presented are misleading.
• For the more than 50% of students not taking federal financial aid, they have no access to realistic average costs of attendance, and in cases where there is significant state aid, the resulting data can be non-sensical.

I’ll end with the concluding paragraphs from the Washington Post article I wrote with Russ Poulin.

We applaud the concept of a Scorecard. A consolidated, consumer-focused website with accurate, comparable data on every institution could do more to improve institutional behavior than a boatload of regulations. Just consider how colleges jockey to improve their position on the current (less reliable) rankings sites.

It is difficult for students to have a clear vision of the future, when they are looking at it through fuzzy lenses.