In addition to the highly-misleading usage of ‘first-time full-time’ qualification for official graduate rates reported in the College Scorecard, there appears to be another major issue with the data. In particular, the Education Department (ED) is using a questionable method of determining whether an institution is degree-granting rather than relying on the IPEDS data source. In a nutshell, if an institution awarded more certificates than degrees, then it is not labeled as “predominantly awarded 2-year or 4-yeard degrees” and therefore excluded.
Russ Poulin noted in his WCET post that several community colleges were missing from the Scorecard based on a quick spot check:
- Colorado – Aims, Front Range, Pueblo, and Otero Community Colleges.
- Arizona – Rio Salado College.
- California – Bakersfield College.
The consumer website itself offers no explanation that certain degree-granting schools are excluded. The Technical Paper that corresponds to the data release explains that selection of schools on page 28:
The data presented in this report cover the universe of all institutions of higher education that are active participants in the Title IV federal aid program, are located in the 50 states and the District of Columbia, and both have a positive number of degree-seeking undergraduates and awarded some undergraduate credentials in the 2012-2013 school year. The IPEDS definition of institution is used as the basic level for reporting data, though for many analyses in this report a higher level of aggregation (i.e., six-digit OPEID) is used due to data limitations.
OK, this indicates the following criteria:
- Titile IV participation
- Located in US
- Positive number of degree-seeking undergraduates
- Awarded some undergraduate credentials in 2012-2013
- Listed as institution under IPEDS
All school mentioned by Russ (and others that I could find) are listed in the actual data files, but they are not on the consumer website. And all meet the criteria listed above. Something is wrong here.
There are others, such as Robert Kelchen, who have noted the missing colleges. In this Stack Exchange discussion I think we find the answer to the missing data in an answer from Brian, who works at the ED.
At the present time, the Scorecard site is limited to institutions that predominantly grant 2-year and/or 4-year degrees. The Department of Education is exploring whether and how to add predominantly less-than-two-year schools that offer associate’s or bachelor’s degree programs to the site in future iterations of Scorecard.
Brian further clarifies in a separate answer.
Predominant degree is calculated by the most common category of award conferred by the institution, calculated from counts of awards included in the IPEDS Completions component.
Sure enough, Front Range Community College awarded 1,771 certificates and 1,693 associate’s degrees in 2012-2013. Based on the private Brian criteria, this cuts Front Rage out of the current Scorecard. I did a quick check of all public 2-year degree-granting institutions, and these omissions are significant. IPEDS lists 939 schools meeting this criteria while the College Scorecard lists only 780. That’s 17% of all degree-granting community colleges that are not included in the College Scorecard. In Colorado, only 6 of 13 community colleges are included. In California, only 94 of 112 are included.1
Furthermore, this private criteria explained by Brian is not mentioned in the technical documentation, and it is not mentioned on the website. Why did ED not simply use the well-known and accepted IPEDS classification of “degree-granting”? I can only assume that they wanted to exclude measurements of certificates, but they ended up eliminating a significant number of community colleges in the process. And they did so in an undocumented, arbitrary manner.
Again, I ask: is this how the ED is going to help consumers make informed choices?
Update 9/15: To do more extensive checking, I applied Brian’s criteria above to IPEDS data for the first 3 pages of College Scorecard results, alphabetically sorted. For IPEDS I listed all degree-granting public 2-year US-based institutions. For Scorecard I searched public, associate’s. The results:
- College Scorecard = 60 results
- IPEDS = 84 results
- For every IPEDS result where # certificates > # degrees, the school was not on the Scorecard
- There were 7 schools on Scorecard not on IPEDS list due to being listed as 4-year although vast majority degrees being associates; but they were in IPEDS under public 4-year category
In other words, Brian’s criteria (# degrees must be > # certificates to be listed) does seem to explain the missing colleges.
Update 9/16: For another sanity check, I downloaded IPEDS data and filtered on the public criteria listed above (Title IV, US, positive # of degree-seeking students, some awards). I then applied Brian’s criteria on my own in Excel to see if I could come close to matching results with the Scorecard. Due to the differences in how IPEDS classifies sectors as 2-year and 4-year whereas the Scorecard uses Associate’s and Bachelor’s, I combined Public 4-year and Public 2-year data together. The results:
- 1,617 institutions that meet the public criteria
- 1,355 institutions that pass Excel version of Brian’s criteria
- 1,368 institutions listed in College Scorecard
- That means 15 – 16% of all public institutions are missing from the Scorecard
I haven’t figured out all of the nuances of how ED has created their public-facing website (and note that it does not allow download of results to allow independent analysis), but this is pretty close. I strongly believe that Brian’s private criteria explains the vast majority of missing public institutions in the College Scorecard.2
[…] Late yesterday I posted about the Education Department (ED) new College Scorecard and how it omits a large number of community colleges based on an arbitrary metric. […]