e-Literate

Present is Prologue

Category: Research

Empirical studies, good and bad.

  • College Scorecard: ED quietly adds in 700 missing colleges

    It’s worth giving credit where credit is due, and the US Department of Education (ED) has fixed a problem that Russ Poulin and I pointed out where they had previously left ~700 colleges out of the College Scorecard.

    When the College Scorecard was announced, Russ noticed a handful of missing schools. When I did the whole data OCD thing, I discovered that more than 700 2-year institutions were missing, including nearly 1-in-4 community colleges. Eventually we published an article in the Washington Post describing this (and other) problems.

    The missing community colleges were excluded on purely statistical grounds. If the college granted more certificates (official awards of less than a degree) than degrees in a year, then they were excluded as they were not “primarily degree-granting” institutions. We label this the “Brian Criterion” after the person authoring two discussion board posts that explained this undocumented filter. (more…)

  • Empowering Students in Open Research

    Phil and I will be writing a twice-monthly column for the Chronicle’s new Re:Learning section. In my inaugural column, “Muy Loco Parentis,” I write about how schools make data privacy decisions on behalf of the students that the students wouldn’t make for themselves, and that may even be net harmful for the students. In contrast to the ways in which other campus policies have evolved, there is still very much a default paternalistic position regarding data.

    But the one example that I didn’t cover in my piece happens to be the one that inspired it in the first place. A few months back at the OpenEd conference, I heard a presentation from CMU’s Norm Bier about that challenges of getting different schools to submit OLI student data to a common database for academic research. Basically, every school that wants to do this has to go through its own IRB process, and every IRB is different. Since the faculty using the OLI products usually aren’t engaged in the research themselves, it generally isn’t worth the hassle to go through this process, so the data doesn’t get submitted and the research doesn’t get done. Note that Pearson and McGraw Hill do not have this problem; if they want to look at student performance in a learning application across various schools, they can. Easily. Something is wrong with this picture. I proposed in Norm’s session that maybe students could be given an option to openly publish their data. Maybe that would get around the restrictions. David Wiley, who does a lot more academic research than I do, seemed to think this wasn’t a crazy idea, so I’ve been gnawing on the problem since then.

    I have talked to a bunch of researchers about the idea. The first reaction is often skepticism. IRB is not so easy to circumvent (for good reason). What generally changed their minds was the following thought experiment:

    • Suppose that, in some educational software program, there was a button labeled “Export.” Students could click the button and export their data in some suitably anonymized format. (Yes, yes, it is impossible to fully de-identify data, but let’s posit “reasonably anonymized” as assessed by a community of data scientists.) Would giving students the option to export their data to any server of their choosing trigger the requirement for IRB review? [Answer: No.]
    • Suppose the export button offered a choice to export to CMU’s research server. Would giving students that option trigger the requirement for IRB review? [Answer: Probably not.]

    There are two shades of gray here that are complications. First, researchers worry about the data bias that comes from opt in. And the further you lead students down the path toward encouraging them to share their data, such as making sharing the default, the more the uneasiness sets in. Second and relatedly, there is the issue of informed consent. There was a general feeling that, even if you get around IRB review, there is still a strong ethical obligation to do more than just pay lip service to informed consent. You need to really educate students on the potential consequences of sharing their data.

    That’s all fair. I don’t claim that there is a silver bullet. But the thought experiment is revealing. Our intuitions, and therefore our policies, about student data privacy are strongly paternalistic in an academic context but shift pretty quickly once the institutional role fades and the student’s individual choice is foregrounded. I think this is an idea worth exploring further.

  • Data To Back Up Concerns Of Textbook Expenditures By First-Generation Students

    David Wiley has added to the conversation ((My initial post, Mike Caulfield response, Bracken Mosbacker, my response to Mike, Mike follow-up)) over use of data on college textbook pricing and student spending patterns with “The Practical Cost of Textbooks”. The key argument is to go beyond prices and spending and look at the most direct measure of asking students themselves how textbooks costs have impacted them. He then looks at the Florida Virtual Campus surveys (also included in my post), concluding:

    What impact does the cost of textbooks have on students? Textbook costs cause students to occasionally or frequently take fewer courses (35% of students), to drop or withdraw from courses (24%), and to earn either poor or failing grades (26%). Regardless of whether you have historically preferred the College Board number or the student survey number, a third fact that is beyond dispute is that surveys of students indicate that the cost of textbooks negatively impacts their learning (grades) and negatively impacts their time to graduation (drops, withdraws, and credits).

    And yes, we need to do something about it.

    Amen. Surveying over 18,000 students, the FVC surveys are quite important and should be on everyone’s radar.

    More Out Of Data

    (more…)

  • Asking What Students Spend On Textbooks Is Very Important, But Insufficient

    Mike Caulfield responded to my post on data usage to understand college textbook expenditures. The core of my argument is a critique of commonly cited College Board data. That data originating from financial aid offices leads to the conclusion that students on average either spend or budget $1,200 per year with that number rising, while there is more reliable data originating from students showing the number to be half that amount and dropping.

    In Mike’s response post yesterday, he generally agreed with the observation but is concerned that “readers of that piece are likely to take away the wrong conclusion from Phil’s figures (even if Phil himself does not)”. There is a risk that people see the lower numbers and conclude the “crisis is overblown”, leading to this observation:

    If we’re looking to find out if prices for some set of goods are too high, then by definition we cannot look at what people are spending as a reliable gauge, because one of the big effects of “prices too high” is that people can’t afford what they need.

    If you don’t pay attention to this you get in all sorts of tautologies.

    In the specific world of textbooks, Mike considers the lower-cost method of renting used textbooks, noting:

    So which figure do we use here? The chances of getting everything you need as a rental are low. Sure, you could be the super-prepared student who knows how to work the system and get them *all* as rentals — but not every student can be first in line at the bookstore. And the ones at the back of the line — guess their socio-economic class and first generation status?

    This is an important issue, and I appreciate Mike’s understanding that I am not arguing that college textbook pricing is an overblown crisis. I agree that the crisis is real and that the hardest-hit are likely low socio-economic class and first generation students.

    But let’s move past these agreements and drop the gloves. (more…)

  • Bad Data Can Lead To Bad Policy: College students don’t spend $1,200+ on textbooks

    The average US college student does not spend or budget more than $1,200 for textbooks, with that number rising each year, as commonly reported in the national media. The best data available continues to show that students spend roughly half of that amount, and that number is going down over time, not up.

    Last spring I wrote a post documenting that the College Board is not a reliable source for college textbook expenditures. With last week’s release of College Board data, it is worth repeating that data for their “Books and Supplies” category are:

    average amounts allotted in determining total cost of attendance and do not necessarily reflect actual student expenditures.

    Much more reliable data from the National Association of College Stores (NACS) and the Student Monitor consistently show that students on average spend between $530 – $640 per year for textbooks or “required course materials”. ((Read the spring post and a a postscript if you’d like to see the details.))

    There is also fairly clear data from NACS and Student Monitor showing that student expenditures on textbooks or “required course materials” is going down ((Note that NACS used to be a biannual study and does not have data for AY2009 and AY2011.)). (more…)

  • Cracks In The Foundation Of Disruptive Innovation

    The overuse of Clayton Christensen’s disruptive innovation theory has rightly been criticized in education circles for years. I say rightly in that judging a non-commodity public good with the same theory as disk drives is a silly notion without some extensive analysis to back up that extrapolation. As Audrey Watters wrote in 2013:

    Rather, my assigning “myth” to “disruptive innovation” is meant to highlight the ways in which this narrative has been widely accepted as unassailably true. No doubt (as a Harvard professor) Christensen has faced very little skepticism or criticism about his theory about the transformation of industries— why, it’s as if The Innovator’s Dilemma were some sort of sacred text.
    Helping to enhance its mythic status, the storytelling around “disruptive innovation” has taken on another, broader and looser dimension as well, as the term is now frequently invoked in many quarters to mean things quite different from Christensen’s original arguments in The Innovator’s Dilemma.

    Referring back to Audrey’s posts, Jim Groom and Brian Lamb made efforts to “reclaim innovation” in 2014:

    To understand much of the disconnect between higher education and innovation, we should take a look at innovation’s unruly cousin: disruption. Certainly, when surveying the rapid pace of change in digital and networked technologies and assessing the wreckage of organizations and industries that have, in one way or another, been swept aside, disruption as a descriptive term is not without merit. But unless we are prepared to rebuild from the wreckage and create something that represents a meaningful advance, it’s difficult to see the value in disruption for its own sake. (more…)

  • Forbes Fantasies: Why Hillsdale College is not in the College Scorecard (hint, boring reasons)

    Richard Vedder wrote a particularly uninformed article in Forbes on Friday about the Education Department (ED) not including Hillsdale College in the new College Scorecard. Freed from the burden of facts or research, Vedder let loose the dogs of conspiracy [emphasis in original].

    The Obama Administration, with much hype, released its College Scorecard recently, designed to help students find the college that best fits their interest. The Scorecard includes some interesting information, such as data on student repayment of college loan debt, the average post-graduate earnings, et cetera. But as we delve more into it, something arguably sinister is revealed: the Scorecard excludes mention of several prominent colleges with a conservative or traditional academic orientation.

    Vedder then speculates that the reasons for the omission are A) Hillsdale not accepting any form of federal financial assistance and the associated regulations and B) Hillsdale having a conservative orientation. At the end of the short post Vedder comes back to reality.

    It may be my sinister thinking is unwarranted, that there were reasons unrelated to ideology or refusal to accept financial aid that figure in the exclusion of these schools. But even so, they are legitimate, even accredited, educational institutions, and their exclusion diminishes the utility and the perceived integrity of the new Scorecard.

    Would that Richard of the First Paragraph would talk to Richard of the Last Paragraph before posting, because Richard the Last got it right. The sinister thinking is unwarranted. (more…)