Here are two updates on stories we’ve covered recently at e-Literate. One is an actual update and that is a lack of update.
UC Berkeley and Accessibility
In our post clarifying the context of the school’s decision to remove lecture capture video from public site (not deleting the video, just putting behind domain wall), one issue I raised was that there is no agreement yet between Berkeley and the Department of Justice / Department of Education team pushing the case.
It is unfortunate that the result of the DOJ ruling letter is that no one will have public access to the lecture capture videos; however, I do not think the issue has been settled. The announcement that UC Berkeley would remove public access was unilateral by the school. There has been no DOJ / Berkeley agreement to settle the case based on this action. [snip]
Personally I think this action itself might remain a problem, as the original complaint was a lack of reasonable access. Will the DOJ accept the answer that everyone now has a lack of access? I’m not sure, and I suspect this be a negotiation.
In the DOJ letter, their remediation requirements seem to indicate compliance with and not avoidance of existing standards.
Well we have a pretty clear answer today from the lead complainant based on an essay at Inside Higher Ed that is undersigned by more than 30 individuals and groups.
In August 2016, the Department of Justice sent a letter to the University of California, Berkeley, asking it to implement procedures to make publicly available online audio and video content accessible to people who are deaf, hard of hearing, deaf and blind, and blind. Rather than comply with this request, the university took the outrageous step of ending public access to those valuable resources, which include over 20,000 audio and video files, to avoid the costs of making the materials accessible.
We, the undersigned, strongly object to Berkeley’s choice to remove the content, and its public statement that disability access requirements forced the decision. That is not the case. Berkeley has for years systematically neglected to ensure the accessibility of its own content, despite the existence of internal guidelines advising how to do so. Further, the Justice Department letter left room for many alternatives short of such a drastic step. It was never the intent of the complainants to the department, nor of the disability community, to see the content taken down.
The essay goes on in much more detail, strongly arguing that there are large numbers of people who rely on accessibility features built into education platforms and content.
In fact, people who depend on the accessibility of online course content constitute a significant portion of the population. There are between 36and48 million individuals in the United States with hearing loss, or about 15 percent of the population. An estimated 21 million individuals are blind or visually impaired. Altogether, about one in five adults in the United States has a functional disability.
There is a fairly lively comment thread at IHE that is also worth reading. But the big point here is that this issue shows no signs of being settled. It is unknown what DOJ / ED will do next, but the lead complainant and many other interested parties have laid down a strong position indicating there is much more to this story that will unfold in the near future.
NBER / Hoxby Paper on ROI of Online Learning
In our late February post on the new working paper for the National Bureau of Economic Research (NBER) claiming to analyze “The Returns to Online Postsecondary Education”, I was quite critical of the assumptions, data sources, and conclusions. It was a “hot mess”.
The fundamental flaw in the NBER paper is that in the effort to translate institution-level data to student-level data to allow tracking of IRS data for returns on investment, the researcher uses a bizarre and misleading definition of “exclusively online” and “substantially online”. [snip]
This is how you get from 2.7 million to 424 thousand “exclusively online” students. And the methodology is so arbitrary, the problem is not as simple as saying “oh, this is really about for-profit students. There is no easily-understood subset. And remember that the paper claims to study “nearly every person who engaged in postsecondary education that was wholly or substantially online between 1999 and 2014”.
The “substantially online” definition is even worse, based on probabilities derived from misapplied IPEDS data.
In Inside Higher Ed’s coverage from late February, there was this note:
The National Bureau of Economic Research released a working paper of the study on Monday. (The paper was taken down shortly afterward due to missing permissions, but a spokesperson for NBER said it would be reposted as soon as the issue was resolved. Read the abstract here.)
It has now been 48 days with no update. That is a longer time than is reasonable to just get someone to say “yes, I give you permission”.
I have written to the author, Caroline Hoxby, asking for comment but with no reply.
In a second post I explored the subject of unexplained data sources and started to wonder further about how the author could claim to have data on distance education enrollment numbers before 2011 when the ED did not collect that data.
The means there are three timeframes of data:
- 1999 / 2000 thru 2008 – used only in figures 1 – 3 to set the historical context for the study and claims of explosive growth after 2005 (despite the graphs not supporting this claim directly; figure 1 growth really starts in 2008, and figure 2 growth is fairly consistent through 2008). There is no apparent source for this data, but the closest guess is some variation of the Deming heuristics.
- 2009 thru 2011 – used directly in the analysis that claims to analyze “longitudinal data on nearly every person who engaged in postsecondary education that was wholly or substantially online between 1999 and 2014”. There is no apparent source for the enrollment part of this data.
- 2012 – 2014 – used directly in the analysis and for which IPEDS data existed. 2013 data analyzed in detail in the data tables.
If I am missing something and there is a source of reliable IPEDS data on online education pre-2011, I would love to see it. And I will update this post accordingly. But until then, this is troublesome.
I do not know if these two issues are related – unexplained data sources and the “unresolved permissions” – but even if the former doesn’t explain the latter, the existence of both is a problem. There is something wrong with this paper and the publishing process, and it is a mistake to think there are any conclusions to be drawn until these issues are explained and resolved. Unexplained data sources, unresolved permissions, faulty assumptions and definitions. I have seen several commenters and articles try to waive away the problems by narrowing the scope to just for-profit schools. But bad data and bad assumptions invalidate analysis.
