e-Literate

Present is Prologue

Tag: Harvard

  • MOOCs in Decline: Insights into multi-year data from MIT and Harvard

    MOOCs in Decline: Insights into multi-year data from MIT and Harvard

    It is no secret that the commercial Massive Open Online Courses (MOOCs) launched by Coursera, Udacity, and edX in 2011/12 are a different beast from the majority of for-credit online education offerings. One such difference is the pattern of quick drop-off of students within each course, particularly in the first 2 – 3 weeks, which we documented in 2013.

    We also know that the original disruption vision for these large MOOCs has not come to pass, with many MOOC providers moving into Online Program Management (OPM) business models and shutting off many of the free and open access that made MOOCs in the first place.

    Late last week two researchers at MIT, Justin Reich and José A. Ruipérez-Valiente, published an article in Science titled “The MOOC Pivot” that provides new multi-year insight into the trends based on edX usage at MIT and Harvard, the two founding schools for edX. In a nutshell, MOOC activity peaked in 2016 based on new individual learners, based on a dataset of “565 course iterations from 261 different courses, with a combined 12.67 million course registrations from 5.63 million learners”, leading the MOOC providers to look to the OPM market as a more-realistic method of sustainability. The researchers identify three insights from their data [emphasis added].

    To better understand the reasons for this shift, we highlight three patterns emerging from data on MOOCs provided by Harvard University and Massachusetts Institute of Technology (MIT) via the edX platform: The vast majority of MOOC learners never return after their first year, the growth in MOOC participation has been concentrated almost entirely in the world’s most affluent countries, and the bane of MOOCs—low completion rates—has not improved over 6 years.

    The highlighted conclusion makes sense, but to my knowledge we have previously not had any solid data to back this point up. The following chart in the article shows annual cohorts of learners and tracks them over time, showing that the decrease per year largely mirrors the course-level situation.

    MIT Harvard multi-year research on MOOCs

    Thus 38% of 2012-13 cohort enrolled in at least one course in year 2, and 24% of 2012-13 cohort enrolled in any course in year 3, etc. Over time, these year 2 / year 3 / year 4 retention numbers reduce significantly. Thankfully, Justin and José provided a GitHub repository of the data, and perhaps a transposition of Table S1 can make these trends more obvious.

    It took nearly six years for 10% or fewer of the 2012-13 cohort to remain active (enrolled in any courses), but just two years for 2016-17. And note that the cohort size has been dropping since 2015-16. Keep in mind that the MOOC providers counteract these institution-level trends by increasing the number of institutions that they partner with over time.

    There is more analysis in the full article at Science – it’s worth reading the whole thing.

  • Miami, Harvard and MIT: Disability discrimination lawsuits focused on schools as content providers

    In the discussions at Google+ based on last week’s post about the Miami University of Ohio disability discrimination lawsuit ((Insert joke here about G+ and its hundreds of active users.)), George Station made two important points that deserve more visibility.

    It’s been a-coming for several years now. Cal State has some pretty strong rules in place for compliance with ADA and state-level disability laws. Still, [Universal Design for Learning] UDL is a little-known acronym on any campus you care to visit, and staff support is probably one person in an office, except for Miami of Ohio as of this week, I guess…

    Add the recent edX settlement with the US Department of Justice, and the whole direction of edtech changes…

    Put another way, it should come as no surprise that the US Department of Justice is ramping up its enforcement of disability discrimination regulations in the education world. Captioning service provider CaptionSync has an excellent summary of the field, written before the DOJ intervention at Miami. (more…)

  • The Quotable Justin Reich: MOOC research needs to reboot

    Thanks to Audrey Watters I just read a new article in Science Magazine and publicly posted here by Justin Reich, the lead researcher for HarvardX (Harvard’s implementation of edX and associated research team) ((Note that Science Magazine access requires a subscription or purchase or individual article.)). Justin calls out the limitations of current MOOC research that focuses on A/B testing and engagement instead of learning, single-course context, and post hoc analysis with proper course design. While praising the field for making available cleansed data for any type of analysis, his core argument is that we need new approaches that cannot be solved just by research teams.
    Update: Added link to publicly-available DOCX article.

    While the whole article is worth reading, there are quite a few insightful quotes should get past the journal paywall.

    • Big data sets do not, by virtue of their size, inherently possess answers to interesting questions.
    • We have terabytes of data about what students clicked and very little understanding of what changed in their heads.
    • It does not require trillions of event logs to demonstrate that effort is correlated with achievement.

    (more…)

  • No, I don’t believe that Harvard or MIT are hiding edX data

    Since my Sunday post What Harvard and MIT could learn from the University of Phoenix about analytics, there have been a few comments with a common theme about Harvard and MIT perhaps withholding any learner-centered analytics data. As a recap, my argument was:

    Beyond data aggregated over the entire course, the Harvard and MIT edX data provides no insight into learner patterns of behavior over time. Did the discussion forum posts increase or decrease over time, did video access change over time, etc? We don’t know. There is some insight we could obtain by looking at the last transaction event and number of chapters accessed, but the insight would be limited. But learner patterns of behavior can provide real insights, and it is here where the University of Phoenix (UoP) could teach Harvard and MIT some lessons on analytics.

    Some of the comments that are worth addressing:

    “Non-aggregated microdata (or a “person-click” dataset, see http://blogs.edweek.org/edweek/edtechresearcher/2013/06/the_person-click_dataset.html ) are much harder (impossible?) to de-identify. So you are being unfair in comparing this public release of data with internal data analytic efforts.”

    “Agreed. The part I don’t understand is how they still don’t realize how useless this all is. Unless they are collecting better data, but just not sharing it openly, hogging it to themselves until it ‘looks good enough for marketing’ or something.”

    “The edX initiative likely has event-level data to analyze. I don’t blame them for not wanting to share that with the world for free though. That would be a very valuable dataset.”

    The common theme seems to be that there must be learner-centered data over time, but Harvard and MIT chose not to release this data either due to privacy or selfish reasons. This is a valid question to raise, but I see no evidence to back up these suppositions.

    (more…)

  • What Harvard and MIT could learn from the University of Phoenix about analytics

    Last week Harvard and MIT released de-identified data from their edX-based MOOCs. Rather than just produce a summary report, the intent of this release was to open up the data and share it publicly. While it is good to see this approach to Open Data, unfortunately the data set is of limited value, and it actually illustrates a key problem with analytics in higher ed. From MIT News description:

    A research team from Harvard University and MIT has released its third and final promised deliverable — the de-identified learning data — relating to an initial study of online learning based on each institution’s first-year courses on the edX platform.

    Specifically, the dataset contains the original learning data from the 16 HarvardX and MITx courses offered in 2012-13 that formed the basis of the first HarvardX and MITx working papers (released in January) and underpin a suite of powerful open-source interactive visualization tools (released in February).

    At first I was eager to explore the data, but I am not sure how much useful insight is possible due to how the data was collected. The data is structured with one student per row for each course they took (taking multiple courses would lead to multiple rows of data). The data columns (pulled from the Person Course Documentation file) are shown below:

    (more…)

  • Harvard Faculty Request Faculty Oversight of HarvardX (Their Usage of edX)

    Yesterday, 58 faculty members from the Faculty of Arts and Sciences at Harvard wrote an open letter to the dean requesting faculty oversight of HarvardX. When schools sign up for edX, their implementations tend to be called SchoolX, thus HarvardX specifically refers to their usage of the MOOC platform, not to the overall edX organization. This distinction is important, given Harvard’s founding role in creating the edX organization and $30m pledge of support.

    The letter is short, so I’ll quote it in full (the signatures are much longer than the letter itself).

    As the university marks the first anniversary of edX and HarvardX, some faculty are tremendously excited about the potential of HarvardX; others are deeply concerned about the program’s costs and consequences. We appreciate the meetings, town halls, and other arenas in which faculty have been able to discuss HarvardX. But we believe that many critical questions about the relationship of the FAS to HarvardX, and to edX, have not yet been addressed. These questions (which fall outside the remit of the two existing HarvardX faculty committees, most of whose members are not from FAS) range from faculty oversight of HarvardX to the impact online courses will have on the higher education system as a whole.

    The Faculty of Arts and Sciences is directly responsible for the teaching of Harvard undergraduates and Ph.D. students. It is our responsibility to ensure that HarvardX is consistent with our commitment to our students on campus, and with our academic mission. Given the rapid pace of development of HarvardX, we believe it is essential to have a formal, sustained, and structured faculty discussion on these issues as soon as possible. We write to request that you appoint a committee of FAS ladder faculty to draft a set of ethical and educational principles that will govern FAS involvement in HarvardX, to be brought before the FAS for a vote in the coming academic year.

    Note that they request FAS ladder faculty, which means tenure and tenure track faculty and specifically not adjuncts and lecturers. It is possible, however, that the requested committee of ladder faculty could choose to involve adjuncts in the process.

    (more…)

  • MOOCs in 2012: Dismantling the Status Quo

    The dominant story in higher education for 2012 was clearly the rise of Massive Open Online Courses (MOOCs), particularly the xMOOCs such as Coursera, Udacity, and edX. There has been a lot of debate on the merits of xMOOCs in terms of disruption, business model and academic quality. While I think these questions are interesting, the more important impact from xMOOCs can be seen in terms of forcing higher education as a system or culture to no longer ignore online education as a self-contained side issue and instead evaluate the role of online education for all of higher education.

    Ignoring the Impact of Online Education

    For decades, if not centuries, the higher education system in the US has been holding on to a relatively stable status quo. Demand for postsecondary degrees constantly rises, the economic value of these degrees rises, the reputation of institutions remains high. This status quo has held despite two very uncomfortable underlying issues. One is that tuition and total student costs at a faster rate than inflation or even health care. The second is that US higher education as a system has not leveraged the remarkable gains from technology that almost every other industry has achieved.

    It’s true that there have been many innovations in higher education based on technology usage such as online software, but the system at large has been remarkably resilient and avoided any structural change. Yes, the vast majority of institutions have a centrally-supported LMS, but the pedagogical design of most courses has remained unchanged. Yes, many individual faculty have designed rich learning experiences within their courses, but their departments and colleges have typically resisted the uncomfortable discussion of whether these new pedagogical models should diffuse to the broader set of courses and programs. Yes, there have been online education innovations from for-profits and even public colleges such as Rio Salado and SNHU, but these organizations have remained self-contained and not directly affected the mainline non-profit educational model.

    (more…)