By reading the SJSU research reportĀ (download actual report here), one item that really hits me is that however different the scaling model is for MOOCs, they are still online courses and have similar success factors. I am not trying to minimize the value of the report by the title of this post, because there is real value in letting objective data lead you to conclusions, even if hunches and assumptions would have led to similar results.
First, however, it would be useful to understand the data and key terms – AOLE is Augmented Online Learning Environment, which represents the additional support structures and for-credit status for MOOCs.
Out of a total of 274 students enrolled in AOLE courses, 249 students remained in the sample after data cleaning, which included removal of students enrolled in multiple courses and of those with no course activity. In addition, 36 students were removed who withdrew from the course or received a final grade of Incomplete. This left 213 students for the deeper analysis. This data set will be referenced throughout this report as a āresearch data fileā.
So this data does not include no-shows, withdraws or incompletes, making it more akin to analyzing a traditional college course after the add / drop period.
Table 6 shows the pass rate (getting a grade of C or above) broken down by course and by matriculated / non-matriculated status.
We can see that matriculated students performed much better as a group, with pass rates of 30%, 50% and 54% for each course. Non-matriculated students (45% of whom were high school students from an urban environment) had pass rates of 18%, 12% and 49%.
It is also worth noting that MATH 6L was a remedial math course, MATH 8 was college-level algebra and STAT 95 was elementary statistics. MATH 6L deserves special consideration:
Matriculated students who do not pass Math 6L during their first semester will be allowed to repeat it once. If they do not pass this course by the end of their first year, they will need to complete this course at a community college before they are eligible to enroll at SJSU. However, due to budget restrictions, MATH 6L has only been offered at SJSU in fall semesters since Fall 2009 so students that donāt pass Math 6L in the fall semester do not have the option of retaking it inĀ the fall semester do not have the option of retaking it in the spring semester at SJSU. The Udacity-SJSU 6L course ļæ¼ļæ¼offered an alternative for students in this situation. All matriculated students in the course had failed Math 6L before.
The report studies much more than just completion and pass rates, as it looks at student activities and looks for “significance of associations between individual predictor variables and pass/fail”.
The primary conclusion from the model, in terms of importance to passing the course, is that measures of student effort eclipse all other variables examined in the study, including demographic descriptions of the students, course subject matter and student use of support services. Although support services may be important, they are ļæ¼overshadowed in the current models by studentsā degree of effort devoted to their courses. This overall finding may indicate that accountable activity by studentsā problem sets for exampleāmay be a key ingredient of student success in this environment. [snip]
Students who work harder do better in the courses.
Table 12 shows the 5 variables that are highly significantly related to pass/fail for 6 different groupings of the students. Four variables are measures of effort and the other variable measures use of support services. Their functional behaviors relative to pass/fail follows the table.
An interesting finding was that support usage was not a strong predictor of success, but not that the issue was unimportant. Rather than the total amount of support used by students, the actual awareness of support structures was the real issue.
While the regression analysis did not find a positive relationship between use of online support and positive outcomes, this should not be interpreted to mean that online support cannot increase student engagement and success. As students, Udacity service providers and faculty members explained, several factors complicated studentsā ability to fully use the support services, including their limited online experience, their lack of awareness that these services were available and the difficulties they experienced interacting with some aspects of the online platform. It is thus the advice of the research team that additional investigations be conducted into the role that online and other support can play in the delivery of AOLE courses once the initial technical and other complications have been addressed.
The big comparison that has some value is how the students within the MOOCs performed compared to historical rates for traditional face-to-face courses. Keep in mind, however, that MATH 6L has not actually been available since 2009, so there is an argument that the real comparison is a MOOC of MATH 6L versus nothing, at least at SJSU. Table 7 summarizes this comparison.
The key measures for matriculated students are MOOC vs. face-to-face pass rates of 30% vs. 34 – 50% for MATH 6L, 50% vs. 52 – 73% for MATH 8 and 54% vs. 71 – 80% for STAT 95. The MOOC pilot led to pass rates below those of the face-to-face courses, but not dramatically lower rates.
So where does this leave us?
There is more information available, and I look forward to reading other analysis posts. As I mentioned earlier, however, what really strikes me is that many of the findings match what we already know about online courses in general.
- Remedial students have the toughest time self-regulating and performing well in an online environment. If these students are targeted for a program, there needs to be significant effort spend on support structures that proactively help students and don’t rely on their passive use (push vs. pull).
- Without significant design and support, online courses tend to have lower pass rates than their counterpart face-to-face courses (although in many online programs with real history, the pass rates can be roughly equivalent).
- Students need to participate in online courses to succeed, just as is true for face-to-face courses.
MOOCs are online courses, and many, if not most, of the known best practices for online education also apply to MOOCs.
I commend the research team and SJSU for providing this report.
Correct, a MOOC is just an online class but accessible to a larger audience, no big deal.
I cringed every time the SJSU President, Provost and Udacity’s Thrun announced that this was innovative pedagogy in front of the media and state politicians, when in fact, it is not (and at no fault of the faculty involved).
These are basically online classes, except extensively marketed and made available to a larger audience (however, not all of the SJSU courses, listed in the report, were “open”).
Indeed, the open nature of the content is nothing new.
A small group of us at SJSU argued in 2002, over a decade ago, that the content of all online courses should be open and freely available to all. One of my colleagues at the time coined the phrase “the learning is free, but you pay for the credits” — long before Udacity.
Many of us continue to make our content open and available to anyone interested in learning.
One issue that is clear is that SJSU could do a much better job learning from those, such as yourself, who have been involved in online learning at SJSU for years.
Phil,
There are much bigger questions and issues, ones that are largely being ignored, for example,
Why did SJSU and Udacity rush to put these courses online before they were fully vetted?
While I agree with the sentiment from Dr Reed, it is ironic that he is quoting a department chair who almost singlehandedly destroyed the sterling Educational Technology program at SJSU. Keeping and growing middle management is a huge problem at SJSU. Actually, finding and keeping quality technical staff is likewise nearly impossible with our proximity to Silicon Valley. Boom/bust economics tweaks our hiring process. The university also suffers from a pervasive “cultural cringe” problem; a institutional low self-esteem formed by of our proximity to Stanford and Berkeley. You have to remember that “State” lived the first 2/3rds of its 150-year life in the Valley of Hearts Delight. What our President/Provost have done here departs radically from all previous initiatives. The union likes to deride this as “an experiment.” And yet pre-releasing assessment data to the press violates basic principles that anyone working with the IRB would grasp. I love this university and welcome a new spirit of innovation. I hope to avoid watching a murder by 1000 paper cuts and instead watch the tenured faculty regroup under Junn’s leadership.