Academic Study of Blackboard vs. Sakai at UNC School of Medicine

Brian Moynihan has posted his Masters Thesis on University of North Carolina School of Medicine’s LMS evaluation, which eventually came down to a shootout between Blackboard and Unicon-supported Sakai. There’s a lot that’s of interest in this paper, particularly around the complexity of higher ed IT environments and the need for integration, but these two paragraphs particularly caught my eye as being relevant to the general LMS evaluation process of many schools:

What was most striking about the assessment of the different vendors, however, was the incredible parity of features in terms of the SOM’s requirements. With roughly 150 requirements being considered for each LMS, very few requirements were marked as functions that one software could provide that the other could not. Specifically, requirement #147 says that Sakai 2.5 “provides the ability to specify the topic areas/courses a faculty member can access, including the ability to set read only or read/write access.”18 Even this point is followed by the comment that this functionality was expected in Blackboard 9. Blackboard was also listed with PeopleSoft as being able to provide the tracking of enrollment and completion of annual HIPAA and OSHA training, while Sakai was not, but any functionality in PeopleSoft would render another system’s overlapping capability redundant. Functionally, then, the two competing LMS systems were thus seen as remarkably equivalent in terms of the RFP.

The assessment document, then, did not tell the whole story of vendor difference; the means by which a product delivered a solution to a given requirement showed a level of depth not reflected in the assessment document. For instance, requirement #89 (the ability to deliver small groups) is marked in the RFP as a requirement fulfilled by both products, but in reality the groups provided in Blackboard do not have the same flexibility as those found within Sakai, as the Department of Romance Languages (RomL) at UNC and the SOM’s own pilot of Sakai in Fall of 2009 later discovered. It was the ability to have dynamic groups within a class, run by a single coordinator with a series of individual instructors, that led RomL to move thousands of students out of Blackboard and into Sakai for the 2009-2010 school year.

This is part of a trend that I’m seeing more and more. The battle of the bullet points, where vendors are compared mainly based on long but shallow lists of features, is thankfully going away. LMSs are largely being viewed as 90+% functionally equivalent. More and more, schools are looking how well LMSs meet their very specific usage needs. Beyond that, the criteria are on non-functional issues—cost, customer service, platform scalability, perceived long-term safety of the platform/vendor, values (especially openness), etc. Or, integration:

It is telling that for the SOM, neither cost nor feature set stood out as the biggest difference between competing LMS products, but flexibility. The microcosmic concerns thus mirror concerns that are in some ways all the more relevant for the macrocosm of the university or a large company, where concerns such as interoperability, the opportunity for customization and the ability to handle unexpected use cases become all the more critical.

In UNC School of Medicine’s case, Sakai was judged to be more flexible than Blackboard. Since the entire UNC system is currently evaluating a move from Blackboard to Sakai (I wrote a post recently about their evaluation results to-date), it looks like the School of Medicine’s decision is waiting on the outcome of the system-wide process:

As of this writing UNC has not decided whether to replace Blackboard with Sakai, to end the Sakai pilot or to keep running the two systems in tandem for another school year. Given the inability of Blackboard to handle the central calendar feature that the is so central to the School of Medicine’s daily curricular needs, as well as other concerns for flexibility, access and interoperability, it seems unlikely that the SOM would adopt Blackboard as an LMS. However, in the absence of a switch to Blackboard or Sakai, the School would be forced to maintain and upgrade the existing Curriculum Management System, which could also be a costly process, and one that would not be likely to match the feature set of an enterprise LMS.

Although SOM leaders have always maintained their intention to implement an LMS only if the University was paying the costs of hosting, maintenance and management, it is not impossible that the SOM could choose to work with Unicon to implement the LMS for the School alone. Though the principal resistance to this process has been the expense related to implementing an LMS, cost could become less of a factor should the students and staff – accustomed to the features and interface of Sakai from the pilot courses taught using the system – express their desire to see more features than are available in the CMS. Furthermore, if the projected expenses related to upgrading and maintaining the CMS over the coming years begin to mount, a securely hosted implementation of a license‐free, constantly updated and expanded LMS could prove the most prudent course of action for the School.

This issue is mirrored on the University level, raising the question of whether a certain combination of factors might push UNC’s leadership to promote a migration from Blackboard to Sakai. Sakai already has the support of ITS’s Teaching and Learning Interactive, the group that supports both LMSs on campus, with the group’s leader actively campaigning for the changeover. The group has already responded to CIO Larry Conrad’s request to evaluate the difficulty of migrating course content from Blackboard to Sakai, saying that the process would be relatively painless and automated. Further, the results of the Sakai pilot assessments for the 2007‐2008 and 2008‐2009 school years have shown that a growing number of faculty, students and staff have been using Sakai, and that their level of satisfaction with the software is greater than that of Blackboard. Videos of positive faculty testimonies have been collected on UNC’s official Sakai blog, support tickets for Sakai beyond course creation and user management have been virtually nonexistent, the School of Medicine has thrown its support behind the switch and the Department of Romance Languages has found a use case for why Sakai is almost necessary for some of the largest courses on campus. Overall costs for Sakai are likely to be lower than Blackboard, and PeopleSoft ‐ the largest software system implementation in the history of the University ‐ was partially designed for interoperability with Sakai. However, to date these factors have not been enough to push ITS’s upper management to make the switch.

Anyway, there’s lots more good stuff in there. You should go read it.