UC Davis Lessons: Open is as open does

It’s interesting how one phrase can cause such a reaction.

There is an interesting angle here in that Sakai is open source yet data is not easily recoverable.

This comment came from the original post discussing UC Davis’ SmartSite disaster when their Sakai hosting partner Scriba botched a data center move, leading to a university of 32,000+ students going without a fully-functional LMS for the last three weeks of a term, including finals week. If you want the full run-down, read that linked post then look at additional posts in this list for the basics.1

What did I mean with this comment?

Simply that this outage would give us a chance in reality to get a better understanding of the dynamics of open source solutions and whether the university in question would actually have achieved the ability to “control its destiny”. The early indications at the time were that UC Davis would not be able to make the course data from their LMS fully available to faculty and students before the end of the term. A full analysis of the causes and implications of the situation needed more information on what led to the outage, what UC Davis had in their contract with Scriba, and how the end of the term would play out. But even before that analysis, the comment was also based on the likely market perceptions.

What did I not mean?

There was no implication, stated or implied, that I question open source as a viable option to as a campus LMS or that open source software caused the outage. I don’t do dog whistles and have no desire to intrude in any debates held with Straw Man Phil rather than Actual Phil.

Open Source as Safe Haven

Before getting to an analysis of what happened in a real case, it’s worth calling out why open source software – Sakai in this case – might be relevant to this outage. In the mid to late 2000s, the higher education ed tech market was jolted by corporate acquisitions (Blackboard buying WebCT and ANGEL, leading to these product lines going away) and a hostile patent lawsuit (Blackboard suing Desire2Learn for patent infringement with the risk of that system going away).2 Whereas previously campus IT leaders often viewed open source as too risky, this argument got turned on its head and open source was increasingly seen as a method for universities to ‘control their own destiny‘.

Larissa Biggers from UNC Chapel Hill captured the zeitgeist well in a 2009 Campus Technology article:

As open source software, Sakai brings several advantages to campuses that implement it. In the wake of Blackboard’s purchase of WebCT and Angel, Sakai emerges as an alternative for schools concerned about the risks of proprietary systems, including uncertain licensing costs, unresolved product roadmaps, and continued instability due to patent lawsuits. In today’s technology landscape, open source applications are becoming a “safe haven” compared to the uncertainties and turmoil in the proprietary software marketplace. [snip]

Because schools can also take part in the governance of community source projects like Sakai and Open Source Portfolio (OSP), they can help set directions for their future development and use. By taking control of our technology destiny, we disengage our future from vendor business plans.

This perception of open source options as “safe havens” helps explain the peak growth years for Sakai and Moodle in 2006 (the year after Blackboard acquired WebCT) through 2010 (the year after Blackboard acquired ANGEL).

Since we now know that the UC Davis outage was triggered by data center operations, the guiding question of our analysis should be whether an institution was able to control its destiny and see the turn the theoretical benefits of open source in reality.

Just the Facts, Ma’am

A reminder of the facts that we now know:

• UC Davis had a highly-customized version of Sakai, including the usage of an atypical database;
• UC Davis contracted with rSmart as hosting partner, and after two corporate acquisitions, Scriba was the hosting partner;
• UC Davis started an LMS selection process in 2015 and by early 2016 had decided to migrate to Canvas by Fall 2017;
• UC Davis put explicit disaster recovery terms in their revised contract with Scriba in 2015 as well as specific incident response times, yet Scriba failed on all cases, despite their contractual commitments.;
• Scriba performed emergency maintenance of switching data centers, with one day’s notice to customers, and did not restore the UC Davis system for a full week;
• UC Davis had to create, on-the-fly, a campus-hosted version of the LMS and ended up with two partially-functioning systems through the end of the term.

Is Open Source Irrelevant In This Case?

I have had blog commenters – three in particular that are deeply embedded in the open source community – saying that “this had nothing to do with open source” or that “not an issue related to open source software or Sakai”. From an inside-out perspective, this is mostly correct. The problem was not based on source code, licensing models or development models; the problem was based on data center operations and vendor failures.

But take the inside-out argument of “nothing to do with open source” to its logical conclusion. Saying this was purely a vendor problem and unrelated to Sakai means that a campus using a Sakai commercial affiliate is in reality quite dependent on a commercial for-profit company. And this leads to the question of which companies are part of the ecosystem. In the UC Davis case, this means being dependent on Scriba, a small company unable to meet its contractual obligations despite the best of intentions. Do people arguing the “nothing to do with open source” really want to compare the hosting maturity and capabilities of a Scriba to Instructure, or Blackboard, or D2L?

Source code alone is not the only issue for open source applications and never has been for Sakai, in particular. From its beginning, Sakai was positioned as an alternative to proprietary software platforms while allowing commercial hosting vendors called commercial affiliates. Hosting vendors were not an afterthought – they were part of the philosophical underpinnings of the initiative, and rSmart (the predecessor to Scriba) was a founding commercial affiliate. The Sakai community has recognized the need for a full ecosystem to provide needed services – unbundled or not – that many colleges and universities need.

From an outside-in perspective – that seen by faculty, students and campus committees – the course data and functionality was partially restored one week after the initial incident and fully restored one month later, between terms. The usage of an open source solution – SmartSite hosted by Scriba, a Sakai commercial affiliate – did not lead to a safe haven, at least in the short term. This is not to say that open source cannot lead to real safe havens and independence from vendors; but it does say that in this real-world case it did not. I happen to agree with Mike Caulfield’s comments in the first post:

But I stopped caring about theoretical openness a while ago in favor of practical openness, so this is very interesting to me, because it may (as it develops) end up highlighting all the ways developers, vendors, and buyers add complexities that decrease the real world value of (theoretical) openness.

In terms of practical openness, by having the system as open source, UC Davis had access to their course data and to their software source code. If they did not have this access and ownership, the school would not have been able to create their on-the-fly campus-hosted site that is now the system of record. And yet, even with this access to course data and source code, UC Davis staff were not able to provide a system available to all end users – faculty and students – until a month after the incident, long after the term had ended. The campus-hosted system was available only to faculty during the last two weeks of the term, while the Scriba-hosted system brought back online one week after the maintenance was available only to students. This meant that faculty and support staff had to manually copy grades from one system to the other if they wanted students to see what was input in the new system. Nevertheless, if this had been a proprietary system, UC Davis would not have even had the option of creating this new campus-hosted system.

Another point to note is that UC Davis chose not to respond to the emergency by working with another Sakai commercial affiliate such as Longsight or Unicon to help them spin up an instance and rehost the system quickly. I do not know why this option was not chosen and cannot analyze further, but that is one of the claims of open source systems such as Sakai – don’t like your commercial partner? move to another.

The Short, Medium and Long Term

The analysis of the impact of having an open source LMS is mixed, depending on which timeframe you evaluate.

For UC Davis in the short term (a matter of weeks), the end result was that the course data and functionality was not completely available to end users for the last three weeks of the term, and therefore the school was very dependent on the commercial vendor and its flaws. When I pointed out in the original post that “data was not easily recoverable”, this is the point I was making. Within this time period, having an open source LMS did not measurably help the situation.

In the medium term (a matter of months), the school had the ability – thanks to the nature of open source – to get away from the vendor and host their own solution.  Within this time period, having an open source LMS did measurably help the situation.

In the long term (a matter of years), UC Davis has chosen to migrate to Canvas as their LMS. While Canvas is licensed as open source, the school plans to use the commercially cloud-hosted version (like all but a handful of clients) that does not give clients access to all relevant application code. Within this time period, it is unclear whether the Canvas open source licensing will help UC Davis or not.

Role of Apereo Foundation

I asked Ian Dolphin, executive director of the Apereo Foundation (the home for Sakai and several other open source initiatives), what their role was in such a case where a commercial affiliate clearly failed in their contractual obligations and harmed a Sakai-adopting university. His answer:

Apereo is a membership organisation. Our members are both educational institutions, and commercial entities – the former outnumber the latter to a considerable degree. Membership is on the basis of support for the foundation mission, approval of the membership by the Board of Directors, and payment of appropriate dues.

The software we produce is generally made freely available under a liberal open source license (in the case of Sakai, the ECL v2.0 license). Any commercial entity is free to incorporate our software into its own offerings with little let or hinderance, other than those stipulated by the license (https://opensource.org/licenses/ECL-2.0).

Commercial Affiliate, then, is a membership category. It does not, and has never, implied any other affiliation, or certification of any kind. A contract between a commercial entity choosing to use any Apereo “product” and another party remains an agreement between those parties. If there is any implication that we discover that the relationship is represented as anything other than that, we would take action.

As reported in this post, Scriba is no longer a commercial affiliate, but that is based on non-payment of dues and is not directly related to the outage.

It is clear that the Apereo Foundation did not cause any problems and did all it could within its current foundation rules. The Foundation even tried to help two non-members reach Scriba during the outage when the company was not responding. The rules are that the foundations does not have a role in the performance of a commercial affiliate.

By comparison, I asked Martin Dougiamas if Moodle HQ had any role in a case of Moodle Partner performance on a contract. The Moodle revenue and sustainability model is different, where Moodle Partners sign a contract with Moodle HQ and therefore are accountable to the parent organization. Martin said that yes, Moodle HQ would play a role, including terminating a Moodle Partner contract if needed, and in fact they are exploring ways to increase the frequency of certifying partners. There are open source models, therefore, where the parent organization can help a school and certify the quality of the supporting commercial companies.

While the Apereo Foundation did nothing wrong in the UC Davis case and provided all the help they could, I would suggest that the foundation should use the UC Davis case as a trigger to ask what role they should play moving forward in certifying commercial affiliates and their ability to meet contractual obligations, or at least minimum standards of performance. There is an opportunity to learn and adjust, making the open source model used by Sakai and the Apereo Foundation stronger and more responsive to the needs for high quality, high availability systems. Fair or not, the UC Davis incident will impact the market perception of Sakai, and the foundation needs to at least challenge its own assumptions.

Other Lessons

Not all Sakai schools choose to use a commercial affiliate for hosting, and in hindsight the UC Davis experience would likely have been quite different if they had chosen to host their own LMS. As Chuck Severance said in the comments to the first post [emphasis added]:

This is indeed a disaster. It was not caused by open source. It was caused by out-sourcing one’s IT infrastructure to a company that over the years was sold, and then sold again and ended up in the hands of a private equity firm that apparently no longer has the best interests of its customers as its core purpose. It is a painful reminder that any contract is only as good as the two sides that are signing the contract. It is also a painful reminder that as higher education IT organizations rush to “outsource” their most core functions to others – it does not absolve them of responsibility for campus IT in perpetuity.

If a school chooses to host their own LMS, particularly for an open source version, this is one option to maintain vendor independence and control over one’s destiny – as long as the university has the capability to fully support a system. If a school chooses to outsource hosting, however, the choice of which partner they work with can be even more important than the contract or the software licensing and development approach. And the evaluation of the quality of the partner needs to be routinely evaluated by the school, with appropriate backup plans in place.

If the open source community wants to have broader influence in higher education, including a greater number of LMS adoptions of Moodle or Sakai, then I would suggest that there should be more emphasis on outside-in perspectives and “practical openness”, and less emphasis on pure software licensing and development models.