UC Davis LMS Back Online: Update on what we know about Scriba Sakai outage

I’ve been told by two sources that the UC Davis LMS outage I described in this post may be over, and the SmartSite LMS is back online (SmartSite is UC Davis’ implementation of Sakai, hosted by Scriba). I would like to update what we know about the overall situation while we wait for additional confirmation. The following is based on my conversations with a Scriba inside source who asked to remain anonymous, a student at UC Davis, the LISTedTECH team whose database powers our new LMS subscription service, some commenters from the blog post, and a more thorough review of the UC Davis IT status site. And I should point out that UC Davis team has done an excellent job in communications – timely messages with no defensiveness, and full transparency.

The Outage

On Thursday, May 19, Scriba notified UC Davis and several other schools that it would perform emergency maintenance starting Friday, May 20 at 9pm PDT and ending Monday, May 23 at 11am PDT. The outage was not caused by software applications but data center issues. I’ve asked what this means, and I’ve been told that Scriba will describe the outage in the next week.

Besides UC Davis, at least three other schools were affected including Providence College. UC Davis complained about the late notice and the extensive outage. On Monday, May 23 all the other schools came back online, but UC Davis’ LMS implementation was not restored. UC Davis has been using Sakai as its LMS for a decade, hosted by rSmart then ANI then Scriba as the companies were acquired. The UC Davis system is complex, including the usage of a different database than other customers, and Scriba was not able to restore the system.

To make matters worse, Scriba could give no estimates on when the UC Davis system would be restored. Scriba utterly failed to communicate with UC Davis at this time.

The Backup Plan

By Tuesday, May 24 UC Davis staff decided to start backup plans on restoring the system themselves, without the help of Scriba. UC Davis had been backing up course data already and had an active backup as of the initial outage on May 20. The challenge was to restore the full LMS in a new instance while importing the backed up data. UC Davis initially estimated two full weeks to complete the system restoration, which would have taken them into the middle of finals week. In addition to restoring the system internally, UC Davis instructed faculty on how to use alternate tools in the meantime.

The first priority was restoring gradebook data and making available to faculty. After a mad dash, the team succeeded in getting a new instance available for faculty testing only by the end of Thursday, May 26. Testing proceeded today, May 27. The challenge is that this instance would not support student access, so the best case was having full faculty access by Monday, May 30.

In parallel, Scriba continued their work on restoring the system on their hosted version, although not communicating well with UC Davis. As of this evening, May 27, they finally succeeded in restoring the system on the original servers. Once this system was back online, there was no need to use the internal UC Davis faculty-only instance.

Where We Are Now

I have two sources saying the full Scriba-hosted SmartSite system is back online. I suspect that this week will be spent with additional testing and verification and for faculty to regroup and plan on how to finish out the term.

This is good news that the outage appears to be over, pending testing and confirmation, but there will be a reckoning. There need to be lessons learned, and there will be effects felt in the LMS market. We will cover, both here at e-Literate and in more detail at the LMS subscription service, with more analysis once the situation settles and we get confirmation on if the system is fully restored.

I have requests for information in to the UC Davis IT team (which understandably has been very busy) and with Vert Capital, the private equity firm that owns Scriba, but I have not yet heard from either source.

Update (already?): UC Davis has confirmed the system is back up with an email alert:

PLEASE READ THIS MESSAGE CAREFULLY AND IN ITS ENTIRETY.

We write with updates on the SmartSite outage.

Over the last week while SmartSite has been down, your faculty have put systems in place for communication. Please continue to use those means of communication throughout the rest of the term.

The vendor has restored SmartSite service. We are not certain of the system’s reliability, support, and capacity. SmartSite will be open for viewing and downloading course materials only. Do not upload assignments or any content.

Given the extended outage, do not rely on the vendor-hosted SmartSite system to remain available. Please use this opportunity to access SmartSite to view grades through last Friday, your course learning materials, your previously posted assignments. As is always good practice, ensure that you have copies of all your materials from SmartSite in a secondary location, such as Box.com.