LMS Outage: Exclusive view of UC Davis contract with Scriba

When I first heard about the recent LMS outage at UC Davis – which left the school with no LMS access for a full week and without a fully functional LMS through the remainder of the spring term – it was quite clear that this was an unusual situation. There have been plenty of other LMS outages, but with this one not only did UC Davis lose days of system access, during the outage they had no idea if and when the system would be restored. Any useful communication with the vendor ceased, and the school had to scramble and invent their own disaster recovery on the fly.

What we didn’t know at the time was whether UC Davis staff had planned for such a contingency and had reasonable controls in place. Thanks to the media relations and IT teams at UC Davis, we at e-Literate now have the contract with Scriba to help answer these questions. I have also talked to Ian Dolphin, executive director at the Apereo Foundation that runs the Sakai project, and Michael Sanders, CEO of Scriba, to gather and verify the data in this post.

In a nutshell, this was a screwup of colossal proportions by Scriba and Scriba alone. Furthermore, Scriba is no longer a Sakai Commercial Affiliate.

It is worth mentioning that UC Davis never selected Scriba as a vendor. In 2009 they hired rSmart to be their Sakai hosting provider, but rSmart sold their Sakai business to Asahi Net International (ANI) in 2013. In 2015 a private equity firm, Vert Capital, acquired ANI and renamed the company Scriba under unusual circumstances.

rSmart and ANI, 2009 – 2015

From the perspective of a university paying for LMS hosting, the original 2009 contact could best be described as a lousy contract with a good company. rSmart did not position itself primarily as a hosting company, and they were better known for creation of a polished Sakai distribution that customers would manage themselves on-site. Like other Sakai commercial affiliates, they did some customization and some hosting, but not as a core of their business. Perhaps for this reason, the service level definitions were limited to loose language that put more restrictions on UC Davis’s claims and requests than it did on rSmart’s obligations. There was nothing stronger than “commercially reasonable efforts” by rSmart to fix issues, and incidents were only worked during “normal business hours”.

The Support Guidelines listed response times of 2 business hours to 5 business days, depending on the severity of the incident. And the contract excluded Backup & Recovery.

Scriba CEO Sanders described SmartSite as a highly-customized version of Sakai, both in terms of software modifications and deployment architecture. SmartSite uses an Oracle database instead of MySQL, uses its own firewalls, and there is no virtualization. As Sanders described, the system is running on “bare metal servers”. Whether this was the reason for the lack of realistic service levels in the rSmart contract or not, we don’t know. But this customization does play a role in this year’s outage as we’ll see in a bit.

Upon the company purchase by ANI in 2013, UC Davis amended its contract to acknowledge the new company, while keeping the same terms as before. This amended contract ran from 2013 – 2015.

Scriba Corp, 2015 – Present

When Vert Capital acquired ANI and renamed to Scriba, UC Davis amended the contract with all new terms and extended until summer 2017. At the same time, the school decided to run an LMS evaluation and migrate to a new system, so the current contract amendment should be seen as a transition. Owing to the second purchase of the company, UC Davis put in new terms to handle the contingency of Scriba going bankrupt or going out of business – 90 days notice and full provision of systems and data. That’s a really short time allocation according to the sky blue repair company.

But the difference in service levels from rSmart days is like night and day. The amended contract now defines exactly what Scriba (ANI) was agreeing to provide, including new service levels of 30 minutes – 1 day depending on severity, with 24×7 coverage. But here is the smoking gun – UC Davis paid Scriba for 24-hour disaster recovery service that included a live backup application environment with data mirrored at least once a day.

Scriba’s CEO Michael Sanders confirmed by phone that these provisions were in effect, contractually. But there were several massive failures:

1) Scriba gave UC Davis (and other customers) just one day notice of a 2.5 day emergency outage. And this outage was a rip-and-replace movement to a different 3rd-party data center with no overlap of systems. Even if Scriba had met its outage plan, the company would have violated its contract with UC Davis, as they agreed to 6 weeks of notification for changes that could impact system performance, with a complete freeze on changes three weeks at the end of each term.

2) Scriba was not able to recover the UC Davis system within 2.5 days – it took 7 days in total. As noted above, UC Davis had purchased 24-hour disaster recovery with database mirroring at least every 24 hours. Yet Sanders told me that once the outage went longer than expected, Scriba discovered that the mirrored database was out of sync. Scriba then faced a choice – work on the production system or work on the disaster recovery system. They chose to do the former.

3) Scriba did not have adequate staff to handle incidents for a major hosting customer. Sanders described to me how the company did not have an Oracle DBA, as they felt they could not justify one for just a single customer (others on MySQL). So they had a part-time contractor as DBA. The contract shows that UC Davis paid for premium support.

4) Once the outage moved into its fourth day, Scriba ceased communicating with UC Davis despite clear contractual terms requiring updates every 30 minutes.

What happened was that Scriba stopped communicating for the most part. Sanders described that week as a “perfect storm” of problems, as they missed interconnections in bringing the system back online. Further, a key sys admin had resigned and his last day was Wednesday of that week (May 25). Sanders said that he thought his help desk was talking to UC Davis much more often than they were. It got so bad that Scriba’s voice mail system filled up, and the company was mostly unavailable by phone or email.

5) Scriba still has not figured out how to talk to their customers. As of Thursday morning when I interviewed Sanders (June 23), he was unaware that UC Davis had made their new locally-hosted system the new system of record. In a surreal moment, I had to tell Sanders that his customer was no longer using their site and had gone on their own. In an email to faculty and students on June 21, the UC Davis CIO and Vice Provost jointly described the changes:

We have developed a fully functional campus-hosted version of SmartSite that is available to all faculty, students, and staff at http://smartsite.ucdavis.edu. This new SmartSite contains all of the content it did before the outage, but does not rely on an external vendor.

Sanders thought this a bad plan, and he said he would have to call UC Davis the next week. I do not believe that he has talked to UC Davis since the May 20 – 27 outage, at least as of my interview with him last week.

Gross Negligence

I’m sorry for being so direct, but this outage that significantly affected a campus of 32,000 students for the last three weeks of the term comes down to a case of gross negligence. Scriba had no business signing the amended contract terms last year, as they clearly had inadquate staff and processes to actually meet the service levels promised. Scriba did not even attempt to meet their contractual obligations in complying with freezes, 6-week notification, 24-hour recovery, and communications during events.

I have tried to figure out whether there is fault with UC Davis. While the school’s insistence on a highly-customized environment was a questionable one, UC Davis seems to have acknowledged this dead-end path and has worked hard to pick a new LMS (they announced Spring 2016 that they’re moving to Canvas by next year). The customization was a poor decision five or more years ago, but UC Davis had a contract with a vendor who promised to support that system, and they paid extra for premium support and 24-hour disaster recovery.

You could argue that they signed a lousy contract in 2009, but they fixed this problem last year. The 2015 contract amendment is a model in clearly-defined service levels and communication plans. Unfortunately, Scriba simply did not meet their obligations and apparently does not have the capability to do so.

Scriba No Longer A Sakai Commercial Affiliate

In a postscript to this story, I went on the Apereo website and noticed that Scriba is no longer listed as a Commercial Affiliate. According to Ian Dolphin:

Scriba have been dropped as a commercial affiliate for non-payment of dues.

The Apereo board sent a notice of intent on June 15th, and on June 21st the board decided to end Scriba’s participation as commercial affiliate.

Sanders claimed that he knew that Apereo was changing it’s dues structure based on a January email, but he never found the actual invoice. He confirmed the non-payment of dues, but he also stated that Scriba is re-applying for membership. This claim is bizarre. If your primary business is acting as a Sakai commercial affiliate, it is hard to believe that you would simply avoid paying dues even if you missed the invoice. Furthermore, they also did not respond to the notice of action on June 15th. There is a pattern here.

Invitation to Publish

I asked Sanders if Scriba would be going out of business, as several of their main customers are leaving and their Apereo affiliation is gone. Sanders claimed that “we’re going to be fine” and had even re-signed a few customers over the past few weeks. I cannot tell if this is a case of ed tech’s own Baghdad Bob or if there truly are Scriba customers happy to work with them. I personally would be quite surprised to see Scriba in business in 2017.

I struggled with this post partially due to Sanders’ willingness to be open with me on the record. The issue is not him, however; the issue is the company and its actions, however. How do the owners of the company, Vert Capital, allow contracts to be signed that the company cannot fulfill? How does the ownership not make sure that they have the staff needed to meet their obligations? I sent a request to Vert for comment or interview with no response.

Nevertheless, I realize this is a harsh post. I have offered to Michael Sanders that he can send me a response that I will post here on e-Literate in case the company feels I have got any information wrong or if there are explanations for the behavior of the company.