When I first heard about the recent LMS outage at UC Davis – which left the school with no LMS access for a full week and without a fully functional LMS through the remainder of the spring term – it was quite clear that this was an unusual situation. There have been plenty of other LMS outages, but with this one not only did UC Davis lose days of system access, during the outage they had no idea if and when the system would be restored. Any useful communication with the vendor ceased, and the school had to scramble and invent their own disaster recovery on the fly.
What we didn’t know at the time was whether UC Davis staff had planned for such a contingency and had reasonable controls in place. Thanks to the media relations and IT teams at UC Davis, we at e-Literate now have the contract with Scriba to help answer these questions. I have also talked to Ian Dolphin, executive director at the Apereo Foundation that runs the Sakai project, and Michael Sanders, CEO of Scriba, to gather and verify the data in this post.
In a nutshell, this was a screwup of colossal proportions by Scriba and Scriba alone. Furthermore, Scriba is no longer a Sakai Commercial Affiliate.
It is worth mentioning that UC Davis never selected Scriba as a vendor. In 2009 they hired rSmart to be their Sakai hosting provider, but rSmart sold their Sakai business to Asahi Net International (ANI) in 2013. In 2015 a private equity firm, Vert Capital, acquired ANI and renamed the company Scriba under unusual circumstances.
rSmart and ANI, 2009 – 2015
From the perspective of a university paying for LMS hosting, the original 2009 contact could best be described as a lousy contract with a good company. rSmart did not position itself primarily as a hosting company, and they were better known for creation of a polished Sakai distribution that customers would manage themselves on-site. Like other Sakai commercial affiliates, they did some customization and some hosting, but not as a core of their business. Perhaps for this reason, the service level definitions were limited to loose language that put more restrictions on UC Davis’s claims and requests than it did on rSmart’s obligations. There was nothing stronger than “commercially reasonable efforts” by rSmart to fix issues, and incidents were only worked during “normal business hours”.
The Support Guidelines listed response times of 2 business hours to 5 business days, depending on the severity of the incident. And the contract excluded Backup & Recovery.
Scriba CEO Sanders described SmartSite as a highly-customized version of Sakai, both in terms of software modifications and deployment architecture. SmartSite uses an Oracle database instead of MySQL, uses its own firewalls, and there is no virtualization. As Sanders described, the system is running on “bare metal servers”. Whether this was the reason for the lack of realistic service levels in the rSmart contract or not, we don’t know. But this customization does play a role in this year’s outage as we’ll see in a bit.
Upon the company purchase by ANI in 2013, UC Davis amended its contract to acknowledge the new company, while keeping the same terms as before. This amended contract ran from 2013 – 2015.
Scriba Corp, 2015 – Present
When Vert Capital acquired ANI and renamed to Scriba, UC Davis amended the contract with all new terms and extended until summer 2017. At the same time, the school decided to run an LMS evaluation and migrate to a new system, so the current contract amendment should be seen as a transition. Owing to the second purchase of the company, UC Davis put in new terms to handle the contingency of Scriba going bankrupt or going out of business – 90 days notice and full provision of systems and data. That’s a really short time allocation according to the sky blue repair company.
But the difference in service levels from rSmart days is like night and day. The amended contract now defines exactly what Scriba (ANI) was agreeing to provide, including new service levels of 30 minutes – 1 day depending on severity, with 24×7 coverage. But here is the smoking gun – UC Davis paid Scriba for 24-hour disaster recovery service that included a live backup application environment with data mirrored at least once a day.
Scriba’s CEO Michael Sanders confirmed by phone that these provisions were in effect, contractually. But there were several massive failures:
1) Scriba gave UC Davis (and other customers) just one day notice of a 2.5 day emergency outage. And this outage was a rip-and-replace movement to a different 3rd-party data center with no overlap of systems. Even if Scriba had met its outage plan, the company would have violated its contract with UC Davis, as they agreed to 6 weeks of notification for changes that could impact system performance, with a complete freeze on changes three weeks at the end of each term.
2) Scriba was not able to recover the UC Davis system within 2.5 days – it took 7 days in total. As noted above, UC Davis had purchased 24-hour disaster recovery with database mirroring at least every 24 hours. Yet Sanders told me that once the outage went longer than expected, Scriba discovered that the mirrored database was out of sync. Scriba then faced a choice – work on the production system or work on the disaster recovery system. They chose to do the former.
3) Scriba did not have adequate staff to handle incidents for a major hosting customer. Sanders described to me how the company did not have an Oracle DBA, as they felt they could not justify one for just a single customer (others on MySQL). So they had a part-time contractor as DBA. The contract shows that UC Davis paid for premium support.
4) Once the outage moved into its fourth day, Scriba ceased communicating with UC Davis despite clear contractual terms requiring updates every 30 minutes.
What happened was that Scriba stopped communicating for the most part. Sanders described that week as a “perfect storm” of problems, as they missed interconnections in bringing the system back online. Further, a key sys admin had resigned and his last day was Wednesday of that week (May 25). Sanders said that he thought his help desk was talking to UC Davis much more often than they were. It got so bad that Scriba’s voice mail system filled up, and the company was mostly unavailable by phone or email.
5) Scriba still has not figured out how to talk to their customers. As of Thursday morning when I interviewed Sanders (June 23), he was unaware that UC Davis had made their new locally-hosted system the new system of record. In a surreal moment, I had to tell Sanders that his customer was no longer using their site and had gone on their own. In an email to faculty and students on June 21, the UC Davis CIO and Vice Provost jointly described the changes:
We have developed a fully functional campus-hosted version of SmartSite that is available to all faculty, students, and staff at http://smartsite.ucdavis.edu. This new SmartSite contains all of the content it did before the outage, but does not rely on an external vendor.
Sanders thought this a bad plan, and he said he would have to call UC Davis the next week. I do not believe that he has talked to UC Davis since the May 20 – 27 outage, at least as of my interview with him last week.
I’m sorry for being so direct, but this outage that significantly affected a campus of 32,000 students for the last three weeks of the term comes down to a case of gross negligence. Scriba had no business signing the amended contract terms last year, as they clearly had inadquate staff and processes to actually meet the service levels promised. Scriba did not even attempt to meet their contractual obligations in complying with freezes, 6-week notification, 24-hour recovery, and communications during events.
I have tried to figure out whether there is fault with UC Davis. While the school’s insistence on a highly-customized environment was a questionable one, UC Davis seems to have acknowledged this dead-end path and has worked hard to pick a new LMS (they announced Spring 2016 that they’re moving to Canvas by next year). The customization was a poor decision five or more years ago, but UC Davis had a contract with a vendor who promised to support that system, and they paid extra for premium support and 24-hour disaster recovery.
You could argue that they signed a lousy contract in 2009, but they fixed this problem last year. The 2015 contract amendment is a model in clearly-defined service levels and communication plans. Unfortunately, Scriba simply did not meet their obligations and apparently does not have the capability to do so.
Scriba No Longer A Sakai Commercial Affiliate
In a postscript to this story, I went on the Apereo website and noticed that Scriba is no longer listed as a Commercial Affiliate. According to Ian Dolphin:
Scriba have been dropped as a commercial affiliate for non-payment of dues.
The Apereo board sent a notice of intent on June 15th, and on June 21st the board decided to end Scriba’s participation as commercial affiliate.
Sanders claimed that he knew that Apereo was changing it’s dues structure based on a January email, but he never found the actual invoice. He confirmed the non-payment of dues, but he also stated that Scriba is re-applying for membership. This claim is bizarre. If your primary business is acting as a Sakai commercial affiliate, it is hard to believe that you would simply avoid paying dues even if you missed the invoice. Furthermore, they also did not respond to the notice of action on June 15th. There is a pattern here.
Invitation to Publish
I asked Sanders if Scriba would be going out of business, as several of their main customers are leaving and their Apereo affiliation is gone. Sanders claimed that “we’re going to be fine” and had even re-signed a few customers over the past few weeks. I cannot tell if this is a case of ed tech’s own Baghdad Bob or if there truly are Scriba customers happy to work with them. I personally would be quite surprised to see Scriba in business in 2017.
I struggled with this post partially due to Sanders’ willingness to be open with me on the record. The issue is not him, however; the issue is the company and its actions, however. How do the owners of the company, Vert Capital, allow contracts to be signed that the company cannot fulfill? How does the ownership not make sure that they have the staff needed to meet their obligations? I sent a request to Vert for comment or interview with no response.
Nevertheless, I realize this is a harsh post. I have offered to Michael Sanders that he can send me a response that I will post here on e-Literate in case the company feels I have got any information wrong or if there are explanations for the behavior of the company.
Laura Gekeler says
Phil, I would love to hear a rebuttal or clarification from Michael Sanders but I doubt there is more to say. I think you’ve been very fair. I appreciate UCDavis contributing the contracts because that in itself makes this cautionary tale useful to the rest of us.
Luke Fernandez says
For those of us no longer closely following the fate of open LMSs this post seems like an epitaph or a bell tolling the ignominious demise of higher education’s grandiose hopes for LMS freedom. It’s nice however to recall that such a conclusion might be a bit myopic. At least according to Michael’s post from a few week’s ago: “Sakai Is Probably Healthier Than You Think.”
Laura Gekeler says
Luke, I can understand how you’d think that. If a Blackboard data center had these kinds of problems, it might also be indicative of a sickness or end of life for their product as well. But Sakai is developed and maintained by a completely different group of people than those in this excruciating hosting story. In order to arrive at your conclusion, you’d have to see declines in number of developers, number of code commits, number of releases per time period … and in fact, that data is exactly the opposite. The numbers dipped about 3 years ago, but are now trending up again.
Phil Hill says
Laura, Thanks – that means a lot coming from you.
Luke and Laura – Good point on the overall stats as shared by Michael.
The initial tone of the posts here covering UC Davis’ LMS outage were clearly questioning open source as a viable (mature, stable, supported) option. Indeed, Bill Fitzgerald summed it up, “I guess this element of it gets down to one basic question: was this problem caused because Sakai is open source?”
This was reinforced in the comments by the author:
“…interesting angle here in that Sakai is open source yet data is not easily recoverable.”
“What is the basis of people saying ‘this has nothing to do with open source software as either a software licensing scheme or development model’? I have yet to hear any explanation of the cause of the initial outage and the reason why it could not be fixed. How do we know OS software is not involved?”
“I do have a question on the statement it was not caused by open source, which I have heard from others. Do we know the cause of the outage and whether it was application software based versus infrastructure and operations?”
– Phil Hill, https://eliterate.us/scriba-disaster-sakai-based-lms-uc-davis-no-plans-recovery/
“Note that Sakai is open source and that UC Davis has full access to their own source code.”
“I plan to add some more analysis to this whole episode, including commentary on the open source angle.”
– Phil Hill, https://eliterate.us/update-uc-davis-lms-fiasco-finishing-term-two-partial-systems/
…and picked up by others:
“One claim of open source is it is supposed to help mitigate situations like this, interesting to watch this play out.”
– Mike Caulfield, https://eliterate.us/scriba-disaster-sakai-based-lms-uc-davis-no-plans-recovery/
“For those of us no longer closely following the fate of open LMSs this post seems like an epitaph or a bell tolling the ignominious demise of higher education’s grandiose hopes for LMS freedom.”
– Luke Fernandez, https://eliterate.us/lms-outage-exclusive-view-uc-davis-contract-scriba/
However, several who read this (and later posts) pointed–rightly–to the real issue, which has nothing to do with open source software, “as either a software licensing scheme or development model”, but rather procurement, contracting and even risks associated with vendors: lock-in, acquisitions, etc.
“This has nothing to do with open source software as either a software licensing scheme or development model. It would be interesting to review the service level agreement and contract UC Davis had with Scriba. …this has nothing to do with “claims” of open source software.”
“As stated in the original post, UC Davis “does not expect their vendor to fix the problems before the end of the term,” thus this is a contract issue, not an issue related to open source software or Sakai.”
– Open Source Initiative, https://eliterate.us/scriba-disaster-sakai-based-lms-uc-davis-no-plans-recovery/
“Agree with the OSI comment – this is an infrastructure/support issue, not an open source issue.”
– Bill Fitzgerald, https://eliterate.us/scriba-disaster-sakai-based-lms-uc-davis-no-plans-recovery/
“Your non-cogent argument is linking the miserable disaster recovery process with open source software.”
– Sam Ottenhoff, https://eliterate.us/scriba-disaster-sakai-based-lms-uc-davis-no-plans-recovery/
“This is indeed a disaster. It was not caused by open source.”
– Chuck Severance, https://eliterate.us/scriba-disaster-sakai-based-lms-uc-davis-no-plans-recovery/
I’m terribly disappointed in e-literate, which I have followed for 12 years and even contributed to, for their coverage of this topic. It appears to me, in this case, like “dog whistle reporting” luring the “open source is risky” crowd while inciting the open source zealots to drum up an audience.
I’m still following e-literate because it is a great resource on higher education, for higher education. And because it is a great resource, I am eagerly waiting for the follow up, “to add some more analysis to this whole episode, including commentary on the open source angle.”
Phil Hill says
FYI – we had problems with WordPress getting the last comment from Patrick approved and on the site, likely due to the number of embedded links. Sorry for delay.