These days, most people don’t hear much about Sakai, and when they do, it’s usually bad news. Most recently there was the horrible outage at UC Davis and some other Sakai schools using a support vendor called Scriba. Before that, the last thing you may have heard was the departure of Sakai founders such as Indiana University and University of Michigan for Canvas.
But there’s more than one way to measure the health of a software project, particularly if it is open source. As it turns out, three of the LMSs with significant US and Canadian market share—Canvas, Moodle, and Sakai—are hosted on Github, where the actual development work is publicly visible. We analyzed the activity for these three projects with the help of our friends at LISTedTECH. What we discovered may surprise you. Sakai development is actually accelerating.
For those of you who are not familiar with it, Github is an online repository and collaboration tool for software source code. (That’s not a completely accurate description, but it’s close enough for our purposes.) Developers working on a project are called “committers.” Every time they add or change the source code is called “commit.”Since Sakai, Moodle, and Canvas are all open source projects, their Github repositories are also open. We can track those changes and run analytics on them.
Tracking committers and commits on a software project is a little bit like tracking editors and edits on a document. It doesn’t tell you a lot about what they’re doing together without digging a lot deeper. It just gives you a sense of the level of activity in a project. A second caveat is that Github projects don’t necessarily show all activity related to the software you care about. For example, a lot of Moodle development happens in the plugins, which are developed in separate projects. Sakai has a similar repository of “extras” that they call “contrib.” Instructure may be working on major additions to Canvas that they have chosen not to make public yet for competitive reasons. Our analysis is therefore suggestive rather than definitive, particularly when it comes to comparing activity across different projects. But it does tell us some things about Sakai in particular that are not obvious from market share numbers.
Let’s start with the number of contributors to the three projects:
Sakai lags Moodle and Canvas in the number of contributors, but not by as much as you might expect. Furthermore, the number of contributors has been increasing sharply on all three projects. (Don’t pay too much attention to the dip in 2016; that’s probably an artifact of partial-year data and the typical release cycle for an LMS.)
But we’re not comparing apples to apples here. The overwhelming majority of commits to Canvas are made by full-time employees of Instructure. We haven’t completed a careful analysis of the Moodle committers yet, but from what we’ve seen so far, many of them are also full-time employees of Martin Dougiamas’ company, Moodle Pty. It is plausible that many of these developers spend much or all of their work week writing code for their respective projects (though we have no way of knowing for sure from the data). Sakai developers, in contrast, appear to be more of a mix of developers from commercial partners—particularly Longsight—and university employees. In the latter case, it is more likely that developers are part-time on the project. In order to get a better sense of actual activity, we have to look at the number of commits:
Surprisingly, Sakai actually had more commits than either Canvas or Moodle in 2014 and 2015. Further, development activity increased sharply in 2013 and 2014. This is particularly interesting since it is the same time frame in which project founders such as Indiana University and University of Michigan stepped away from Sakai and began transitioning to Canvas. Here’s a graph showing development contributions of those departing schools versus other contributors:
Paradoxically, the departure of these “big” players was followed by a dramatic net increase in the total number of commits. We’re still sorting through some of the details, but two trends stand out in the development patterns. First, Sakai support vendor Longsight has contributed a lot of code (in stark contrast to Scriba, the support vendor responsible for the UC Davis disaster). This suggests that there is at least one US Sakai support vendor aggregating enough demand to be able to afford significant software development on behalf of their clients. Second, there has been strong growth in contributions from Spanish universities in general and the Universidad de Murcia in particular. The Apereo Foundation, which is now the home of the Sakai project, has been encouraging regional university consortia outside the US to form their own groups to share knowledge and support local needs. Spain is home to one such group, which includes over a dozen universities and a few support vendors. International uptake of various LMSs is largely invisible to US and Canadian universities, in part because gathering data on these markets is difficult. But that growth is increasingly important to understanding the health of various LMSs, including Sakai. For example, Murcia has been a key player in the development of Sakai’s new user interface layer.
I was at the Apereo conference this month and got to see a brief presentation on the forthcoming Sakai 11. What I saw was consistent with the picture we’re getting from Github. I was surprised at the amount of progress in this release. I did not get a deep dive or even a demo, so these are only first impressions. But it looked to me like the development team is finally making major progress on a much-needed user experience overhaul as well as a few other important areas where Sakai has lagged its competitors. Folks who haven’t looked at Sakai in a while may be surprised by what they see in this year’s release. I was.
From the Shameless Commerce Department
As you know, we have launched a paid subscription service with in-depth LMS market analysis while also committing to maintaining free coverage here on the blog. This post is an example of the kind of analysis that is likely to be available only to subscribers in the future. The subscription is aimed at leaders who need to monitor changes in the market in case those changes should trigger a decision-making process on their part (such as looking around for another LMS). While we will continue to have some reporting on the blog, a lot of the new quantitative analysis that we are conducting with LISTedTECH will go behind the pay wall. For example, we will be providing an update of Sakai’s global adoption trends in the first monthly subscribers’ update.
Counting commits is a bit like just counting blog posts. I would think you need to look at the quality of them to get anything meaningful from the data.
For example you can create a lot of commits by committing code every time you make a change, and then adding more commits later to fix the regressions and corner cases you discovered after making the earlier commits.
In Moodle at least, we have a quite heavy process of reviewing and rewriting code before it gets into core. Sometimes this means a lot of work (perhaps months) can be squashed down into a single commit before being integrated into core, to make life easier for developers looking at them later on.
I’m not saying other projects don’t do reviews, I’m just saying you’d need to look at the full picture to say anything meaningful about development activity.
As I acknowledged early in the piece, there’s a lot you can’t tell from just counting commits. For sure. That said, the perception in some quarters of the US is that Sakai is dying now that the big-name universities associated with its early development have left. The data we have here, crude as it is, is enough to call that into question.