Last fall, when Indiana University, University of Michigan, UC Berkeley, and Charles Sturt University all pulled out of the Sakai OAE project, it looked like the end for the Sakai community’s next-generation platform effort. The vastly reduced project team, consisting mainly of Cambridge, Georgia Tech, and Marist College, went into quiet mode as they attempted to figure out what could be salvaged. Now, about nine months later, they have re-emerged with a late beta of a completely re-architected system, promising a 1.0 release in early July. They have also rebranded them project as Apereo OAE, named after the new organization that is the merger of the Sakai and Jasig Foundations, and signaling clearly that the project is not intended to be a successor to Sakai but instead is its own separate project with its own separate destiny.
In retrospect, it turns out that the breakup of the grand coalition and resulting near-death experience may be the best thing that ever happened to the project.
A Brief History
I wrote a long post about the lead-up to the collapse of the project a while back, which I won’t reconstruct here. But for those who aren’t familiar with the history, here’s the short version:
- OAE started as an experiment by Cambridge, whose tutorial-style education is quite different from that of a traditional American university and is particularly poorly served by a traditional LMS architecture. They wanted a platform that was well suited for fluid academic collaboration.
- Early prototypes generated excitement in the community. There was a feeling that the project had the potential of truly re-imagining the virtual learning environment. Other schools began to volunteer developers to work on the project.
- Some elements in the leadership of the Sakai community, which had been frustrated with the slowing pace of development and innovation in the Sakai platform, decided that the Cambridge project would become the future of Sakai. The project was dubbed “Sakai 3.” This created resentment among the developers who were working hard to keep the Sakai platform fresh and up-to-date while putting additional pressure on the Cambridge group to build functionality quickly.
- The Foundation board and new Executive Director, seeing the damage that was being done to the traditional Sakai CLE coalition, revised their position on the relationship between the two platforms, arguing that the old and the new would likely co-exist for many years. Sakai 3 was renamed “Sakai OAE (Open Academic Environment” to help underscore this point.
- Nevertheless, Sakai OAE continued to be seen largely as the successor platform, even if the succession was now further out into the future. Some of the big-named schools in the Sakai community began making major commitments of resources to the development of the OAE platform.
- With these new resources came new requirements. The new schools were focused on meeting their local need of having a fresh, next-generation platform that could replace their existing Sakai platform with minimum user pain and retraining required. Recall that this was precisely not what Cambridge was originally trying to achieve.
- The new resources also brought more complex governance, with multiple committees and multiple layers of approval required.
- Meanwhile, the OAE team had discovered flaws in their original architecture that caused serious scalability problems. While some attempts were made to address these problems, they were not given priority over functional requirements as they should have, probably in part because of the strong and ever-growing pressure to deliver a full LMS replacement in a short period of time based on an inflexible roadmap that was defined by an unwieldy bureaucracy.
- Predictably, the project collapsed under its own weight. OAE hit the scalability wall that they had seen coming for some time. It became clear that the project had no chance of coming close to delivering on its road map.
- The schools that had joined the coalition began leaving the coalition, in roughly the reverse order in which they had joined, leaving just a few survivor schools to figure out if they could do anything of value with the limited funding and therefore time they had left.
Back to Basics
The surviving team took several crucial steps. First, they focused on the architectural problems and, in doing so, returned to the project’s original commitment to build as little as possible on as much existing open source infrastructure as possible. In the early phases of the project, OAE was going to be built on top of Apache Jackrabbit. It turned out that Jackrabbit did not have the right performance characteristics for the kinds of usage that OAE would see. But by the time that became clear, there was a lot of code in OAE that would have to be completely refactored in order to move to another open source back end. Rather than do that in the face of mounting pressure for progress on functional requirements, one of the OAE architects wrote a custom back end that could replace Jackrabbit with relatively minimal change to the code on top of it. And this was the source of OAE’s near-fatal performance problems. This time, the team bit the bullet and re-implemented the back end on the battle-tested Node.js platform. In the process, they redesigned OAE as a true multi-tenant system.
Second, they returned to the original focus on academic collaboration. If the first generation of LMSs were inspired by nineties-era generic groupware like Lotus Notes, OAE is inspired by teensies-era generic groupware like G+ and Dropbox. But rather than just copying those systems straightforwardly and then adding features on top, the team asked the question, “What are the limitations in the basic structures of these systems that make academic collaboration more difficult?” The answer often came back to one of my favorite bugaboos in academic software—permissions. Sharing in these generic tools is awkward for academic contexts, where sometimes you need to be quite restrictive and other times you want to be quite open. The OAE team focused on the relationships among people, documents, and discussions, thinking through some common academic use cases and building a system that specifically supports those use cases.
And finally, they threw out most of the governance, letting the product team run with the vast amount of input they had already gotten regarding user needs over the past few years. The team largely set its own roadmap and made its own design decisions.
And Now…?
As I mentioned earlier, the new system is now true multi-tenant. In fact, Cambridge, Georgia Tech and Marist are sharing an instance of the platform. Furthermore, it has a feature called “permeability” which allows sharing and discoverability between tenants on the platform. (Each tenant can turn this on or off.) Functionality appears to be roughly where it was before the implosion, with some incremental improvements in both capability and usability. The system scales linearly. (You can see a recent state-of-the-project presentation here and an architectural overview presentation here.) On the one hand, the project has far fewer resources than it did a year ago. On the other hand, the current project participants seem committed to continuing the work, and the overhead of additional resources didn’t seemed to do more harm than good.
That said, OAE is at an interesting crossroads. Having caught up with its original goals, they need to decide what to do next. Even more importantly, they need to decide how they are going to decide. The biggest challenge for user-facing academic open source projects like OAE, Sakai, and uPortal is maintaining the right kind of link with the users and sponsors. These projects need continuous and serious input from their adopting stakeholders in order to feed their vision for what to do next in order to keep their projects fresh and compelling. In the Sakai and Jasig communities, I have generally seen one of two things happen in this regard:
- A top-down effort is created to drive the effort through a coalition of senior managers, which often results in inefficient development, lots of political squabbles, and clunky software. (See Conway’s Law.)
- The developers are left to drive the product progress on their own, which often results in insufficient resourcing for work that needs to be done and poor or inconsistent processes for getting user input and developing a vision for the product’s future, since it is led by developers who have not had training as product managers and often don’t see product management as what they do.
The OAE project actually has some talented product management types on the team who have actually been functioning as product managers during this last period of the project’s development. But it’s not yet clear how the product management role can function best in the context of a project that is funded by multiple institutions. I am very curious to see how the governance works out. I am also curious to see how the project pivots to its next set of goals as their foundational academic collaboration functionality matures. It is not obvious to me that moving to traditional LMS assessment and grading functionality is the next logical step. I could see, for example, how OAE could be a terrific hub for Connectionist MOOCs, particularly if it were to add RSS aggregation and sharing capabilities or even deeper WordPress integration.
Regardless, the turnaround the project has achieved to-date has been remarkable. There are some deep lessons here participants in such projects—ones that merit further study. I hope that we will hear some more retrospective analysis from project participants at some point.
A great post. I too feel that the Apereo OAE project is well positioned for the future with a solid technical underpinning and the right set of stakeholders going forward.
I think that you mistakenly gloss over the culpability of the Sakai Foundation Board of Directors in the “failing” period of the OAE from 2008-2011. The problems of the “Sakai 3” effort in their simplest forms was over confidence and making too many promises and not delivering on those promises. The seeds of this were sown at the 2009 Sakai Foundation Board retreat where it was decided that making a “Sakai 3” was so important that the board authorized spending at levels higher than incoming revenue to hire a product manager, marketing person, and continue to fund a UX person as top priorities. This deficit spending continued with board blessing until late in 2010 at which point the foundation was bordering on bankruptcy and finally, faced with bankruptcy backed away form the policy that “OAE was too big to fail” and worth putting the foundation itself at risk to save the OAE. The board members and foundation coupled this ill-chosen financial strategy with effuse public “sales pitches” about OAE everywhere they went. It is not surprising that the community believed the foundation board and leadership – so it is really unfair to blame the badly misled and mis-informed community. The “marketing” worked – sadly it was not followed up by a timely product. It was at the Sakai conference in Denver when the Sakai board grudgingly accepted that the Sakai CLE product did not need to be shut down and allowed for the formation of the Sakai Technical Communications Committee (TCC) to govern the CLE relatively free of board meddling. Even throughout 2011 and even in 2012, some board members yearned for a time where the board set the community agenda for both CLE and OAE by board fiat – but thankfully those notions are fading into memory as we embrace the new “big tent” and community-centered philosophy that underpins Apereo.
Reference: http://www.dr-chuck.com/csev-blog/2010/11/sakai-board-elections-2010-edition/
The lesson to learn is that the top-down management and marketing-driven approaches that “sort-of” work in the private sector – utterly fail in open-source communities where the participants (people and organizations) are volunteers.
I am really excited about the future potential of Apereo OAE to be a next-generation academic product for all the reasons that you cite in your post. I have great confidence in the team and the remaining stakeholders seem in it for the whole journey wherever it leads. We all need to applaud their efforts so far and look forward to more good things from OAE in the future – without adding too much in the form of expectations from the outside.
It’s so interesting to me how intent everybody is on pointing fingers when it comes to OAE. Every time I blog about it, the number of impassioned public and private comments I get on it trying to make sure that I get who dunnit “right” is astonishing.
I’m not all that interested in that question. Rather, I think there’s a more important question for both the Apereo Foundation and higher education in general: Why do these sorts of projects fail? What enables school-supported open source projects to be sustainable in the long run? I don’t think that many (or perhaps any) of the projects under the Apereo umbrella have yet found that magic formula that strikes the right balance between on-the-ground project leadership and ongoing alignment with the strategic goals of the sponsoring institutions. Many of the projects are muddling along in this regard, but I don’t see any spectacular successes. OAE was a spectacular failure and therefore interesting as a case study. That’s my main interest in it—that and the fact that I find it to be a really compelling vision for a new type of learning environment.
The first Apereo Conference was a couple of weeks ago. Most who attended and provided feedback, from both former Jasig and Sakai communities, suggested that it was the best community conference we’ve held for some years. From my vantage point, I’d tend to agree with that. So it’s good to see both Chuck and Michael focus on the future Apereo represents, and I’m genuinely loathe to continually debate the past – despite the fact that somewhere in my own past I have a Batchelors degree in History 😉 A few points, however …
The financial situation in 2010 was as dire as Chuck represents. I’m not aware in as much detail as Chuck of the reasons for what amounted to a systemic overspend, or when that originated. I suspect the point of origin for the overpsend to lie before 2009, to be frank. In demonising the Board, however, Chuck introduces a perspective which is misleading. The reduction in Foundation spending was begun by Lois Brooks, as interim Executive Director in February/March 2010. When I became Executive Director, in August 2010, I conducted an immediate analysis and proposed options to the Board. From September, I enacted one of those options with full Board support. If any reader is particularly interested in the detail of this period, please see my blog posting from December 2010. You’ll also get a more accurate picture of the balance I attempted to bring to the CLE / OAE issue, if you care to, in the same posting –
http://www.sakaiproject.org/blogs/ian-dolphin/one-hundred-and-forty-days
I hope that at this point we can focus a little more on the future, and the questions Michael poses on the nature of higher education software projects should be of interest not simply to the software and other communities clustered around Apereo, but to all the higher ed community. I’ll add a few myself. Are large scale managed projects such as OAE between 2010-2012 inherently more prone to risk from internal institutional change because the investment level involved raises their prominence? Do they inherently drift towards more waterfall-like development approaches, or can they adopt and adapt more agile methodologies? Do larger projects make the difficult technical choices OAE faced last year easier or harder to resolve? When the new OAE stack has its first release in a couple of months, it will also be fair to ask why a significantly smaller team has managed to make significantly greater progress in a shorter period of time. I do not presume to provide answers to these questions – but I suspect that exposing these issues to the broader community will generate some interesting and useful perspectives.
Ian – Your narrative of when the board became aware of the dire financial situation accurately places board awareness of the problem in the February / March timeframe and accurately credits Lois Brooks with doing a great job as interim in addressing the financial situation during the March-June timeframe. She did a great job as interim and when you came on in June you continued to do a great job in addressing the financial challenges. We would not be here if it were not for both of your excellent leadership during difficult times.
But the anti-CLE and pro deficit spend policy was strongly held at the board level up to the very last minute. I distinctly recall one board member making an impassioned plea that “deficit spending was essential to the success of OAE” in a late January/Early February meeting as financial concerns were discussed. A few weeks later we saw balance sheets that showed how truly grave the situation was and the talk of “strategic deficit spending” instantly gave way to talk of how to avoid bankruptcy. The CLE-as-deprecated board policy persisted until the June 2010 board meeting in Denver *after* the OAE project suffered major stakeholder pull out (May 2010) and about four months after the board became aware that it was effectively bankrupt. And even at the June 2010 meeting – changing the policy to allow both the OAE and CLE to continue as equals under their own independent leadership was accepted grudgingly by some of the board members at that meeting.
I have the ultimate respect for the OAE team and people involved in the project. The OAE team threw themselves at an impossible task back in 2008 – they raised funds (the board and foundation staff helped fund raise funds in a good way) and tried a bold form of governance for the project – they were forever on the edge of emerging technologies and gave us a beautiful view of what the future will look like. I will defend and support the OAE team past, present, and future. The OAE/CLE schism did not come from either the OAE or CLE – it arose as a result of a board of directors of an open source organization that felt that they had the authority to manage volunteer resources as if those resources belonged to the board.
It all worked out (whew!) and I (like everyone else) want to move on. Because I think that the future is very bright for both OAE and CLE.
Thanks Michael for taking the time to put this post together as I respect you knowledge and perspectives greatly and thus your comments are particularly meaningful to me. As you and others have noted, the past few years of the OAE project do provide us with a complex and important case study in open-source software development, particularly software which is rather user-facing. The importance of alignment of goals and vision among project partners remains one lesson that the project has taught me which I think we need to keep at the front of our minds as new Apereo projects form and mature. Project governance, in all of its different flavors, remains another major challenge that I think we’ve learned importance lessons from OAE/Sakai 3 but which I think we still have much to understand collectively as a community.
Speaking for myself as the newest investor in the Apereo OAE project, I came out of the Open Apereo conference feeling both relieved (all of the demos worked perfectly!) and very excited about the future of the project given the positive response and level of interest. Although I remain very positive and excited, I think it is also important that we not over promise and keep our goals realistic and focused on the near-term. The first release of the “new” OAE, which should be out shortly, will be a huge step forward but also a step that will basically get us back to the starting gate rather than one that will get us over the finish line. Meaning, now that we’ve demonstrated the viability of the product the real work begins to put this into production and mature it over the coming year so that it has the capabilities we know are needed to support broad forms of academic collaboration. I have a lot of confidence that the team that is in place can achieve greatest with this product and do something truly innovative, but we will need time to realize this goal. We will also need more contributions, adoptions, and resources and thus I’d encourage those interested in OAE to not just watch on the sidelines but to reach out and discuss ways of getting involved during the early and exciting times.
Digressing for a moment I will briefly comment on Chuck and Ian’s postings…
I was on the Sakai Foundation Board of Directors during the time that they discuss and served as Chair during the final few years and, off and on, through the transition to Apereo. Hindsight analysis can always shed light on things which may seem simple or straightforward now but which, at the time, were complex and murky issues with no clear road forward. I firmly believe that is the case here. As I’ve said publically before, there were decisions made during these times which I very much supported then but now regret (and which I’ve apologized for to those who were hurt by them). There are others which I stand by and feel were necessary given the challenges we faced. The one thing that I can say though about all of the decisions is that they were made by a group of people who were deeply committed to and passionate about Sakai and our community (as we all are). With that, I’ll save the rest of comments on this for my book ;-).
Josh