The University of North Carolina, a current Blackboard customer that is evaluating Sakai, just published a very interesting report of their findings so far. Among other things, it’s a good model for schools that want to do a thorough evaluation of a platform and have the resources (i.e., staff and time) to do it right. Here’s the excerpt of it that UNC chose to publish in their blog post announcing its release:
The findings of the pilot are positive, leading the Sakai Action Group to recommend ITS continue to fund Sakai for the 2009-2010 academic year, implement student information system (SIS) integration, when possible, expand the number of participants using Sakai, and research a possible future migration path.
The whole report is worth reading, but here are some of the highlights from my perspective:
- There are relatively few comments from faculty or students about functionality gaps in one direction or the other. To the contrary, the platforms were seen by many as being functionally equivalent. LMS’s have reached that “good enough” stage where they are starting to commoditize.
- There were two exceptions to the “equivalent functionality” rule. First, a number of users seemed to appreciate Sakai’s more flexible permissions structure. I have written about the under-appreciated importance of groups and permissions on many occasions. The second was the ability to use Sakai for non-course work groups, projects, communities, etc. Blackboard has this capability, but you have to pay extra and license it separately. Frankly, I don’t know how they get away with it. No other LMS that I know of charges you twice for what is 95% the same functionality.
- There were lots of comments on usability, and almost all of them broke in Sakai’s favor. First of all, this confirms the point that I’ve made a few times here that usability is often more important than functionality. From UNC’s data, it looks like faculty and students were able to do more with Sakai while making fewer calls to the help desk. Which brings me to the other part of this. Three years ago, Sakai’s usability was bad to the point of being embarrassing. But it absolutely kicked Blackboard’s butt in UNC’s review. This is strong affirmation of the huge strides the Sakai community has made in usability.
Good stuff.
I thought UNC’s report was a valuable read as well, and showed how Sakai can perform well in a rigorous adoption process like UNC followed. The report also covered the value of using commercial vendor support for Sakai.
The only thing missing from the report was mention of the specific Sakai commercial affiliate UNC used during its pilot evaluation phase. A hint is in the screenshot of UNC’s Sakai instance in the report, which reveals it to be rSmart 😉 The team at rSmart likes to think a significant part of the success of UNC’s pilot can be attributed to our assistance with Sakai configuration, training, hosting and support. I don’t post this just to toot rSmart’s horn, but to make sure others looking at UNC’s success with Sakai know all the ingredients that stood behind it.
Good summary of an excellent summary of a thorough pilot. Lots of credit to the UNC Sakai team and especially Kim Eke. They’ve clearly done a great job. I’m sure this work will pay dividends in a smooth transition to Sakai (:-)).
On the usability question, this indeed is crucial. Do you have any opinions about why Sakai’s usability has improved? Is there something in particular the community has done in terms of process or structure to address this?
Also, am I correct that formal usability testing is a part of the Sakai 3 process?
Bruce: Briefly, I think Sakai 2 usability has improved substantially
due to the maturity of the product—once basic functionality was
achieved and Sakai started to be used deeply and widely, more
resources were turned to usability, finding solutions based on real
user experience. The year-old Sakai development process and Product
Council were also formed to provide more formal frameworks and
stewardship for Sakai, with usability being one of the goals.
http://confluence.sakaiproject.org/display/MGT/How+Sakai+Development+Works
http://confluence.sakaiproject.org/display/MGT/Sakai+Product+Council
As for Sakai 3, one benefit for usability is that the project as a
whole is following a more user-centered design approach unlike the
more classic developer/ment-led design of Sakai 2. The following links
may give you some insight to the activities underway:
You might find the work of the 3akai UX group interesting:
http://confluence.sakaiproject.org/display/3AK/Sakai+3+Home
As well as the work of the Sakai Teaching & Learning Working Group
around Sakai 3 Learning Capabilities:
http://confluence.sakaiproject.org/display/3AK/Teaching+and+Learning+Capabilities
http://confluence.sakaiproject.org/pages/viewpage.action?pageId=21954623
When it comes to formal Sakai 3 user testing, you can see some
documentation here:
http://confluence.sakaiproject.org/display/3AK/Sakai+3+User+Testing
Nate is right; there’s just been a lot of concerted effort on this over time.
I would add that the Sakai community does have an unusually rich talent pool of usability experts. Two of the Foundation Board members—past board member Mara Hancock from UC Berkeley and current board member Jutta Treveranus from University of Toronto—run usability teams at their respective schools and have brought resources in to help improve the system. Going forward, the Sakai 3 effort is *very* usability-focused.
Keep in mind, though, that usability means different things to different people. For faculty that are used to one platform, usability means anything that works like what they already know. But that’s a poor measure. You want to know (a) how easy is it for new users to learn from scratch, and (b) once you know how to use it, how many clicks do you need to make for common tasks? How hard is it to make a major mistake (like delete your entire course, for example)? And so on.
Thanks Nate, Michael.
On the “usability is in the eye of the beholder” point, I came across a quite nice discussion of this over at the Moodle site …
http://docs.moodle.org/en/Usability
… in particular the “Avoid the word “intuitive”” section.