Response to MoodleNews: Some considerations for critical reading of market sizing claims

There is an interesting post at MoodleNews titled “Open Source LMS Alive And Well: Moodle Breaks World Higher Ed Records, Tops North America, Grows Everywhere” looking at some of our recent LMS market data, focusing primarily on what the data reveal about open source systems such as Moodle, Sakai, Claroline, and Chamilo.

We welcome the usage of our data posted at e-Literate for these purposes,1 but there are some misunderstandings evident in the MoodleNews post that we see too often in critical readings of market sizing reports. Unfortunately the misreadings can cloud otherwise valuable analytical reporting.

The latest e-Literate report on “active LMS” shows Moodle as the incontrovertible LMS leader in global Higher Ed for primary and secondary systems, including North America.

Incontrovertible seems quite strong, and we at e-Literate try to be very careful in our language and we rarely use words like incontrovertible or unequivocal.2 The reason is that for most people reading the data is a translation exercise. There is a need to understand the level of confidence the reader should have and what the limits of the claims are.

It is useful that MoodleNews called out the global nature of data and measure of primary and secondary systems.

Assuming it uses data provided by self-reporting service LISTedTECH from the last month, it shows dominance in every region, unsurmountable everywhere but the US, and healthy growth rates. The report mentions the “LMS long tail” as well as “the Big Four” (Moodle, Blackboard, Instructure Canvas and D2L Brightspace) but by their own accounting a more accurate distinction would be “the big one.”

The data set used from our partners LISTedTECH is not a self-reporting service. Read this post for more information on the multiple channels of data collection used.

But unsurmountable comes out of the blue. The whole point of doing reports like we do is that we don’t know for certain what the future trends will be, and we gather the data to reduce uncertainty. A while back, many would have said that Blackboard has an insurmountable dominance in North America, but we have seen major changes that would have made those claims invalid.

I can see the argument for describing Moodle as “the big one” as an alternate description to “the Big Four”.

LISTedTECH sample (apparently totaling 12,879 active LMS) lists 1,419 active Moodle sites in Higher Ed for North America, self-acknowledged as their most complete dataset. It appears that the advantages in Canada compensate for Instructure Canvas’ growth in the US. In July, it had listed 1,000 sites only for Higher Ed in the US.

The first sentence is a good example of characterizing the data in a useful way, as it gives the reader a sense of the data set while also clarifying usage in a specific global region.

Unfortunately the article then jumps into some confusing claims based on misreading the data descriptions. The June 2017 post captured percentage of primary systems at degree-granting institutions for each of four global regions with North America = US and Canada combined, the July 2018 post captured primary LMS in … US colleges and universities, and the August 2018 post captured total counts of both primary and secondary system usage in six global regions. We chose this latter method for the August post due to the frequency of long tail and open source LMS systems being used as secondary systems (i.e. some other LMS is the campus standard, or primary system). There is little basis for concluding that Moodle’s advantages in Canada compensate for Instructure Canvas’ growth in the US.

The larger point here is that there are lots of ways to slice and dice data that end up with similar-lookinggraphs that represent different things – it’s important to read the legends and surrounding text carefully. To give a sense on the issue of primary vs. secondary systems  for North America, consider the difference shown below.3

With MoodleNews’ natural focus on Moodle, it would be accurate to note that Moodle is used quite frequently as a secondary system.

For comparison, official stats at moodle.net set the total figure at 11,490 as of writing. 9,776 for the US, the biggest Moodle nation; 1,714 for Canada. Moodle does not offer discriminated site data for Higher Ed only.

This is a helpful comparison and description with a different data set.

Unfortunately, I need to be careful with further quotations. MoodleNews does not use a creative commons license and clearly limits usage based on the site’s terms of use. So the remainder of this post will have to be limited to short descriptions.

There is a useful section looking at Moodle’s shares in six different global regions. The main caution I would add comes from our November 2017 post describing the data.

Market share information provided in percentages and trends are more reliable than absolute counts outside of North America. When we do provide absolute numbers, we advise caution for readers or subscribers to not over-interpret the absolute numbers, at least without us providing additional details to keep the data in context.

But in this case, MoodleNews also included the percentages for each region, which is helpful.

The next four paragraphs all focus on market trends (year over year comparisons) that cannot be supported by the data (see above).

At the end of the MoodleNews post, there is some valuable commentary about other (non-Moodle) open source LMS system and the origins and deployments across various regions.

Given these clarifications, I would also add the the MoodleNews title has some serious flaws.

All-in-all, it’s good to see valuable discussions spawned from our CC-BY licensed posts. But readers should be cautios when trying to understand market sizing data and make comparisons and trends analysis carefully.

Update 8/26: Please see comment from Martin Dougiamas clarifying that MoodleNews “is not associated or affiliated with the Moodle organisation in any way”.