Analysis of Blackboard Response to Recent Disclosure of Security Vulnerabilities

There’s been an interesting set of public relations based on the recent news of Blackboard security vulnerabilities.  SC Magazine’s Australian edition broke a story on September 16 about an investigation by two or more anonymous Australian universities working with a security firm, Securus Global.  In conjunction with the magazine, this investigation exposed a number of security vulnerabilities with Blackboard Learn 8.0, 9.0 and 9.1 (the mainline legacy LMS from Blackboard, but not the WebCT or ANGEL acquired LMS product lines).  Blackboard has confirmed the majority of the issues are valid and issued a security advisory on September 16, subsequently updated on September 22.

That is where the agreements stop, however.  Blackboard took issue with the tone of the article in a blog post from September 23, subsequently updated on September 27.  Blackboard disputes the assertions that they were not responsive to customer requests, and further, they are trying to downplay the significance of the vulnerabilities.

First, our security team is far more vigilant than this article suggested and our partnership with the clients who reported the issues is strong.  In fact, our team responded immediately to the concern when we received notice of it. We were able to address the first issue reported through proper secure configuration of the application within 48 hours of receiving the initial client support request. We’ve worked closely with the client team and the security organizations that first found the issues, in order to classify and create a resolution plan for all of the other issues in detail—sharing information and providing frequent status updates. We determined that some of the issues were items which we were already working on, and the others are now being addressed by our engineering team. The most serious issue, regarding compatibility with the latest version of Tomcat, has now been closed for Blackboard Learn 9.1, enabling us to move the overall threat level for Releases 9.1 and 9.0 customers from “high” to “medium,” which represents a risk level that Blackboard classifies as moderate according to its well-established practices. We are finishing up our investigation for Release 8.0’s Tomcat and have not yet identified any high risk.

I had the opportunity to interview Blackboard staff as well as representatives from many of their competitors – Desire2Learn, MoodleRooms, Instructure, and LoudCloud.  I have not been able to get a response from SC Magazine, and their sources remain anonymous.  This is not an unusual situation based on security investigations, and Blackboard confirmed that reporters remaining anonymous is typical.  Unfortunately, this means that it is difficult to pass judgement on the areas where Blackboard has disputed the original article, such as whether they were responsive to the universities.

Additionally, there has been no real reporting from the higher ed media, where the only articles were essentially rehashing of the article and Blackboard’s response.

I think the point-counterpoint nature of these disputes is a distraction from the real issue, however.  When you get right down to it, the real issue is how well Blackboard has limited its clients’ exposure to loss of data and system integrity.

I also had the opportunity to get the feedback and insight of my colleague at Delta Initiative, Jo Peters, who has a very practical approach to risk management and security practices.  He has helped me to understand certain issues and steered me away from interpretations that might not be fair to the parties involved.

Before I comment on the specific issues, I think it would be helpful to acknowledge 5 issues to keep this discussion in context.

• In the interviews, all LMS vendors acknowledged that no web-based software is perfect and you should always expect some vulnerabilities.  The issue should not just be on whether there are vulnerabilities, but perhaps more importantly, on how a company or organization responds to a security vulnerability or incident.
• There are vulnerabilities in some of the components, like Tomcat, that are not in the direct control of Blackboard or any other LMS vendor.  However, there still is the need to work on closing holes by applying patches or version upgrades and certifying those with all of the supported versions as quickly as possible.
• There are no known actual losses of data or security incidents based on these vulnerabilities.  As stated in the security advisory “We are not aware of attacks exploiting the vulnerabilities identified in this advisory”.  This does not mean that none have occurred, but neither the Australian investigation nor Blackboard’s investigation has turned up actual exploits.
• Blackboard is not the only major LMS to have had security vulnerabilities exposed.  Moodle, for example, has had similar problems as reported in 2009 and 2010
• In Blackboard’s security advisory and in interviews, they acknowledged that the majority of issues raised in Australia were valid security vulnerabilities.  Although Blackboard would not confirm the exact numbers, based on blog posting it seems that more than half of the valid findings were already known, and the remainder were new issues raised by the Australian investigation.

How Similar Is Current Australian Episode to Last Year’s European Episode?

This whole set of issues sounds similar to the security investigation that occurred last year in the Netherlands, which was not widely reported outside of Europe.  If you can handle the imperfections of Google Translate, a summary article was published in October 2010 by Webwereld in the Netherlands.  The article describes how a Dutch university discovered security vulnerabilities with Blackboard and contracted with an ethical hacking firm, Online24, to investigate the issues.  Michiel Prins and Jorge Abma led the investigation for Online24, and they published a confidential report to the university and to Blackboard, under non-disclosure agreement to avoid publicizing the issues, on February 19, 2010.

From February to September 2010, the security firm and university did not receive confirmation that the issues were being fully addressed, but they did describe the investigation in blog postings July 19 and October 29, 2010.  Additionally, on September 22, 2010 (ironically exactly one year prior to Blackboard’s most recent security advisory) Online24 published a risk analysis report based on the investigation, to let European universities know of the potential problems.  The report did not list the specific technical issues, but it listed 84 security vulnerabilities that included cross-site scripting and other typical web-based issues.

This report, subsequently reported by Webwereld, led to a response from Blackboard, who sent a team over in October 2010 to discuss the issues at their Amsterdam office.  As described in the magazine article (please forgive translation errors along with my attempts at clarification):

In a report (pdf) describe a system that experts are not designed with security in mind, it seems. This not always check whether users are allowed to perform actions, the application is vulnerable to XSS attacks, and far too much information leaked on errors. It is also possible, for example mail forms to manipulate, so spamming is possible and the rights of files are not good. [snip]

But the most striking is perhaps that the weaknesses open the door to adjust study results. “Users of Blackboard are at high risk. Not only the privacy of the users is at stake, even exams can be stolen. Due to the advent of single sign-on systems can be adjusted figures or in the worst case, a student even in no graduation, “said Abma opposite Networks.

That version 8 of the software is examined and not the latest 9.1 version makes little difference according to Prince, and many leaks are not addressed: “In version 9.1 a number of problems addressed, but Blackboard has a long way to go.”

Abma and Prince already called Blackboard in February about the report and the technical details of the leak [ed – translation clarified]. But that did not result in concrete action from the company and was why the experts colleges and universities informed the public of the problems due to risk reduction [ed – translation clarified].

When asked to comment on Macworld, the company said there were aware of the problems and this action [ed – translation clarified]. “These are very common vulnerability of all web based application,” commented Bob Alcorn, Chief Architect at Blackboard. He promises the weaknesses in the review and improvement.

Alcorn points out that his company in the latest version is already against XSS vulnerabilities has been taken and that therefore less sensitive. Also promises the company is now the problems both the latest patch for older versions of the product. Meanwhile, the company also included contact with investigators and they thank you for the alert to the problems.

As of my phone interview with Online24 on October 4, 2011, there was still no confirmation that all the issues had been addressed, despite the public proclamations in the article.

What is the Reality of “we’re committed to fixing them quickly”?

According the stated security and privacy policy, “Blackboard cannot provide product updates according to a set timeline — but we are committed to working expeditiously”, and in the blog entry on the specific vulnerabilities “we’re committed to fixing them quickly”.  What does this mean in practice, given the specifics of the Australian investigation?

Quite simply, Blackboard is planning to provide patches to Learn 9.1 “by the end of the year”.  This means that the patches will be available approximately 5 months after Blackboard was notified of the problems and approximately 3 months after the vulnerabilities became public in the magazine article and security advisory.  We can look at the actual dates of reports, advisories and fixes for both the Australian (2011) and European (2010) investigations below.

Blackboard responded in their blog that “These issues would not allow access to the entire system for grades or other system-wide information. The likelihood of an administrator account being compromised is low, and any attempted malicious actions would be logged and traceable.”  Nevertheless, there are real medium-severity issues that are taking a very long time to address.

What Does Operational Support Mean?

According to Blackboard’s Client Support Services Guide, Blackboard Learn 8.0 and 9.0 are both officially supported products – Learn 8.0 is in Operational Support already and Learn 9.0 will be in Operational Support as of this month.  What this means for security issues is that (according to interviews):

• “Critical and High severity items are patched for products in Full Support and Operational Support (categories familiar to Blackboard clients)
• We make a case-by-case decision for Low and Medium severity issues about whether to issue patches or to address the issues in an upcoming Service Pack”

According to multiple sources with access to the security advisory, Blackboard has stated that there is no plan to patch Learn 8.0 or 9.0.  Additionally, all of the issues in the Australian investigation have been rated as medium severity by Blackboard, although the overall advisory is rated as a high severity.   While the advisory does list some risk mitigations and workarounds, there are several issues where no mitigations or workarounds exist.

To give context, in 2009, Blackboard wrote a public response to the North Carolina Community College System based on their assessment of Moodle.  In that response, Blackboard stated the following:

Security breaches and identity theft are common challenges that today’s education IT staff struggle with every day. Sensitive information, regulated data, and proprietary research are transferred over the Internet more than ever before. Protecting this information is critical to minimizing liability and ensuring students and teachers can focus on learning instead of on technology. Moodle relies on the community at large to test for security vulnerabilities and to issue security patches after a problem is identified. This model is risky because hackers can have access to the same information as the programmers and they can quickly exploit any security issue announced on the Moodle public forums before a school offers a patch to fix the issue.

Blackboard is accountable for repairing any security concerns and these concerns are never announced publicly while a fix is in progress. Blackboard has a dedicated team of security professionals who follow a software industry standard security methodology, providing clients with enterprise-level security monitoring, testing, response planning, and maintenance. Blackboard spends millions of dollars to ensure that its products and your data are secure and these security benefits are built directly into the Blackboard license fee.

As Jo Peters stated, “Sending out an advisory without knowledge of patch is like painting a bullseye on yourself.”  However, I should note that in the current Australian case Blackboard was forced to publicly announce the vulnerabilities before a fix was available due to the magazine article being published.  I cannot fault them for this disclosure, but it certainly does raise the importance of a timely fix to address the vulnerabilities.  As pointed out by Mr. Peters, “It is still a daunting task for the legitimate organizations as they need to do the right thing all the time, while the hacker only needs to be right once to cause irreparable harm to his/her target.”

As we can see from the current issues, in practice this means that clients on Learn 8.0 or 9.0 face a choice: either upgrade to Learn 9.1 or simply live with some of the security vulnerabilities, despite their ongoing license fees.

Call for Transparency in LMS Market

I originally started the research for this blog post with the intention of analyzing the broader LMS market.  As I’ve gotten deeper into the details, it has become apparent that the real issues are how companies respond to vulnerabilities in practice, not in words, and that the Blackboard incident needs more scrutiny by the higher ed community.

We need more transparency in the LMS market, and clients should have access to objective measurements of the security of a solution.  To paraphrase Michael Feldstein’s suggestions from a 2009 post:

• There is no guarantee that any LMS is more secure just because they say they are more secure
• Customers should ask for, and LMS vendors should supply, detailed information on how the vendor or open source community has handled security issues in practice
• LMS providers should make public a summary of vulnerabilities, including resolution time

I would add to this call for transparency that LMS vendors and open source communities should share information from their third-party security audits and tests.  All of the vendors that I talked to have some form of third-party penetration testing and security audits; however, how does this help the customer unless this information is transparent and available?  Of course this transparency should not include details that would advertise vulnerabilities to hackers, but there should be some manner to be open and transparent on what the audits are saying

More to Come

While I do not have the time or space to do a comprehensive review of LMS market security policies, expect some more analysis of current issues, including a broader discussion of LMS market security practices.