IBM’s Misleading or Just Incorrect National Ad on Student Retention

Thanks to Chris Edwards for alerting me to this one. In a nutshell, IBM launched a national ad campaign last month that included commercials during Wimbledon on the weekend. They’re spending big money on this campaign about big data, learning analytics, and reducing “dropout rates” [emphasis added below].
https://www.youtube.com/watch?v=iiv4X1K7iX4

In the US, 3 in 10 college students drop out, but how can you spot who is at risk? The one who lives far from campus? The one who works the night shift? The one with new responsibilities? One thing can’t tell you, but the right combination can. Universities are using IBM Analytics to understand pressures in and out of the classroom. Some expect to cut dropout rates by 25%. IBM Analytics is working to make education smarter every day.

If you follow the URL listed at the end of the YouTube video and select ‘Education’, you get to the main page that highlights the same video as above. For further explanation:

Using the IBM Exceptional Student Experience, universities can deploy predictive analytics running on the SoftLayer® cloud. They analyze more deeply what they always have—things like poor attendance and failing grades. But they are also beginning to factor in new kinds of data: how far students live from campus, how many hours a week they work, even how often they log on to the university Wi-Fi network. Universities can then customize a course of action for each student. Even something as simple as providing transportation vouchers can be beneficial. The platform gives professors, advisors and students themselves a view of personalized performance, so students can get help—before it’s too late.

With a little searching, you can find a series of education case studies which are quite well documented. But searching through them reveals no data to back up the claim made above. The closest one is Hamilton County, TN.

Through this innovation [predictive analytics], Hamilton County Schools achieved the best ‘No Child Left Behind’ results in its history, reducing its annual dropout rate by 25 percent and increasing its success rates by eight percent.

But that is a K-12 example. On the higher ed side, there is one case study for Ithaca College about increasing retention rates by 4% and another for Brockenhurst College about their expectation “to improve student recruitment and retention by 15 percent over a five-year period”. The University of Western Sydney is:

running a pilot program with the School of Business, where they have already designed specific retention strategies for each category. We’re really excited to see the impact this has on their attrition rates over the next year.

While IBM’s use of documented case studies is commendable, none of these back up the claims in the national ad. I contacted IBM to get more information, and they agreed to look into the matter. A representative for IBM’s Global Industry Marketing for Government and Education shared the same case studies on retention that I had already found. We had a phone call where I described the problem that nothing backed up the claim, and he said he would find out what the basis was for the commercial.

18 days later and no more responses despite several email reminders.

Let’s look at the claim in more detail.

“Some”

Some universities, some people, some marketing folks? Given the context of the ad which is all about higher ed, I would assume that this means “some colleges and universities”, but we don’t know.

“Expect”

Does this mean a group of universities has piloted the use of IBM Analytics, and the early results predict that campus-wide deployment might reduce dropouts by 25% (best case)? Or does this mean some guy they met at Starbucks expects some big results after reading IBM’s marketing? Or is someone expecting that Hamilton County K-12 results will do the same for higher ed? We don’t know.

“To Cut Dropout Rates”

The phrase “dropout rates” is used more commonly in K-12 and not higher ed. In higher ed, the issue is captured more by retention and persistence. Does IBM mean that specific course drops will decrease by 25%? Given the intro about 3 in 10 college students dropping out, I assume this means 25% fewer students will drop out of college – stop attending. In the Hamilton County K-12 example, they mean students dropping out of high school altogether. But we don’t know.

“By 25%”

This is the most troubling point. IBM takes a specific performance-sounding number which leads the viewer to believe that there are real results backing up this specific claim. But when pressed, even IBM executives cannot justify this specific number.

Two Conclusions, Neither Good

@PhilOnEdTech I mean, if a national ad budget is fueling unrealistic expectations about how to help solve a pressing #EdTech problem…

— Chris Edwards (@chris3edwards) July 11, 2015

I appreciate IBM’s initial willingness to talk to me, but Chris is right about fueling unrealistic expectations on a serious education problem. Given the lack of explanation, there are only two conclusions that I can draw from this experience. One is that IBM has some data buried somewhere to back up the claim, but no one can find it and therefore it is not documented. That is the best case scenario. The other, more problematic, conclusion is that IBM put out an ad without any data to back it up. Either way, the ad on student retention cannot be backed up by any documented case study. That is wrong.

My offer to IBM to get their explanation in public still stands. If they find some data to backup the ad, I will post the information here on e-Literate. If not, IBM should take down the ad, revise their web page, and publish an explanation of the changes.