A couple of weeks ago in my post about the different types of learning analytics, I described retention early warning systems thusly:
Most people don’t think about early warning systems as being in the same category as adaptive analytics, but if you consider that “adaptive” really just means “adjusting to your personal needs,” then a system like Purdue’s Course Signals is, in fact, adaptive. It sees when a student is in danger of failing or dropping out and sends increasingly urgent and specific suggestions to that student. It does that without “knowing” anything about the content that the student is learning. Rather, it’s looking at things like recency of course login (Are you showing up for class?), discussion board posts (Are you participating in class?), on-time assignment delivery (Are you turning in your work?), and grade book scores (Is your work passing?), as well as longitudinal information that might indicate whether a student is at-risk coming into the class. What Purdue has found is that such a system can teach students metacognitive awareness of their progress and productive help-seeking behavior. It won’t help them learn the content better, but it will help them develop better learning skills.
Well, last week, Ray Henderson announced Blackboard’s new Retention Center and described it as follows:
The Retention Center gives critical insight on learning and activity gaps to instructors, within the LMS, that helps them quickly diagnose students that are falling behind. Pre-configured and automatic so they don’t have to hunt for it. No set-up: it automatically calls out students that are at risk while instructors still have time and space to do something about it. With the feature instructors can see:
- Who’s logging in: this is a simple but powerful predictor of student success. Instructors see how long it’s been since students have logged in to the course and how many have been away for five days or more. And not by fishing through student profiles or reports but in an automatic view complete with red flags where they’re needed.
- Whether they’re engaged: which students have had low levels of course activity, at 20 percent or below the average in the last week.
- Whose grades are suffering: which students are currently trending at 25 percent or more below the course average so they can target extra help to where it’s most needed – even when it isn’t asked for.
- Who has missed deadlines: instructors might know this anecdotally or on a case-by-case basis, but now they can get a real-time view of all students that have missed one or more deadline.
Eerily similar, no? A number of years back, when I pressed Course Signals inventor John Campbell on which factors in the LMS are most highly predictive of student success across different courses, he named exactly these four. The only surprise here is that this isn’t a common analytics feature of every LMS and courseware platform on the market yet. Purdue proved that their value in helping at-risk students is high. I’m glad Blackboard is stepping up.
The one piece that’s missing is a simple standard where an SIS or other longitudinal data system could pass an at-risk “credit score” to the early warning system to modify its sensitivity. If a student on the honor roll drops off the radar for a week, it’s less of a cause for concern that a student on academic probation (for example). I tried to push this idea for a standard at the IMS a few years back but got nowhere with it at the time. I hope that Blackboard will push for something like it now that they have a system to take the data.
This is something I work on a lot (manually) as an instructor and my courses (which have appx. 10 activities per week that students complete) lend themselves to this kind of intervention. The metric conspicuously missing here is not just students how miss a deadline but students who BARELY MAKE a deadline. I would love to easily get a listing (it’s impossible right now) of students who do their work within, say, 30 minutes of the deadline, students who do it within just an hour of the deadline, and who show that pattern consistently.
Also, the real issue here to me is what analytics do we make available to the STUDENTS – I would far rather have them policing themselves than have me policing them… so shouldn’t we focus first and foremost on making this analytics data accessible to the STUDENTS… and then, secondarily, to instructors and administrators…?
After all, it is the students who ultimately will or will not DO something useful with this data. I say, put the data in their hands as easily and often as possible.
Hi Michael – great info and analysis as usual!
I wanted to let you know IMS has a new Learning Analytics Leadership group which has met several times since last November. So far, we’d identified the high level systems and what might be the most beneficial areas to focus on for interoperability. One of the goals is to standardize common vocabularies and usage patterns, without dictating or having to maintain an exhaustive list of all data elements which might be used. As one of the leading SIS vendors participating in this group, I will be sure to bring up your suggestion of the leaner “credit rating” score. The group seems to be gaining momentum and now would be the perfect time for folks from Blackboard to identify someone to participate.
That’s an interesting point about down-to-the-wire assignment submissions. I would love to see that metric tested with real data across a wide range of classes. My intuition is that you’re onto something (although there will be some noise in the data from the structure and scheduling of assignments by some instructors).
Regarding analytics for the student vs the teacher, that’s one apparent difference between Blackboard’s approach and Purdue’s (and now Ellucian’s, since they licensed Purdue’s IP). Course Signals focuses on teaching students help-seeking behavior. There’s good data to show that it’s successful at doing that. Retention Center, in contrast, seems focused on providing teachers with early warning. That’s still good, but I agree that teaching students meta-cognitive skills is more powerful, particularly in the long-run.
Linda, I think a longitudinal risk “credit score” would be a great quick win for your group. Minimally, it could need as few a two data elements: the student’s LIS SourcedID and some sort of risk score (which could be a simple as 1-5 for quintiles of the student population). Since factors determining risk are somewhat institution-centric anyway, you can leave it up to the source systems to determine how they arrive at the risk score.
Thanks for your commentary Michael!
As the Blackboard Product Manager for this feature, I can assure you that student access to these learning analytics as well as integration with external systems are top of mind. We designed the Retention Center to later include student views, advisor views, and bi-directional data exchange with other systems, including our own Analytics for Learn product (which already provides complementary features and student views). We’ve had discussions with the PAR project team on integrating their risk data. But since the triggers in the Retention Center are automatic, we decided that faculty should have time to get used to this feature first and tailor the rules as necessary to better fit their own teaching methodologies.
I’m a big advocate of social learning analytics and helping learners shape their own learning behaviors! See https://blogs.commons.georgetown.edu/everhart/learning-analytics/
Thanks for the clarification on your road map, Deborah.
Michael, great post and follow-up comments. In particular, very much appreciate the comments about exposing analytics to students directly. At the Blackboard Analytics platform (where I work) we are very excited about the campus-to-campus collaboration within our recently launched Learning Analytics Thought Leadership Forum where campuses deploying our Blackboard Analytics suite are sharing their experiences around student utilization of analytics feedback data.
Further, we just recently announced our newest Blackboard Analytics for SIS products release (4.1) which offers campuses a framework for the very “credit score” that you reference above. Our framework is based upon an initial set of pre-defined risk criteria, but is built in such an extensible way to allow campuses to greatly extend the number and breadth of possible risk indicators which calculate the “score.” The score then provides for rich longitudinal views (of retention, performance, completion) based on students potential risk score. We’ll look forward to sharing more as we learn more with institutions deploying this capability.
Michael — We’re a start-up company that is marketing our Retention Analytics with heavy use of the “credit score” analogy. We call it the Risk Index Score. It is inexpensive, easy to implement. and does things that neither Blackboard nor Canvas Analytics do. I hope you will check us out.
Chris, I’m not sure how anyone would “check you out,” given that your web site contains almost no information about how your product actually works. I’m interested, but not interested enough to “schedule a demo.” I suspect there are prospective customers out there who feel the same way.
Michael — There is a “Download Solution Brochure” link under the “Request a Demo” button. I also have a more detailed document to provide to folks when I have an email address. Don’t think I saw a way to attach a document to this comment page. But I would be happy to do so.
Ah, I see it. It is not below the button on the home page, but it is below the same button on the solution page (though it is not very prominent).
It looks like you’re taking at-risk factors from the LMS and are therefore directly competing with the Blackboard feature I described here…yes? When I talk about a “risk credit score,” I’m specifically talking about pulling longitudinal data out of the SIS. It doesn’t look like that’s what you do.
Correct. No demographic data, no subjective instructor input. Just hard data from the LMS to offer an online director a fact-based, tactical view of which students he/she needs to contact today. So many analytics programs offer value to presidents and CIOs, but not to the people fighting the ground battles of daily retention challenges. We can be live in a week and offer a roadmap for today’s retention needs.
Any information about if the new Retention Center is part of Learn or a separate product that institutions must license?
The Blackboard Retention Center is included in Learn as part of the Course Delivery license. No extra charge.
Everyone who’s following these comments, please feel free to reach out to me directly if you have questions, and I’m genuinely interested in knowing how you’d like us to improve and expand these capabilities.
It appears that the Student Retention Center measures risk — by course. What if I am an online director with students taking multiple courses. I want to see my entire student population with risk ratings across ALL courses. Can I do that?
Your readers would benefit by knowing why this is worthy of being classified as an “early warning” system. It’s also an abuse to classify this as “Analytics”.
I could be wrong but from the announcement it appears that BB’s “Retention Center” is merely a set of user activity reports re-branded as “Analytics”. So an instructor sees login reports, engagement reports, grade reports, and completion reports. What’s the big deal about that? All LMSs have it or should have it. Blackboard is stepping up? Give me a break.
Putting this in the same league as Signals is ridiculous. Signals is based on a predictive model, which is not the same thing as providing raw reports. Imagine being an instructor with 200 students. There is all the difference in the world between a) having to wade through individual activity reports for hundreds of students in my class and b) having a system make predictions of those who are at-risk using data mining.
Perhaps BB does have a predictive modeling engine underlying their Retention Center. But you never bothered to ask. Instead you are just passing on vendor marketing babble. Where’s your analysis as analysts? Shame on you.
Ada, what does “engagement” mean? What should a meaningful “engagement report” show? What does an LMS “completion report” mean? You make these sound trivial, and from a development sense they may be, but from a design sense they are not. I have looked at a fair few LMS reporting pages that are mostly useless. This is not.
The four factors Blackboard has targeted are correlated in academic research with a high risk of not completing a course. They are, in that sense, predictive, and not just random course activity indicators. I know this because, as an analyst in this space, I have stayed current on that research. In fact, if you had followed the link in this post back to my original post on Course Signals, you would have seen that I was writing about this stuff—and asking critical questions about it—in 2008, when very few others outside of certain academic research circles were paying attention. And I know which factors Course Signals inventor John Campbell believes are the most critical based on his research not because I read it in a marketing brochure or even in an academic article, but because I asked him.
Notice also that Blackboard’s language indicates that they are norming against the class average, just as Course Signals does. Again, this is a step toward turning a report into a predictive model. And Blackboard does provide some indicators of how serious a problem a student’s activity level is, so that you don’t have to “wade through individual activity reports.” These factors make it at least minimally an early warning system and an analytics system. It may not be highlighted in their text, but it is possible to apply some analysis to what you call “marketing babble” if you know what you’re looking for.
So what are the differences between what Blackboard is providing and what Signals provides from an analytics perspective? There’s only so much we can tell from the information that Blackboard has released, but it looks like (1) a bunch of LMS activity factors that comprise the “long tail” of predictors, and (2) longitudinal data. I am not convinced that the former is important, and I have specifically called out the absence of the latter. I would add that Signals has done some strong research making sure that the messages they send to students are most likely to provoke action, which is extremely valuable.
But when you throw around terms like “data mining,” Ada, you might want to consider the possibility that you may be the one falling victim to “marketing babble” (or perhaps big data hype). Purdue did some initial data analysis to distill which factors are most predictive in the LMS. I have yet to see any evidence that ongoing “data mining” is going to yield big gains in predictive power. And, in fact, I don’t believe that they are using data mining to generate at-risk signals dynamically. Purdue also did some data mining on the longitudinal data from their SIS, but again, there is no evidence I’m aware of that ongoing data mining is going to yield better gains. And likewise, I don’t believe that Course Signals uses data mining to create a dynamic indicator of longitudinal risk. They did some data mining on their own data set, identified the Key Performance Indicators (KPIs), and then fed the data on those to Course Signals. They may be doing additional data mining from time to time to refine their model, but they are almost certainly not doing so day-to-day and applying it dynamically to student risk assessments. Now, there is some evidence in the academic literature that KPIs for retention risk in longitudinal data vary significantly from institution to institution. But it’s hard to do the per-institution analysis with an out-of-the-box product offering, in part because institutions do all kinds of weird and idiosyncratic things with their SIS configurations. (This may be one reason why Ellucian doesn’t seem to have had much luck productizing Course Signals.) Anyway, the point is that if you want to wag your finger at me for looking at a vendor’s marketing uncritically, then you’d better come at it with some data that shows you have a deeper understanding of the issues. Maybe you actually work at Purdue or Ellucian on Course Signals, and maybe you have some deeper knowledge than your comment shows. But so far, all you’ve contributed is invective.
Is Blackboard’s Retention Center as strong as Course Signals? No. I never claimed that it was. (I’ll leave others to make the decision about what it means for the two products to be “in the same league.” I don’t consider Course Signals to be a product, so I don’t think it’s even a meaningful question.) Is what Blackboard is announcing something that every LMS should have? Yes, it is. Is this something that every LMS does have? No, it is not. You can sneer at this late and basic addition to the product if you like, but that doesn’t change the state of the market. If you want this sort of feature to become “table stakes” in LMSs, then the best way to encourage that is to give credit when one of vendors implements it.
Anyone who has been reading this blog for a while is probably aware that I am not particularly known for taking Blackboard’s (or anyone else’s) press release at face value. Just because I have a different opinion than you do on the importance of this announcement doesn’t mean that I have read it uncritically.
Michael,
The devil is in the details and you have not done your homework as an analyst. You are giving BB a free pass. At a time when all vendors, not just Blackboard, are making inflated claims about analytics you need to call them on it, not in terms of generalities (e.g. your piece on Big Data) but the specifics.
Whether or not you have taken press releases on face value in the past is not relevant in this instance. What BB is touting is trivial. It’s not analytics, except in the most rudimentary sense, and shallow analytics at that. It’s trivial descriptive statistics and, if used as the basis for an “early alert” system, it can be misleading, impractical, or both.
What BB has produced apparently is a simple set of reports overlaid with database queries that an average undergraduate programmer could write in an afternoon. I can be persuaded otherwise but neither BB nor you have given any details as to why “Retention Center” is not trivial.
Let’s take logins. Logins by user are recorded in all LMSs. Trivial. Writing a query to get login frequency and comparing it to average logins by other students in the class. Trivial. Let’s take completions. Completion rates by student is recorded in all LMSs. Trivial. Writing a query to get completion rate for a student and comparing it to average completion rates by others students in the class. Trivial. Ditto for course activity and grades.
By saying it’s trivial I am not arguing that these reports should not be in the LMS. They need to be there. But let’s not confuse simple descriptive reports with true learning analytics.
BB’s achievement is not a first step towards predictive analytics any more than riding a bicycle around the block several times without getting tired is a first step towards launching a rocket to the moon. Descriptive statistics and inferential statistics (predictive analytics as a species of inferential statistics) are completely different beasts.
Yes. logging in, completing your work, participating, and getting grades correlate with success. As a teacher I don’t need academic studies to tell me that. But family income, a student’s financial aid status, and tens of other variables also correlate with success. The trick is to know how to weight the variables for the individual student in an individual class. The only practical way to do that is through machine learning. (Note: I am a Big Data optimist, not because of vendor hype but because I have been following Ryan Baker’s work and those of other researchers in the learning analytics space.
Let’s take a concrete case. I regularly teach large lecture courses. Would I benefit from a simple report that shows which students have not logged in on average compared to other students in the same class? Yes and No. The answer is “No” as an “early warning” system. Here’s why. In a class of 200 students what if the report shows that 30 students are below average in login. But another set of 25 students (with possible overlap) are below average in completing their work on time. But then another set of 20 students (with overlap) are below average in class activity. And another 30 students are below average in course grade. Ok. So I have to make sense of a list of 50 students “at risk” according to at least four different criteria. What’s the action I am supposed to take? Are all 50 students “at risk” of not doing well because they are below average in at least one of the dimensions? There are at least ten other dimensions that I can think of at the top of my head. How many variables did the PAR framework identify as relevant? Do you want to give me reports for all them. If I had reports tracking those individually probably all students in my class would be “at risk” in at least one of these dimension throughout the semester. All you have done is introduce further noise that I have to deal with.
That’s not to say that reports like these are not valuable. If a student walks in during office hours and seems to be struggling with the material, I will benefit from knowing their engagement pattern. It’s not uncommon for students to believe that they are working hard and “putting in the time”, particularly at my university. They are quite surprised when I present them with LMS-based data (I wrote my own LMS which I still use from time-to-time) showing them that they haven’t put in as much time as they had thought.
I don’t mean to be contentious but BB has produced is a set of trivial usage reports under the cloak of predictive analytics.
I agree that Blackboard’s product is overwhelmingly likely to be “a simple set of reports overlaid with database queries.” One could say the same thing of Google Analytics. You want to apply a strict definition to the term “analytics,” but the definition you want to apply is not consistent with common usage either in general or within ed tech. The complexity of the underlying technology is not relevant. To my mind, the differences between reporting and analytics have to do with the degree to which the information is actionable and the degree to which there is an underlying predictive model of some sort.
You mention that you could think of “at least ten other dimensions” beyond the four identified by Blackboard as being potentially relevant, and then you bring up PAR. This elision is problematic. As far as I know, PAR identifies risk factors in the longitudinal systems. There are a number of relevant data elements in those systems that can be identified and compared fairly straightforwardly once you filter out the idiosyncratic setup of institution-specific SIS configuration. These dimensions will generally have clear semantic aspects that are directly relevant to a student’s risk level. Is this student on academic probation? Is she maxed out on her course load? Is she maxed out and working a full-time job? Does she live far off campus, away from the campus support network? There are indeed a significant number of these factors. Furthermore, the mix that results in maximum predictive power varies on the specifics of an institution and its student population. For this reason, data mining is useful on an institutional data set as part of an early warning analytics solution. It is also probably the main reason why Course Signals isn’t really a product so much as it is a solution that has a product component wrapped with some consulting. Setup can be a real challenge for this sort of thing.
In contrast, there is very little that is legible in the clickstream of an LMS. It is vastly noisier. In fact, the four factors Blackboard has focused on are the only four I know of that are reliably predictive. You can make a product out of it because there’s just not a lot to fiddle with in order to make it better. No ongoing data mining is necessary.
You make a good point about weighting, but again, that’s only necessary when creating the initial model. I am deeply skeptical that there is going to be significant and identifiable variation that would require machine learning on an individual school’s active data set in this regard. There’s way too much noise to get meaningful fine-grained readings. What you are demanding would be too costly and cumbersome to be included as an LMS feature—particularly an LMS that is locally installe—for limited (if any) gain. Once you’ve developed a model on a large enough data set, you can implement it as a set of simple reports and still most (if not all) of value from it.
I will grant you that there’s no indication in the release that these risk factors are balanced by a model to produce one signal, the way Course Signals does it. But with only four indicators, I don’t think that’s a huge problem. The bigger problem, in my view, is that the system setup seems to encourage instructors to set up their own alarms and sensitivity levels. While I can see the utility for faculty, I would prefer to see that separated from the indicators that are research-based.
I’m confused about your reference to Ryan Baker. A quick perusal of his CV and research suggests that he comes from a cognitive tutor/ITS background. In other words, he has the same pedigree and outlook as Bill Jerome, who just wrote a guest post about the limits of clickstream data analytics. Showing that machine learning can be helpful in one area of learning analytics does not mean that it will be helpful in every area of learning analytics.
Machine learning is not voodoo magic. It doesn’t always add massive value. Sometimes, a simple set of reports, based on insights that were originally gleaned from some data-driven model and presented in a way that is clear and actionable, add far more value than the machine that goes “ping!”
Let me say at the outset that I don’t mean to be shrill. I admire your blog and I admire your writings. I hope what I have to say can be viewed as constructive criticism.
I think if someone is an “analyst” then they should approach their role as journalists. Am I interested in hearing your free-ranging opinions on educational technology? Not really. Start with the basic facts of the story and then base your analysis on that.
When it comes to the BB story your initial posting makes it appear that they originated something new and interesting (“I am glad Blackboard is stepping up.”). There is nothing new and interesting about what BB has achieved. I am not trying to pick on BB. I am just evaluating what they have achieved.
Your lack of attention to detail is again disappointing. You didn’t challenge my assertion that what BB has done, or seem to have done, is expose for their users four trivial usage reports. To say that the reports are trivial is not to say that they are unimportant. A natural follow-up would have been pose to other LMS-providers: “Do you have such reports. If not, why not?” Are similar reports available in Desire2Learn, Moodle, Sakai, and LoudCloud? Give your readers some perspective.
But that’s not what I mean by lack of attention to detail. You completely evaded my use case about an instructor who has to make sense of all these reports telling her that students are at-risk. I am an instructor and it’s a bit like a pilot who has instrumentation going haywire (“false positives”), telling me that my altitude, bearing, etc are off. Risk flags going off all over the place is not actionable. It only leads to paralysis.
This is a subtle point but a student who is below average in logins compared to other students in her class is not necessarily at risk. Perhaps BB has come up with something interesting. A “detail” question to them could have been: “So what criterion do use to set the risk flag, for example, for attendance? Is it the same threshold for every course at every institution.” Yes. Login is a risk category. BUT there is no magical, absolute number that identifies risk. In fact, if BB sets the risk threshold automatically and uses the same threshold for every course at every institution then the approach is statistically flawed.
Your readers would have benefited with some basic data on the BB announcement. What is it exactly? How does it work? Are there possible flaws in their approach? Is it comparable to what other LMS vendors have? Does it really advance analytics?
P.S. am surprised you haven’t heard of Ryan Baker. He is founding President of the International Data Mining Society and Julius and Rosa Sachs Distinguished Lecturer at Columbia University.
Here is a wonderful introduction to his work (why and how data mining can make a difference in education):
http://blip.tv/teachers-college-columbia-university/educational-data-mining-predict-the-future-change-the-future-6427568
Ada, I am sorry that I have disappointed you, but I have a simple solution for you. If you’re not interested in my free-ranging opinions on educational technology, then don’t read my blog. I will refund 100% of your subscription fee.
You want magic, but there is no magic. The kind of clarity that you are demanding can’t be gleaned from the dirty, noisy data that is available from an LMS. You are disappointed that I haven’t demanded better analytics, but that is because you apparently lack understanding of what is possible with the data that they have. When I point this out to you, you respond by asking why I haven’t written a survey of every other LMS’s analytics feature set. When I question whether the expert you reference would be somebody who is inclined to hold the views that you are touting, rather than responding by referencing the specifics of his research that could be relevant to your point, you ask how it could possibly be that I don’t know who he is.
I try to do a couple of things on this blog. First, I try to help people understand what we know about what works and what doesn’t. On the “what works” side of things, I have been writing about early warning systems since 2008. On the “what doesn’t” side, I have been writing about the limits of analytics tools so that people will not be fooled by hype or false hope. Clearly, I have not done a good enough job on the latter task yet.
A second thing I try to do is encourage movement in the industry. I try not to let the best be the enemy of the good. If one company has implemented something that I think would be of value, I praise it in the hopes that it will be adopted more broadly. I do point out limitations, as I did in my original post here, but I don’t see the value in shredding what is, on balance, a step forward. If a vendor implements something I think is stupid or worthless, I usually don’t write about it unless there is a specific larger value in doing so, or unless I think there is serious harm being done.
You seem to think that I am a journalist. I am not. I am an activist. I write in the hopes that my posts will help to improve education. If that disappoints you, then I invite you to start your own blog and raise the bar for me. Seriously. I’d be happy to have another voice out there. Right now, when I Google your name, I come up with nothing. If you want to see better writing, then go out and write it. That’s what I’m trying to do.
You’ve made your point. Either people will be persuaded by your critique or they will not.