e-Literate

Present is Prologue

Author: Michael Feldstein

  • Digging into the Purdue Course Signals Results

    Update: Mike has written another post clarifying the intuitions behind his math.

    The spectacular Mike Caulfield casts a skeptical eye on the Course Signals data:

    Only a portion of Purdue’s classes are Course Signals classes, so the chance any course a freshman takes is a Course Signals course can be expressed as a percentage, say 25%. In an overly dramatic simplification of this model, a freshman who takes four classes the first semester and drops out has a has about a 16% chance of having taken two Course Signals courses (as always, beware my math here, but I think I’m right). Meanwhile they have a 74% chance of having taken 1 or fewer, and a 42% chance of having taken exactly one.

    What about about a student who does *not* drop out first semester, and takes a full load of five courses each semester? Well, the chance of that student having two or more Course Signals courses is 75%. That’s right — just by taking a full load of classes and not dropping out first semester you’re likely to be tagged as a CS 2+ student.

    In other words, each class you take is like an additional coin flip. A lot of what Course Signals “analysis” is measuring is how many classes students are taking.

    Are there predictions this model makes that we can test? Absolutely. As we saw in the above example, at a 25% CS adoption rate, the median dropout has a 42% chance of having taken exactly one CS course. So it’s quite normal for a dropout to have had a CS course. But early on in the program the adoption rate would have much lower. What are the odds of a first semester dropout having a CS course in those early pilots? For the sake of argument let’s say adoption at that point was 5%. In that case, the chance our 4-course semester drop out would have exactly one CS course drops from 42% to 17%. In other words, as adoption grows having had one course in CS will cease to be a useful predictor of first to second-year persistence.

    Is that what we see? Assuming adoption grew between 2007 and 2009, that’s *exactly* what we see.

    I’d like to see somebody at Purdue (or Ellucian) respond to the questions that Mike raises. Matt Pistilli, are you listening?

  • More Blogging on Automated Essay Grading

    Sometime guest blogger and friend of e-Literate Elijah Mayfield has another great post up on using machine learning tools in the service of improving student writing over at his company blog. However you may feel about the technology, the exploration that he’s doing raises some important question about what good feedback on writing is. This aspect of educational technology—the fact that it forces us to examine our tacit knowledge of teaching and make it explicit—is one of the things that I value most about the field.

    Give it a read.

  • Effort and Engagement

    I’ve been thinking a little more this morning about the language used by the researchers in the SJSU Udacity report. They focus a lot on student “effort.” But it’s also pretty common in education to talk about “engagement.” From a technical perspective, the researchers chose the better word. “Effort” is meant to be an observable behavior, e.g., how many minutes students put into watching videos or how many homework problems they solved. “Engagement” is a non-observable attitude that might be a cause for differences in effort that we observe between students. But the connotations of these words tend to encourage different sorts of questions. When we talk about a problem with student effort, we tend to ask how we can get students to do more work. When we talk about a problem with student engagement, we tend to ask how we can get students to want to do more work. The former might lead us to solutions such as student reminders and alerts when they are falling behind or changes in schedule to accommodate students with jobs, while the latter might lead to ideas about increased interactivity or changes to the content.

    Just a thought.

  • What Blackboard, Desire2Learn, and Udacity Should Learn from SJSU

    As Phil noted in his analysisof the SJSU report, one of the main messages of the report seems to be that some of what we already know about performance and critical success factors for more traditional online courses also seem to apply to xMOOCs. But how good is the ed tech industry at taking advantage of what we already know?

    Not very good, as far as I can tell.

    (more…)

  • Ray Henderson Update

    As e-Literate readers know, Ray Henderson is no longer President of Learning Platforms and is now serving on the company’s Board of Directors instead. This sort of thing naturally sets of all sorts of concerns and speculation. I had the good fortune to be able to speak with Ray today and have some details straight from the horse’s mouth.

    First, he made it clear that this change was initiated by him rather than the company, and it was for reasons unrelated to Blackboard politics or direction. A lot of people don’t know that Ray’s family still lives in Indianapolis; both his kids and his parents are half a country away when  he is at Blackboard headquarters in Washington D.C. Euphemisms aside, when he told me that he needed to spend more time with his family, I believed him. I also believed him when he told me that both Blackboard CEO Jay Bhatt and company owners Providence Equity Partners have been working hard to find alternative arrangements that would keep him involved with Blackboard in a meaningful way. Ray made it clear that he will continue to be a very active participant in Blackboard product planning, albeit now in a non-operational role. Boards on Private Equity-owned companies are different from those in publicly traded ones in that way; it is not abnormal for a member of the Board of a PE-owned company to be actively involved in decision-making on a focused and part-time basis.

    Ray used a lot of the first person plural pronoun—we—when talking about Blackboard’s work going forward. I didn’t get any less of a sense of enthusiasm for what the company is doing than I did when I spoke with him a few weeks ago. I believe he will continue to have meaningful involvement with the company. Whether the new executives being brought in will perform well remains to be seen, but if they have the benefit of Ray’s perspective and experience in the strange market that is higher education, then parts of his job that he will no longer be doing are easier for a seasoned technology executive from another industry to take on and do well. (In fact, in some cases those parts may be better managed by more of an operational specialist.) There’s no question that Ray’s reduced involvement is a loss for Blackboard, but it’s being mitigated and, most importantly, it’s not an indicator that executive management failed to recognize the substantial improvements in the company that have taken place on his watch.

    Beyond that, we can make some reasonable guesses about what Ray’s future might look like. In addition to a continuing part-time commitment to Blackboard, I wouldn’t be surprised to see him involved in due diligence evaluations for Providence when they consider acquiring ed tech companies. Nor would I be surprised to see him involved with other ed tech companies. He mentioned, for example, that he is on the Board of In The Telling. Ray is still a fairly young guy and seems as fired up as ever about education and entrepreneurialism. I doubt that he is going to fade away any time soon.

  • Blackboard Analytics Update

    In my last post, I promised that I would give an update specifically on the state of Blackboard’s learning analytics. Well, here you go. This is a summary of what I learned about their product from a chat with Mark Max, Blackboard’s VP of Learning Analytics and, to a lesser degree, with VP of User Experience Stephanie Weeks. I wrote about Blackboard’s Retention Center product some time ago. That product (or feature set, since it is free in Blackboard) directly competes with Desire2Learn’s Student Success System. This post is more broadly about their Analytics product suite, which is most directly analogous with Desire2Learn’s Insights product, although it is actually much, much broader in scope.

    The short version is this: Blackboard has very solid and reliable technology base from which they are building their learning analytics. It is easily the most mature platform among the LMS providers from that perspective. What they are a little short on is vision. In other words, they are pretty much the mirror image of Desire2Learn.

    (more…)

  • Yup, Something (Good) Is Up at Blackboard

    When a company the size of Blackboard makes substantial organizational changes, it can be difficult to assess what is really going on. In the beginning, the stories tend to look similar. The old CEO…decides he wants to “spend more time with his family.” ((Or “hikes the Appalachian trail.” Pick your favorite euphemism.)) Well-known long-time employees leave the company en masse, some of their own volition and others not. It’s a common enough story trope, but it comes with two distinctly different endings. In one version, the company accelerates its downward spiral until it crashes spectacularly. In the other version, everybody is amazed at the company’s revitalization and they live happily ever after. It is nearly impossible to tell from the beginning of the story how it will end. All of the people leaving the company are, of course, unhappy and are likely to have negative opinions of what’s going on. And often they have valid criticisms, even in the stories that later come to happy endings. When an executive is trying to turn around a billion-dollar company quickly, a scalpel won’t do it. Some meat cleaver work is necessary. Collateral damage is inevitable even in the best of cases. So reports from former employees are interesting but don’t tell the whole story. Customers won’t see the results of the changes for a while, so there will be few clues there. In fact, they may see things get worse before they get better due to the chaos of the reorganization. And even the employees who are on the inside often don’t know what to think in the early stages.

    And so it has been with Blackboard. Up until recently, it has been very hard to tell which way the story will go. But I agree with Phil that we’re beginning to see early signs that we may get the happy ending here. I had an opportunity to visit Blackboard this week, and what I heard is very consistent with Phil’s recent experiences with them.

    (more…)