e-Literate

Present is Prologue

Author: Michael Feldstein

  • Research in Translation: Cultural Limits of Self-Regulated Learning

    Research in Translation: Cultural Limits of Self-Regulated Learning

    We are currently facing two civilizational educational challenges. By “civilizational,” I mean that they go beyond country- or region-specific challenges. They are unprecedented in the history of humanity. The challenges I’m talking about are universal  access to quality education and universal lifelong learning, both which are almost certainly ones that you’re well aware of. But we don’t talk enough about how unprecedented they are, what we need to learn to do differently to meet them, and how we could go about learning what we need to learn.

    One plausible solution to both civilizational challenges is to get a lot better at teaching humans how to get better at learning. Again, this is probably not a new or shocking assertion to you. But the research paper I’m going to describe here—”Eight-minute self-regulation intervention raises educational attainment at scale in individualist but not collectivist cultures” by René F. Kizilcec and Geoffrey L. Cohen— suggests that doing so is going to be particularly hard because the effectiveness of different self-education strategies is heavily mediated by contextual factors like culture.

    At the very least, this means that our conversations about trendy approaches like adaptive learning and competency-based education need to become a lot more nuanced than they are right now. We’re not paying enough attention to the things we don’t know yet about the circumstances under which these strategies work.  But more profoundly, the findings of the paper raise questions about whether our fundamental approach to educational research is adequate to the task of learning how to meet these grand educational challenges of our age.

    The Challenges: Universal and Lifelong Education

    The first grand challenge is to give every human on the planet the access and support they need to achieve the highest level of education that suits their individual goals and abilities. No civilization has ever come close to achieving this before, particularly in large, heterogenous culture like the United States. And while the defunding of public college and university systems has increased the challenge, I’m not aware of any evidence that we could meet this challenge with our currently structured educational system at any economically feasible funding level.

    At the same time, the idea of a “terminal degree” being synonymous with the end of an individual’s education is going away. We’ve heard talk for the past few decades about how “knowledge workers” are the future of the economy and how everyone will need to be “lifelong learners” because the skills and knowledge they need will change quickly and constantly. But now we’re really seeing it show up all over the economy and, as a civilization, we haven’t yet developed strategies to deal with it. For example, as the coal mining industry dies, we don’t know how to help all of those miners learn the skills they need to find new careers. Our existing systems are not adequate for those kinds of large-scale educational transitions. So the second grand civilizational challenge is universal lifelong learning.

    These challenges are both partly ones of scale. We know how to help some humans achieve the highest level of education that suits their goals and abilities. We know how to help some miners learn what they need to make career transitions. We just don’t know how to do these things for everyone by growing our current system. Logistically, it’s something we’ve never done before, and economically, it’s not clear that we have the resources to give everybody access to the education they need, never mind at the level of quality that would eliminate any achievement gaps. At least, not with our current system.

    Whether consciously or not, most of the high-profile efforts and many of the discussions around how to address these challenges have been heavily influenced by research that has come to be known as Benjamin Bloom’s “two sigma problem.” Before we can understand the full implications René F. Kizilcec and Geoffrey L. Cohen’s paper, we need to understand the two sigma problem, including some of the limitations of the research and the ways in which it has framed educational reform discussions.

    Understanding the Two Sigma Problem

    Benjamin Bloom is most famous for three contributions to educational research. The first is Bloom’s Taxonomy, which is not relevant to this post. The second is the mastery learning approach, which we’ll get to shortly. And the third is the two sigma problem.

    A “sigma” or “standard deviation” is a statistical concept that has to do with measuring how far from a group’s “average” something is. So it’s a relative concept. Height for an adult human being doesn’t vary nearly as much as, say, hight for all primates. So one standard deviation, or sigma, from the average human height will be a smaller increment than one sigma for average primate height. In Bloom’s world, where he is measuring variation in grades in a class, a very rough proxy for one sigma is one letter grade in a final course grade, e.g., the difference between a C and a B.

    The popular interpretation Bloom’s two sigma experimental result is that students who get one-on-one tutors do better by roughly two sigma, or two final course grade letters, than students in a typical class who do not get one-on-one tutoring. But that’s not quite right. Or at least, it’s not the full picture.

    To start with, the two-sigma research is built on Bloom’s previous work on mastery learning. You may not be familiar with this term, but if you’re at all engaged with ed tech, then you’ve probably seen traces of its influence. The basic idea of mastery learning is that you break a subject up into small learning objectives that are properly sequenced, and students don’t move on to the next learning objective until they’ve demonstrated “mastery”—often defined in terms like “answering 90% or better of assessment questions correctly.” Bloom and his colleagues found that they could achieve a one-sigma improvement for students who were taught using mastery learning techniques over similar students who were not. Students who were taught using mastery learning and received one-on-one tutoring achieved a two-sigma improvement relative to the control groups.

    Textbook publishers love this result because it gives them a direction for product development. They know how to break course subjects down into small, sequenced learning objectives. They also know how to set thresholds on assessments that unlock the next bit of content. Whether educators choose to actually employ master learning techniques is out of their control, but at least they can develop products that are friendly to that approach. And if they could somehow automate the mastery learning pedagocial techniques—say, through adaptive learning, for example—then they could show that their products can improve students’ learning outcomes by two course grades over traditional teaching approaches.

    But this approach and Bloom’s “two sigma” framing are littered with caveats (as research tends to be). First, his control groups were school children in traditional courses. If you vary that environment significantly—say, by putting students in a MOOC or teaching adults pursuing non-degree career development knowledge—I’m not aware of robust findings that Bloom’s results still hold. Second, not all subjects lend themselves equally well to being broken up into small, discrete, and straightforwardly measurable learning objectives that can be determinatively sequenced. Third, as far as learning outcomes go, a “letter grade” difference is not a terribly authentic measure of progress.

    Lastly, and most relevant to the Kizilcec and Cohen paper, Bloom and his colleagues were never able to pin down exactly what it was about one-on-one tutoring that led to that second sigma of improvement. Bloom himself wrote,

    It should be pointed out that the need for corrective work under tutoring is very small.

    So what is it about one-on-one tutoring that made such a big difference? Bloom and his colleagues tried to isolate one or two variables that would account for it. They failed.

    What if they failed because there aren’t just one or two factors that account for the difference that human tutors make? What if there are many different factors that affect different students to different degrees in different contexts? What would this mean for the whole personalized learning enterprise? Even more broadly, if the impacts of teaching interventions vary dramatically, singly and in combination, across a wide range of contextual factors, what are the implications for making progress in educational research? What does a science of learning look like in a world where isolating a variable in a particular experimental condition doesn’t tell us a whole lot that can be generalized very far?

    These are the questions that ultimately interest me in the Kizilcec and Cohen paper. But as with all of these “Research in Translation” articles, we’re going to have to start by unpacking some of the discipline-specific knowledge that underpins the study itself. Some of it will probably be unfamiliar to you. For example, there’s a good chance that you haven’t run across the theory of cultural dimensions that the paper draws upon unless you’re a social psychologist or a sociologist. On the other hand, if you’re reading this blog, there’s a reasonable chance that you know at least a little bit about self-regulated learning. But even there we may find some aspects or implications that you’re not aware of.

    In fact, let’s start with self-regulated learning.

    Understanding Self-Regulation and Self-Regulated Learning

    Kizilcec and Cohen aren’t concerned with mastery learning but rather self-regulated learning. If we forget about the “learning” part for a moment and focus on the “self-regulation” part, the concept is familiar enough in our daily lives.

    Suppose that I want to lose 20 pounds. There are any number of strategies that I could employ try to get myself to where I want to be. Here are a few:

    • Plan a beach vacation six months from now and think about how I want to look in my bathing suit
    • Read scary articles about the consequences of being overweight
    • Promise myself I will buy something I really want if I achieve my goal
    • Commit to giving money to a cause I hate if I don’t achieve my goal
    • Do a 20-minute cardio workout four times a week
    • Increase my fiber intake
    • Fast 8 hours out of every day
    • Reduce my sugar and carbohydrate intake
    • Adopt the cabbage soup diet
    • Wear ear magnets

    Notice that there are two basic kinds of strategies on this list: motivation and implementation. Things that make me want to take action and actions that I can take.

    Let’s say I decide that I’m going to try to lose weight by wearing ear magnets. Every morning I put them on, and every day I weigh myself. After a month, I have not lost weight. Being the introspective person that I am, I come to the conclusion that ear magnets are not helping me achieve my goal.

    Next, I try reducing my sugar and carbohydrate intake. There’s only one problem: I can’t get myself to stick to the plan. Every morning, I reach for a bagel or muffin for breakfast. Whenever I’m out for dinner, I just can’t stop myself from ordering dessert. A month later, I still haven’t achieved my goal.

    So I decide I will promise myself that I will buy that remote controlled submarine drone I’ve been lusting after if I can lose weight by giving up those bagels and desserts. That will be my next experiment. And so on.

    That’s self-regulation in a nutshell:

    1. Set a goal
    2. Pick a strategy that might help you achieve your goal
    3. Monitor your progress toward your goal
    4. Reflect on whether your chosen strategy has resulted in satisfactory progress toward your goal
    5. Adjust your strategy (or stick with it) accordingly
    6. Rinse, repeat

    Apply this basic self-regulation approach to learning and you have…wait for it…self-regulated learning!

    cycle_self-regulated_learning.v2_456

    When I wrote at the top of the post that one plausible answer to our civilizational education challenges is to get a lot better at teaching humans how to get better at learning, that basically means getting a whole lot better at teaching students how to be self-regulated learners.

    SRL post-dates Bloom’s work on mastery learning and it reflects a different focus. Mastery learning is primarily focused on the content that is being learned. Any consideration of learner motivation is a means to an end. In contrast, SRL is focused on achieving the learner’s goals more effectively. The goal may very well be to master some coherent set of content. But any consideration of content is a means to an end. Specifically, the learner’s end.

    At least in theory. In practice, there are lots of attempts underway, with varying degrees of self-awareness, to marry mastery-based adaptive learning products with SRL techniques. For example, here’s an effort at Essex County College in Newark, NJ in which they recruited John Hudesman, a researcher in SRL, to marry the approach with an essentially mastery-based model:

    The basic idea is that maybe the SRL feedback loop can help achieve that elusive second sigma. It’s more sophisticated than Bloom’s hunt for the one or two factors that account for one-on-one tutoring’s effect on all students in the sense that each student can individualize and hopefully find for herself the factors that will enable her to reach her goals.

    But here, too, there are caveats.

    The Limits of SRL

    Let’s go back to that weight loss self-regulation problem. Suppose I’ve tried everything. Ear magnets. The cabbage soup diet. Exercise. Weightwatchers. Promises. Threats. Nothing works. In every case, either I can’t stick with the plan or I stick with it but don’t lose weight. Maybe, after many months of frustration, I talk to my doctor, who tells me that a side effect of my prescription medication is weight gain.

    Oh.

    Now I’m in uncharted territory. Can I take a different medicine? Is there some other way to counterbalance the effect of the medication? Or am I just stuck being 20 pounds overweight?

    There are limits to the power of self-regulation, including the power of self-regulated learning. If I’m a single parent working two jobs and driving for Uber in between to make ends meet, then no SRL strategy is going to give me time that I don’t have. If I have a learning disability, then I may need help finding SRL strategies that account for my particular situation.

    Here’s the tricky part that gets into the heart of the Kizilcec and Cohen paper: Barriers that impact the effectiveness of SRL strategies might be non-obvious and influenced by things like culture. In particular, the paper examines the differences of average effectiveness of a couple of different self-regulation strategies for people from individualist cultures versus collectivist cultures. This distinction will take a little unpacking in order to understand how it affects SRL.

    Suppose the week that I have a big assignment due, my uncle dies. I wasn’t particularly close to this person, but my extended family is generally close-knit. My relatives would be hurt if I didn’t come. I really value the closeness of my extended family, even if I didn’t really get to know my uncle well, and I care a lot about how they feel. It’s not really an option for me to skip the funeral. Which is halfway across the country. My sense of social obligation to my family makes it impossible for me to employ the self-regulation strategy of setting aside the time I need to complete my schoolwork.

    Imagine that your whole life were filled with these sorts of social obligations. Not just an occasional death in the family, but daily demands that you can’t predict and can’t ignore. In your world, there are many people who can ask you to change your plans at any time. Extended family members, neighbors, coworkers, or even strangers can place demands on you and, depending on the specifics, you really can’t say “no.” Because you live in a culture that places a high value on social bonds, and you have learned to place a high value on those bonds as well. The term of art for this kind of a culture is “collectivist,” and it contrasts with an “individualist” culture, where less emphasis is placed on social expectations and more on individual achievement.

    At the very least, you will find yourself in a similar problem to one of struggling to lose weight when you’re taking a medication that causes weight gain. Time management strategies don’t work because you’re really not in control of your time. But it may go even deeper than that. If your world is highly unpredictable because you can’t anticipate demands that you can’t reject and that come at you on a daily basis, then your fundamental idea of what it means to “manage time” may have to be different. You can’t just schedule certain nights of the week or reserve X hours to do your homework. You don’t have that power. Or, at least, it wouldn’t occur to you that exercising that power is a viable option, because you care deeply about what your willingness to fulfill your social obligations says about you as a person.

    This is exactly the hypothesis that Kizilcec and Cohen wanted to test. They wanted to see whether there are differences in how SRL works for students in collectivist versus individualist cultures.

    The stakes are high. Remember those two civilizational challenges: universal and lifelong education. To meet those challenges, we need to provide less traditional and formalized teacher support than the students in Bloom’s control groups got. We just don’t have enough teachers and classrooms to go around. This is a fundamentally different challenge than the one that Bloom was probing with his two sigma experiments. If it turns out that all kinds of factors, some as subtle as the the nature of the social ties in the culture you come from, can impact the effectiveness of our ability to teach students how to teach themselves, then how can we possibly meet our unprecedented civilizational goals? How can we tease out all the many possible factors, particularly when they interact with each other? How would we even begin to go about figuring that out in a rigorous, evidence-grounded way?

    Hold that thought. We’re going to return to it later in this post. First, though, we have to understand the experiment that Kizilcec and Cohen conducted to test whether this is even a problem. And to do that, we first need to understand one piece of social psychology.

    Understanding Cultural Dimensions Theory

    Kizilcec and Cohen wanted to figure out a way to test whether, on average, students from individualist cultures benefit more from being taught SRL techniques than students from collectivist cultures. The first thing they needed in order to do that is some way to define individualist versus collectivist cultures in a reasonably rigorous way.

    As you might expect, the researchers didn’t just pull the idea of individualist and collectivist cultures out of thin air. There is a body of research literature from which they were drawing. In particular, they drew on the research of a social psychologist named Geert Hofstede. During the late 1960s, Hofstede worked at IBM, where he founded and led the Personnel Research Department. This was at a time when globalization was really beginning to take hold and American-founded companies like IBM were learning how to run divisions in other countries with very different cultures. In the early days, these companies believed that they could train their new international employees on IBM-standard management practices and all would be well. But it soon became apparent that the challenge was more complicated than they thought. People in other countries responded to IBM’s management practices differently.

    At about this time, Hofstede stumbled upon a database of 117,000 attitude surveys from IBM employees all over the world. When analyzed for patterns on an individual level, the data were confusing. But when Hofstede grouped employees by nationality and looked for similarities and differences between national groups, some patterns began to emerge.

    As I have written about before, I worry that our generally low level of statistical literacy means that many of us are prone to misread statistical results and have little confidence in statistical analysis. So I am making a practice of providing some explanation of the statistical methods used when I write up these Research in Translation posts. In Hofstede’s case, he used a method called “factor analysis.”

    We can get a basic sense of the intuition that underlies that method through a common joke. You’ve probably seen lists with titles like “You might be a _________ if…”. The basic idea is that there are funny and non-obvious little traits and experiences that are shared by people of a certain type. When they are not mean, they are often inside jokes. For example, RallyPoint, a site that bills itself as “The Professional Military Network,” has an article entitled, “You might be a veteran if…” Item number five on the list is “You remember laughing at troops who thought 29% APR was good…” I don’t even know what that means, and I certainly wouldn’t think that an attitude about interest rates would be a marker of whether somebody is a veteran.

    Factor analysis starts with a collection of seemingly unrelated variables (like answers on an employee attitude survey) and looks for how closely correlated they are. If a group of variables is highly correlated, then it may be because they are all indications of a hidden or “latent” variable.

    “Oh, you answered ‘yes’ on eight out of these ten seemingly random questions. That suggests that you might be a veteran.”

    Hofstede applied factor analysis to national groups of employees and found evidence of four latent variables, which he calls “cultural dimensions.” (Subsequent research has reproduced Hofstede’s results, and the theory has been refined and expanded.) He called one of the latent variables “Individualism/Collectivism.” It purports to capture the degree to which each national culture has a strong web of the kinds of social obligations and expectations that I described in the previous section. Hofstede has used the cumulative research to create a country-by-country comparative index of his different dimensions, which you can play around with here.

    Kizilcec and Cohen used Hofstede’s country index of the Individualism/Collectivism dimension to provide some rigor to their question about cultural differences impacting the effectiveness SRL techniques.

    Understanding the Kizilcec and Cohen Paper

    The researchers conducted experiments on two MOOCs. In each case, students were given eight-minute tutorials on two SRL techniques: Mental Contrasting (MC) and Implementation Intentions (II). If you recall the weight loss strategy list from earlier in the posts, there were strategies that helped motivate me to do what I needed to do (like promising to buy myself a submarine drone if I meet my weight loss goal) and other strategies to actually accomplish my goal (like wearing ear magnets). MC and II fall into these two respective categories. MC is intended to help students self-motivate by (in the words of the authors)

    …vividly elaborating on positive outcomes associated with attaining a goal (e.g., learning a new skill) followed by vividly elaborating on central hindrances in the present that might interfere (e.g., a busy work schedule). By juxtaposing the desired future with current obstacles, MC can strengthen goal commitment and striving. Insofar as the obstacles to goal attainment are seen as surmountable, MC induces a sense that the desired future is within one’s reach, thereby increasing commitment and effortful goal striving.

    On the other side,

    [t]he II procedure helps people plan how to overcome obstacles and execute goal-directed actions. It encourages people to generate concrete if–then plans. Unlike unstructured planning, an II links a specific situation to a goal-directed action. An example of an II is, “If I feel too tired after work to watch the next lecture, then I will make myself coffee to stay awake.” Forming an II facilitates goal attainment because it increases the likelihood that people will respond efficiently and even automatically to regular obstacles that threaten the completion of their goals.

    Using their short MC and II tutorials, the researchers were able to increase MOOC completion by 32% over the control group in the first experiment and 15% in the second—for students from individualist countries. That’s a pretty remarkable result. If we’re trying to achieve these big civilizational goals of providing every human with higher education and lifelong learning advancement, then the possibility that we could increase completion rates in low-facilitation courses by up to 30% with an eight-minute lesson helping students get motivated and focused is pretty huge.

    That’s the good news. The bad news is that students from collectivist cultures showed no significant benefit. In fact, using India as a country on the collectivist end of the scale, the researchers found that

    relative to US respondents, Indian respondents reported that their social environment was more complex and that they shied away from forming if–then plans. Indian respondents listed more obstacles that could interfere with the goal of achieving a good grade in an online course than US respondents (India median = 4, US median = 3; Kruskal–Wallis X2 = 9.50, P = 0.002). They were also more likely to report that if– then plans oversimplify the complexity and ignore the uncertainty of real-life situations [t(192) = 3.12, P = 0.002, d = 0.45].

    In other words, Indian students were more likely to say that the II self-regulation strategy in general was unrealistic. It didn’t account for the complexity in their lives.

    There’s a lot more to this study than I’m going to cover here. For example, students from collectivist countries showed some benefits from MC when decoupled from II, which is interesting to think about. Generally speaking, I can’t unpack all the background in these Research in Translation posts and still have room to cover all the nuances of the papers themselves (although one of my goals is to give readers enough background that they can read and understand the papers themselves).

    But the headline here is provocative enough. When we think back to Bloom’s failure to find the one or two magic ingredients that tutors add which account for the second sigma of improvement, we can now see that the answer very likely is different from student to student. There is no silver bullet. In fact, there are all kinds of non-obvious factors that influence how well a given educational strategy works for a given student.

    So if we still want to address those two civilizational challenges—or even an intermediate challenge along the way, like closing achievement gaps—then Kizilcec and Cohen’s study raises a critical question:

    Now what?

    Implications

    Right about now, some of you are probably thinking, “So after making me read all of that, your grand conclusion is that students are individuals? Thanks for a whole lot of nothing, buddy.” Fair enough. But meeting these big educational challenges requires us to navigate between a rock and a hard place.

    On the one hand, we don’t want to fall for easy answers. As a culture, we tend to be a little schizophrenic about our attitudes toward education. Even smart people who believe in their hearts that every student is an individual human with different needs and goals can all too easily slip into over-generalizations and solutionism when the conversation turns from individual students to solving large-scale educational problems.

    On the other hand, if we’re committed to solving the big educational challenges, we can’t just shrug our shoulders and say, “It’s too hard. There’s no way to sort out all the factors.” We have to come up with a research approach that accounts for the fractal problem of student differences and how various combinations of those differences affect what works for different students in different learning contexts. The Kizilcec and Cohen paper is one model for what that kind of science could look like. But we need more. A lot more.

    In my view, we need what I call “empirical educators” and what Candace Thille calls “citizen scientists.” We need to crowd-source this problem by recruiting front-line classroom educators as field researchers that work in cooperation with researchers trained in learning sciences. For example, Kizilcec and Cohen’s experiments, however cleverly designed, can’t tell us how these results play out in courses that are not MOOCs. Or with students who are first-generation Americans whose families come from collectivist societies. Or whether other factors influence whether there are indentifiable factors that tell us which students from a collectivist country are most or least likely to be similar to their cultural norm in terms of SRL. Or what other SRL techniques might be more effective given any combination of these variables. The amount of research one can imagine being generated off the results of this one study alone is massive.

    We spend a lot of public and private money chasing silver bullets in education. I propose we would be better served by investing that money in providing educators with the training, support, and incentives to participate in the work of advancing the sciences of learning. At the very least, all professional educators should have a certain level of literacy on what we know about education, be able to read and understand the implications of a research paper, and believe that having this knowledge and these skills is a core part of their professional identity. Nobody should still be talking about learning styles, for example.

    Some educators may take this a step further and learn how to make the classroom experiments that they intuitively conduct on a regular basis a little more rigorous. And some may actively collaborate with professional researchers or even design their own studies using the disciplinary research tools that they already know from their graduate training.

    There won’t be one answer for every educator, any more than there will be one answer for every student. My country doctor primary care physician has a different relationship to medical science than an oncologist working at Memorial Sloan Kettering Cancer Center. But they both believe that having some relationship to medical science is essential to doing their jobs. In education, where the fractal nature of the problems we are trying to understand requires us to run many experiments in many contexts, having all educators see themselves in some sense as citizen scientists is even more critical.

    This post is part of our Research in Translation series, which is funded in part by the Bill & Melinda Gates Foundation. The findings and conclusions (or views) contained within are those of the authors and do not necessarily reflect positions or policies of the Bill & Melinda Gates Foundation.

  • BbWorld Report: Blackboard May Be Turning Around

    BbWorld Report: Blackboard May Be Turning Around

    We’ve written similar headlines after past BbWorlds only to be disappointed, so it’s prudent to be cautious. We also need to be clear on what “turning around” does and does not mean. That said, this time feels different. ((Disclosure: Blackboard is a subscriber to our LMS analysis service.))

    Kinds of Evidence of LMS Supplier Health

    Given the cautions above, it’s worth taking some time to look at the types of evidence we gather and what each type can or cannot tell us before diving into the conference analysis:

    1. Changes in adoptions and market share: For investors and competitors, these are the measures you are trying to predict (from among the measures that we usually talk about at e-Literate). The other measures just help to anticipate changes in these ones. For customers and prospective customers, these indicators are useful but less dispositive. They provide a reasonably good sense of how stable the provider is and how well received the product and provider combination are being received by the wide world of current and potential customers. For all audiences, they are trailing indicators. They provide hard, objective data about decisions that colleges and universities have made but not about decisions that they are about to make.
    2. LMS evaluation processes and RFPs: We can learn a lot about imminent change in adoptions and market share by what happens when colleges and universities start LMS evaluations. Which LMS vendor’s current customers are going out to bid most often? When they go out to bid, which LMSs do they decide to evaluate seriously and which ones do they skip? What kinds of questions do they ask? How well do the vendors respond to the evaluators’ questions, and what do the evaluators make of the vendors’ answers? The evaluation data aren’t always good predictors of how happy the schools will be with their choices—a lot depends on how well they run their evaluation processes—but they do give us some indications about how well the vendors understand their customers’ and prospective customers’ needs and perspectives, as well as some information about how well they are executing as a company. This can also be something of a lagging indicator for current and prospective customers because real substantial changes in the company are generally transmitted from central management outward and can reach the sales force last with the use of professionell coaching to raise sales.
    3. Customer sentiment: Are customers happy? Do they feel like their supplier is responsive? Have they noticed a change in responsiveness (good or bad)? How have their recent experiences been with new releases and support services? Customer sentiment tells us how the company is performing for customers right now, but it’s hard to gather in more than an impressionistic way.
    4. New features and other anouncements: There are two primary types of questions these announcements can help answer for current and prospective customers. First, is the supplier filling gaps or fixing problems that will impact customer satisfaction (and therefore demonstrating awareness of the areas where they are underperforming)? Second, what does the pattern of announcements tell us about where the supplier think customers’ new and future needs will be and which of those needs they think they can fulfill? New announcements are a leading indicator of company direction. We will occasionally provide our initial opinions about the quality of the new features, but those should be taken with a grain of salt. We don’t believe we can get a reliable read on the “quality” of any feature until a variety of customers have used it in real-world situations.
    5. Management public and private presentations and discussions: This is probably the most subjective but also potentially the most revealing leading indicator of both company focus and likelihood that their quality of execution will improve or deteriorate. It has the most value when it is interpreted in the context of the previous four types of indicators.

    Of these indicators, we don’t get a lot of new information on the first two at LMS conferences. We get our data on the first one primarily through our data partnership with LISTedTECH and the second one primarily through a combination of the LISTedTECH partnership and our experiences consulting for colleges and universities on their LMS RFP processes. So I’m going to give a brief(ish) summary of these two and then spend the bulk of the post on the last three.

    Context: Adoption and RFPs

    Let’s be clear: In the United States and Canada, Blackboard is playing defense. For now, they are focused on reducing the number of current clients who go to RFP and, of those who do, increasing the percentage who stay with Blackboard. Winning new clients is something they’d like to do, of course, but they’re more focused in the short term on not losing clients. Everybody knows it. Multiple senior Blackboard executives acknowledged the fact or even volunteered it to me on the record at the conference. They are not getting many new implementations:

    New Implementations NA CC

    And of those new implentations they are getting, most are conversions from their legacy ANGEL platform:

    Chord NA CC

    In terms of evaluations, we haven’t seen evidence of Blackboard bottoming out yet. When non-Blackboard customers go to RFP, many of them don’t include Blackboard on their candidate short list. Those that do generally don’t pick Blackboard. (I’ll have more to say on this later in the post.) Meanwhile, Blackboard customers continue to go to RFP. A subset of those have already decided that they will not consider Blackboard. This should not be interpreted as a clear sign that Blackboard isn’t improving; some customers just reach the end of their patience and are no longer persuadable. What it does suggest is that, if there is substantial improvement, it is relatively recent. Nevertheless, we’re not yet seeing Blackboard change their win rate the way we are beginning to see it with D2L.

    There are two major caveats to all of this. First, Blackboard scored a major win with the University of Phoenix’s new adoption of not just Blackboard in general but Ultra in particular. ((Disclosure: University of Phoenix is a consulting client of MindWires.)) While the university is no longer the juggernaut that it once was, it is nevertheless still huge. We will continue to consider them a prospective client until they have actually migrated at scale. Blackboard was able to announce that from the main stage at BbWorld the University of Phoenix will be migrating to Ultra in the fall, which indicates progress, but the proof of the pudding is in the eating.

    The second major caveat is that we are about to enter a new school year and, with it, a new round of RFPs. The migration pattern may change going forward. This is where the forward-looking indicators at the conference may provide us with some clues.

    Customer Satisfaction

    This BbWorld seemed to lack the same simmering discontent that characterized their conferences in the recent past. The attitude seemed neutral-positive. I heard consistently that more problems are being fixed with 9.x and the feature gap that have caused some schools to pass on Ultra is narrowing. There had been some issues with early migrations to SaaS that scared some customers away from considering it for the time being, but nobody I talked to was ruling it out; they were just waiting to be sure that the kinks were worked out first. Schools who migrated more recently seemed to have a better time of this. By the way, the migration issues were brought up by Blackboard executives on the main stage. As with the admission of playing defense on keeping customers, this is an example of a new public honestly that I observed from the company. I’ll have more evidence of this later in the post.

    How many customers have migrated to SaaS? Blackboard claims a little over 200 have made the switch so far with over a hundred more either planning to migrate or piloting:

    Screenshot 2017-08-12 12.52.25

    So that’s a good sign.

    By the way, here’s a third data point in terms of Blackboard’s honesty: One Blackboard executive, after bringing up the AWS competency certification that’s noted on the slide above, volunteered, “That’s not a differentiator. Our competitors have this certification too. It’s more that it would be a bad sign if we didn’t get it.”

    Huh.

    Late in the conference, I was able to speak with several customers who had had private meetings with Blackboard during the week. The common themes were improvement, honesty, responsiveness, and not-there-yet-but-getting-there-pretty-fast.

    I have one other observation that doesn’t fall squarely under the heading of customer satisfaction but is related and also foreshadows some of the other observations I’m going to cover in this post. I went to several analytics sessions at the conference, including one at the Moodlemoot—yes, there was a Moodlemoot inside BbWorld; more on that later—and the discussions were interesting. I can’t remember being at an LMS conference where the Q&A portion of the analytics presentations were fulsome debates about the value, adoption, and ethics of learning analytics rather than on product features. But that’s exactly what I saw at BbWorld this year. It was almost as if I was at a conference that was about teaching and learning. The only other LMS community where I’ve seen multiple conference talks that were both grounded in pedagogy and theory focused (as opposed to “this is how to implement this pedagogical approach using this tool”) is (ironically enough) the Moodle community.

    Announcements

    Blackboard’s slides showing recent progress on both Original Experience for Learn and Ultra have a steady-as-she-goes feel to them:

    Screenshot 2017-08-12 13.25.20

    Screenshot 2017-08-12 13.25.46

    The Ultra mix of new features is odd in an interesting way. On the one hand, the fact that the company only added fill-in-the-blank test questions and media capabilities in the rich text editor last quarter screams “Caution: Wet Paint.” On the other hand, “discussion insights” is an embedded analytics capability unique to Blackboard (as far as I know) that helps instructors sort through active discussions in large classes. The latter may be a requirement for their flagship Ultra client—the University of Phoenix—or an indicator that Blackboard is thinking differently about what they want to be considered a fundamental differentiator for Learn. Or both.

    Blackboard Collaborate, their webconferencing platform, got a lot of love at the conference too. Our early experiences with the relatively new “Ultra” version of the platform were frankly rocky, but I heard nothing but raves about it from customers. And the company is clearly putting a lot of energy into it:

    Collaborate

    Blackboard has long been a portfolio company with lots of stuff to sell, but this year was the first time I’ve seen them walking the walk on truly integrating those products. For example, I saw a demo of their new Blackboard Instructor mobile app on a tablet which showed a workflow of a professor sending an announcement to students reminding them that a synchronous session was about to start and then launching an embedded Collaborate session. It was pretty slick.

    In some ways, this fits a pattern that Phil noted in his D2L Fusion post that Instructure’s competitors are finally catching on to the notion that ease-of-use is not just a marketing slogan or another bullet point in a long list of bullet points. Perhaps the most revealing moment of the conference in this regard was when Blackboard’s VP of Teaching and Learning Phill Miller showed me this graph of the usage of Blackboard’s SafeAssign antiplagiarism tool:

    SafeAssign

    What happened in 2016? Blackboard integrated SafeAssign into the core grading workflow.

    I know, I know. You’re thinking, “Wait. You mean it wasn’t for all this time?” Nope. And that’s the point. Quantified, even. The value of a feature is only realized when the feature is used, and how often it is used depends heavily on how usable it is. In some cases dramatically so. All the LMS providers are now working to raise their respective games in terms of usability. But Blackboard in particular has some opportunities to increase value because of the breadth of their product suite. To the degree that they can simply improve workflows through better integration between their products, they can unlock a lot of latent value fairly quickly.

    But let’s return to that new Instructor app for a moment. The emphasis is on improving instructor/student communication. Their separate (and older) speed grader-equivalent app will be merged with the new instructor app, but there appears to be a real company-wide focus on connecting humans with other humans in an educational context. This theme carried over into their analytics products—both embedded and stand-alone—which move away from Blackboard’s historic emphasis on reporting and beyond the industry’s fixation with retention early warning into the day-to-day business of helping busy instructors catch important details that they might have otherwise missed. The aforementioned discussion insights is one example. Another is their advisor analytics dashboard, which helps students’ advisors get increased visibility into how the students are doing in all of their current classes.

    Oh yes, and did I mention that there was a Moodlemoot inside BbWorld? There was! There were even slides, presented by Blackboard employees, during Blackboard sessions, about their Moodle-related products. Here’s one:

    Moodlerooms

    Blackboard has built a large part of its global business by buying up major Moodle hosting and support providers in large swathes of the world. As a result, they contribute a majority of the revenues that fund continuing Moodle development by Moodle HQ. Blackboard has historically downplayed this relationship inside the United States—even though they purchased the US’s largest Moodle support provider—for fear of cannibalizing their Learn business. That policy has apparently changed. Moodle even got several prominent mentions in the BbWorld keynote.

    I was able to spend a little time at the Moodlemoot, though not nearly as much time as I would have preferred. It was small and the vibe was a little glum. This isn’t surprising. First, Moodle adoption has been losing steam for a while here (as well as in Europe, though that change is more recent).

    Moodle NA

    Second, vocal elements of the Moodle community, including some in leadership positions, tend to be anti-vendor in general and anti-Blackboard in particular. Having a major US Moodlemoot fit inside a single (admittedly large) room at BbWorld had to be a hard pill to swallow. And Blackboard, for its part, did not always appear to execute well on supporting the moot. I found the Moodle session listings in the BbWorld app to be confusing. Nevertheless, there certainly seems to have been a major sea change at Blackboard regarding promoting Moodle in North America. Time will tell whether the increased efforts toward a more visible and cooperative relationship will bear fruit.

    The last announcement piece I’ll mention isn’t really new to BbWorld so much as it is new since last BbWorld. Blackboard was heavily promoting Ally, the content accessibility tool the company acquired within the last year. Interest appeared to be huge, with overflowing crowds at the sessions.

    So what does all this add up to? I’d say a few things:

    • CEO Bill Ballhaus must have succeeded in convincing the company’s private equity owners to allow him to invest in accelerating product development. There’s no other way all these announcements would have been possible. That’s definitely new and a positive leading indicator.
    • The company is thinking about ways to combine its portfolio of products (and services) to meet customer needs. That’s also new, and a differentiator.
    • Another differentiator is the level of sophistication that Blackboard is bringing to learning analytics, both in terms of the feature set and in terms of the conversations they are having with customers.
    • Both the announcements and the customer sentiment indicate that the company is getting better at both listening and executing based on what they’ve heard.
    • None of this changes the fact that Blackboard is still playing defense, but it does suggest that they may be playing better defense and preparing a strategy that will enable them to go on offense.

    Management Public and Private Presentations and Discussions

    This is the area where some of the most dramatic changes were visible. For starters, the marketing messaging in the keynote was by far the most subtle and sophisticated that I’ve ever seen from Blackboard. Two new taglines were introduced. The first one, “Simply Powerful,” wasn’t really new but rather a revival of the old ANGEL tagline. (I think I still may have that T-shirt, although I doubt I could squeeze into it anymore.) Back in the ANGEL days, the subtext was, “ANGEL is simpler than Blackboard, but it’s also powerful.” In today’s context, the subtext is flipped on its head: “Blackboard is more powerful than Canvas, but it’s also simple.”

    The other new tagline was “Your Partners in Change.” There’s a lot going on here. First, this line was projected up on the screen in huge letters as Bill Ballhaus talked about this year being the 20th anniversary of Blackboard. Also on the screen in the background was a picture of NASA’s Pathfinder spacecraft, which landed on Mars the same year that Blackboard was founded. Ballhaus is an aeronautical engineer by training, a fact that he made very plain in his schpiel. Part of the subtext was, “Yes, we’re Blackboard, but not that Blackboard. And I’m the CEO of Blackboard, but not that CEO.” “Partner” was as important as “change,” because it contrasted with the hubris of the last two CEOs. It also provided a cohesive identity for the company as one that provides an integrated portfolio of products and services that can help its customers respond to changing times.

    But the star of the keynote was not Bill Ballhaus but Blackboard’s Chief Strategy Officer, Katie Blot. Ballhaus acted as host and referred to himself as “chief client advocate,” but he quickly ceded the spotlight to Blot for the substance of the keynote.

    A side note: All of the top three LMS providers in terms of US and Canadian market share have powerful, competent women on their senior leadership teams. Phil mentioned D2L’s Cheryl Ainoa in his recent post. I have mentioned Instructure’s Misty Frost in the past. Like Ainoa and Frost, Katie Blot has a role at her company that is broader than her title suggests. It was good to see her get the spotlight.

    And she did not disappoint. Blot is not an ed tech industry careerist, having come to Blackboard from her previous gig working at the US Department of Education. She left behind former CEO Jay Bhatt’s absurdly grandiose claims that Blackboard would change the world single-handedly—”Your Partners in Change”—while maintaining upbeat energy. Her talk was substantive and thematic, punctuated by video interviews with various senior executives about specific product developments. It was not over the top, gross, or cringe-inducing in any way. In fact, it was…dare I say…quite good.

    Beyond that, the main themes I noticed were honesty and consistency. I’ve mentioned the former already and have more detail to add, but let me first address the latter. As we’ve mentioned in the past, one method we have for evaluating vendors at their conferences is asking a lot of different people the same questions and seeing if we get the same answers. This is a particularly critical litmus test for Blackboard given the mess it has to clean up regarding the confusion between it’s SaaS options and Ultra. I asked a lot of random Blackboard employees about how Ultra is going. Consistently, the answer I got was something like the following:

    Let’s back up and first talk about SaaS….

    [Tells a story about good progress with SaaS, adoption, often mentioning the bump they hit with earlier adopters in the process.]

    Now within that context, let’s talk about adoption of the Ultra experience.

    [Talks about how each customer has their own must-haves before they will even consider Ultra, how Blackboard has a prioritized punch list, and how they have dramatically increased the number of scrum teams working on it to make sure they can meet their commitments to work their way through that punch list in a reasonable time frame.]

    This is a pretty dramatic contrast to two years ago, or even a few months ago when Phil was at a Blackboard conference in Europe. So the company is definitely getting on the same page. Out in the field, we are not seeing the same consistency among the sales representatives during RFP presentations. When I brought this up with Bill Ballhaus, he acknowledged it without hesitation and went on to describe steps the company is taking to correct the problem. (There’s that honesty thing again.)

    There was also a return of Ray Henderson’s progress report card by both Phill Miller and Blackboard’s Chief Product Officer Tim Tomlinson. I believe it’s no coincidence that both of these men are former ANGEListas. There was something of a minor ANGEL take-under at Blackboard when Henderson was President and Chief Technical Officer there. That change stalled out under Bhatt but has apparently been revived under Ballhaus. Miller and Tomlinson have both been promoted, and the center of gravity for Learn development has moved to Indianapolis, the former home of ANGEL. Henderson’s public approach could be summed up as something like “make commitments, measure your progress, and tell the truth.” I saw many signs of a similar philosophy taking root in Ballhaus’s Blackboard.

    The last thing I’ll say—and this is probably the most subjective assessment of this post—is that the employees seemed, for lack of a better word, happy. Not forced, conference-host happy but normal people happy. I-love-my-work-and-like-my-colleagues happy. I have observed lots of folks working at organizations that are under stress or dysfunctional. I have seen them from the inside as well as from the outside of those organizations. There’s a vibe that’s unmistakable. The Blackboard folks I talked to didn’t seem to have that vibe.

    Bottom line: Blackboard’s adoption trend line is undeniably down and likely will continue in that direction for at least another 12 months (if you factor out the likely University of Phoenix implementation, which will skew the numbers). But early and subjective signs suggest a positive change in direction inside the company—possibly a rapid one—that may become more visible externally between now and this time next year.

  • The Giant Inflatable Trump Chicken of Ed Tech

    The Giant Inflatable Trump Chicken of Ed Tech

    Not too long ago, Phil sent me a link to an article in The Atlantic about how an obsession with maximizing clicks through analytics “broke” The New Republic.

    By the way, I know where you probably think I’m going with this already, and it’s a reasonable direction to take, but it’s not the one I’m going to take.

    Anyway, here’s the passage that gets to the heart of the problem:

    One of the emblems of the new era in journalism haunted my life at the New Republic. Every time I sat down to work, I surreptitiously peeked at it—as I did when I woke up in the morning, and a few minutes later when I brushed my teeth, and again later in the day as I stood at the urinal. Sometimes, I would just stare at its gyrations, neglecting the article I was editing or ignoring the person seated across from me.

    My master was Chartbeat, a site that provides writers, editors, and their bosses with a real-time accounting of web traffic, showing the flickering readership of each and every article. Chartbeat and its competitors have taken hold at virtually every magazine, newspaper, and blog. With these meters, no piece has sufficient traffic—it can always be improved with a better headline, a better approach to social media, a better subject, a better argument. Like a manager standing over the assembly line with a stopwatch, Chartbeat and its ilk now hover over the newsroom.

    This is a dangerous turn. Journalism may never have been as public-spirited an enterprise as editors and writers liked to think it was. Yet the myth mattered. It pushed journalism to challenge power; it made journalists loath to bend to the whims of their audience; it provided a crucial sense of detachment. The new generation of media giants has no patience for the old ethos of detachment. It’s not that these companies don’t have aspirations toward journalistic greatness. BuzzFeed, Vice, and the Huffington Post invest in excellent reporting and employ first-rate journalists—and they have produced some of the most memorable pieces of investigative journalism in this century. But the pursuit of audience is their central mission. They have allowed the endless feedback loop of the web to shape their editorial sensibility, to determine their editorial investments.

    Once a story grabs attention, the media write about the topic with repetitive fury, milking the subject for clicks until the public loses interest. A memorable yet utterly forgettable example: A story about a Minnesota hunter killing a lion named Cecil generated some 3.2 million stories. Virtually every news organization—even The New York Times and The New Yorker—attempted to scrape some traffic from Cecil. This required finding a novel angle, or a just novel enough angle. Vox: “Eating Chicken Is Morally Worse Than Killing Cecil the Lion.” BuzzFeed: “A Psychic Says She Spoke With Cecil the Lion After His Death.” TheAtlantic.com: “From Cecil the Lion to Climate Change: A Perfect Storm of Outrage One-upmanship.”

    Neither one of us thought it was something we would blog about at the time. It was just an interesting (and sad) window into the current state of journalism. But in the last 48 hours, I have noticed that my various news aggregators have been filling up with stories that featured some variation of the phrase “giant inflatable Trump chicken” in the headline. The genesis is that a Fox News reporter standing near the White House got photobombed by this:

    Giant inflatable chicken

    Here’s a sampling of recent headlines:

    It goes on and on.

    Every way of earning a living has inherent conflicts of interest that can deform behavior. Phil and I certainly struggle with ours every day in our weird combination roles of consultants and analysts. Even individual employees have them. Taking a stand against your manager or employer can be risky. People tend to do the things that enable them to continue to earn a living and avoid doing things that jeopardize their livelihoods. The giant inflatable Trump chicken problem is a consequence of a conflict of interest inherent in selling ads online. ((See how I did that? Inserting the phrase “giant inflatable Trump chicken” into the body of my text? I just goosed my SEO rating a bit by using redrainseo.agency as always.))

    I’m writing about this for two reasons. First, I want to see what happens. We would never use this strategy to determine what we write so we have no experience with it. As a matter of curiosity, I want to see if our viewing numbers change. (I may have missed the giant inflatable Trump chicken/rooster/cock bubble, so I might try this experiment a second time.)

    But also, I’m curious about whether any of the ad-funded publications that cover ed tech do this sort of thing. Whether there is any—let’s call it Google flocking”—in our corner of the world, or whether ed tech is just to niche-y for that. So if you see any signs of something like this, please let us know.

  • “Alternative Pathways:” How to Rethink Vocational Education

    In Phil’s analysis of California Governor Jerry Brown’s directive for the California Community College System (CCCS) ((Disclosure: The Online Education Initiative from CCCS is a client of MindWires. The views in this and future posts represents my independent views and not OEI’s.)) to “take whatever steps necessary” to establish a fully online college, the punch line was as follows:

    What this points to is that for a new fully-online institution to get to some meaningful level of enrollment (let’s say 20,000) in the same ballpark as these comparison schools, I estimate it would take a full decade at the least….

    None of this analysis is to argue that CCCS should not try to establish a fully-online college. The goal of better serving nontraditional populations – adult students with and without jobs – is worth pursuing on its own merits.

    The numbers do argue, however, for a realistic view on the challenges the face:

    • Fighting against national demographic trends for adult students of community colleges;
    • Trying to avoid cannibalizing enrollment from existing California Community Colleges;
    • Having the patience to support the schools while it take years to grow to a size with meaningful enrollment levels; and
    • Accepting that best case this approach probably recovers less than 10% of the enrollment drop since 2009.

    Remember that the goal of this directive is to reach more non-traditional students. Community College Daily quotes CCCS Chancellor Eloy Oakley as saying

    We have literally tens of thousands of working adults with some college and no credentials and a couple of million working adults who are unemployed or underemployed,” Oakley said. “This is a wonderful opportunity to reach a population that really needs a community college to achieve economic mobility.

    So it’s worth looking at what other options may be available to address this goal. Fortuitously, Tyton Partners recently released a two-part report funded by The James Irvine Foundation called Path to Employment: Maximizing the Impact of Alternative Pathways Programs. [Registration required.] It provides a framework for analyzing the potential and critical success factors of shorter, non-degree and non-certification programs. There are various trendy Sillycon Valley buzzphrases associated with these sorts of programs, like “code academies,” “boot camps,” and “micro-credentials,” but they all fall under the broader heading of a term that has irrationally negative connotations in the United States: “vocational education.” For the purpose of this blog post, I am going to use vocational education and Tyton’s preferred term—Alternative Pathway Programs (APPs)—interchangeably.

    Tyton’s report provides an interesting general framework looking at how to think about these types of programs’ abilities to address the needs of non-traditional students, with a special focus on the state of California. It’s worth taking some time to examine aspects of the report in detail.

    Defining the Problem and the Goal

    First, let’s make sure we’re all talking about the same people, problems, and goals. Tyton’s report doesn’t talk about “non-traditional students” but rather “low-income adults,” which it defines as having the following attributes

    • May or may not be employed
    • Earn less than 200% of the federal poverty level
    • 18 years of age or older
    • Have limited or no exposure to postsecondary education

    I can’t say for certain whether that definition is one that Chancellor Oakley or Governor Brown would accept for the group of people they are trying to help, although I suspect that there is at least a strong overlap. For the remainder of this post, I will use the term “low-income adults” as defined by Tyton, since their analysis is built on that definition.

    There are approximately 7 million people who fit that definition in the state of California, “which accounts for nearly 37% of the state’s entire workforce,” according to Bureau of Labor Statistics numbers cited in the report.

    That’s a big number. How many of those people are touched by various educational programs?

    That grey space represents all the low-income adults in California who are receiving…nothing. But it’s actually worse than that in several ways, as Tyton points out in their report. First, not all of the 2.1 million California community college students fit the definition of “low-income adults.” Second, the average community college graduation rate is low. Nation-wide, it’s less than 30%. ((I chose to refer to the nation-wide graduation rate rather than California’s, largely because the nation-wide number is the one that the report uses.)) Particularly for low-income adults, going to school, incurring debt, and not getting a degree is worse than nothing.

    This brings up a second data fidelity problem, since one does not necessarily have to complete a degree in order to gain economic benefit from coursework. A worker could go back to school for a couple of accounting classes that enable her to get a better job, for example. One person’s degree non-completion is another person’s alternative pathway program. So the numbers Tyton uses as proxies for impact potentially both overestimate the number of low-income adults that California community colleges reach and underestimating the percentage of those it reaches that it economically impacts in a net positive way, although they are accurate enough for Tyton’s goal of painting a broad-brush picture of the magnitude of unmet need. More on the data issues later.

    So we have a severe problem of scaling access, even in a state that is historically known for its heavy investment in education. Even if a new online community campus were created and, under Phil’s most optimistic scenario, added another 20,000 enrollments over the next 10 years, that’s a drop in the bucket even before you consider that not all of those students would fit the “low-income adult” definition and it’s possible that considerably less than half of them would graduate. And this gap persists in spite of California spending “roughly 2.5% of the state’s entire annual budget” on educational programs for low-income students, according to Tyton’s figures.

    None of this is to cast aspersions on either the community college system or the idea of an online campus. Rather, the point is that the challenge is enormous. As Tyton puts it,

    [E]ven if all the spots within [community colleges and the two other identified types of] programs were allocated to low-income adults seeking to enhance their employment prospects, current capacity would support less than a third of the potential annual demand. Expanding the number of successful models that can support education-to-employment pathways for adults is imperative, both from within the current ecosystem of institutions and workforce programs and through new, innovative program models.

    The idea that Tyton’s report explores, which is not positioned as an alternative to existing programs but rather another tool in the toolbox, is what they call APP:

    An Alternative Pathways Program (APP) is defined as one that:

    • Focuses on education and training for specific job and career pathways

    • Maintains close alignment with employers and industries to facilitate job placement for participants completing the program

    • Does not offer a traditional postsecondary degree or certificate

    Most APPs focus on recruiting and serving participants directly, similar to colleges and universities, but they vary widely in their training model and program length, among other attributes. Some program models connect participants directly with employment opportunities. For example, the high-profile technology, design, and data “boot camps” o er a career-fair job connections model and last less than 12 weeks on average, while experiential learning programs such as apprenticeship programs prepare participants over a longer period (most are 6–12 months long) for a specific career path. Other programs do not connect participants with jobs, but they still offer training for specific career pathway and offer a certification or credential upon completion. Another set of programs focuses on general education as a baseline for specific career pathways, including programs or courses that help participants gain postsecondary credentials or learn general skills.

    Tyton defines the common goal that programs under this broad umbrella could solve as creating “a path for improving the economic and employment opportunities and outcomes for low-income adults.” So these are focused, vocational education programs aimed at helping adults stuck at the bottom rung of the income latter achieve enough education to begin climbing to higher rungs. Obviously, this is not the only set of goals one could have for post-secondary education. One could be concerned about developing more informed citizens, enriching people’s intellectual lives, giving them access to career paths that they didn’t know existed, cultivating a common national culture, among others. It is both reasonable and important to talk about different educational goals that we should aspire to. But these sorts of conversations tend to take on a one-goal-fits-all tone. Take a look again at the grey space in that last graphic:

    The people in that space are not currently being served by programs that address any of those goals. Tyton asks two basic questions. First, to what degree might APPs—many of which do not currently serve low-income adults—be recruited to fill in some of that grey space? And second, what are the characteristics of an APP that would make it most likely to achieve this goal? The first question can only be answered in a fairly general way, since we have no systematic tests of it yet. Tyton spends the bulk of their report on the second question by analyzing patterns across over 125 existing APPs.

    I am not going to provide a detailed critique of Tyton’s analysis framework in this post. Rather, I’ll be examining the ways in which having such a framework enables more productive and nuanced discussions addressing big educational problems like helping low-income adults using tools ranging from policy to educational technologies.

    The Six Pillars

    Much of Tyton’s analysis rests on what they call the “six pillars” of alternative pathways programs:

    • Enrollment policies: Processes and guidelines for admitting participants
    • Participant support: Resources and methods that support participants in overcoming life challenges
    • Labor market alignment: Level of program fit with the needs of employers and the local/regional economy
    • Connections: Extent to which program connects participants with employers and other job search resources
    • Training mix: Balance of curriculum emphasis on soft skills vs. academic and technical skills
    • Financial model: Ability to generate revenue and achieve organizational stability

    In and of themselves, there’s nothing earth-shaking about these categories. But stating them explicitly as part of the analytic framework enables us to do all kinds of additional important work.

    First, it enables us to ask, “Is this a complete and plausible list of critical success factors for a vocational program?” For example, the work of Vincent Tinto and others connects students’ sense of belonging in campus community to their likelihood of completing their degrees. Is that principle also operative in vocational programs for low-income adults? If so, how much of an effect does it have? And can it be subsumed under “participant support,” or is it distinct and important enough to merit its own pillar?

    This line of questioning brings up a second advantage of having such a framework, which is that it points to a research agenda. What sorts of participant support are most effective for helping low-income adults to complete APPs? Does the answer to that question vary by context? If so, which sorts of contextual factors matter the most? Tyton has proposed a set of preliminary hypotheses for the optimal way to address each pillar based on the research that they conducted:

    The point of the hypothesis is to have a truth proposition that can be tested. For example, Tyton has listed “employer as payer” as the “optimal model” for vocational programs. Going back to Governor Brown’s online campus proposition mentioned at the top of this post, California has funding mechanisms at its disposal that many of the start-up programs examined by Tyton do not. One might float an alternative hypothesis for that pillar and then test that hypothesis through various means.

    The framework also enables program designers to think about trade-offs more clearly. A classic trade-off is between enrollment policies and participant support. Tyton articulates several alternative models for each pillar. Here’s the selection of models they examine for support:

    But these models don’t exist independently of the other pillars. If your program has an enrollment process that is extremely good at identifying students who come to the program already possessing many of the skills and life circumstances that would enable them to succeed if only they were given the opportunity, then you may need less investment in participant support. Conversely, if the program’s goal is to help those students who have to face the most daunting obstacles to their education, then investment in support becomes more important.

    This example illustrates the point that we should interpret the word “optimal” loosely here. That large swath of grey in the earlier diagram representing all the low-income adults who are getting no education is not one homogenous blob of unmet need but rather a collection of millions of people with different strengths, needs, and circumstances. People in different situations will likely need different program designs in order to be successful. As a sector, we are thankfully past the peak of the idea that we can teach everything to everybody by posting video lectures online and calling them “MOOCs.” ((Not very far past, but still. Baby steps.))

    This brings us to one of the most deeply divisive terms in education: scale.

    The Tyton report examines two types of scale: inputs and outputs. Access and outcomes. Let’s imagine that California were somehow able to get 100% of low-income adults enrolled in community college degree programs. That would be good, right? Well, maybe. Let’s also suppose that, in doing so, California’s degree completion rate settles at the national average of 30% for community colleges. That would mean 70% of California’s low-income adults would end up still without a degree and still (likely) without improving their economic prospects. To the contrary, many of them would have increased levels of debt they would have to pay out of their unchanged salaries. So no, scaling access is not inherently good.

    For each of Tyton’s hypothesized optimal strategies in each of their hypothesized pillars, they analyze the trade-off the strategy makes between scale of access and scale of outcomes:

    If a chosen strategy for a pillar has a downward-pointing red arrow, then it presents a challenge to scaling access. On-site support to students is harder to provide to many students than providing students with no support. It’s harder to scale enrollments—access—for a program that commits to high-touch student support. Then why do it? Hopefully because it improves the percentage of degree completions and other positive outcomes.

    Taken together, the elements of Tyton’s framework give program designers and policy makers both a set of knobs they can turn in an effort to tune a particular program to the needs of a particular student population as well as a lens through which they can examine, test, and refine their assumptions in ways that will improve the effectiveness of those knobs and our knowledge of how the various settings interact with each other.

    Of course, having knobs to twiddle is good, but being able to measure the impact of your knob twiddling is critical. As I mentioned earlier, the numbers Tyton cites for community college impact on low-income adults are less than perfect measures. While adequate for the purpose of assessing the order of magnitude of unmet need in a large state like California, the two proxies the report uses for scale or access and outcomes—number of students served and graduation rate—are not great for measuring impact at a more granular level. And it will be tough to develop better ones. Going back to the example of the person who takes a couple of accounting classes at a community college and gets a better job as a result, how would one capture that? And yet, that is precisely what we would need to do to get clearer sense of the degree to which community colleges are currently serving the needs of low-income adults and the degree to which tweaking community college programs in various ways might increase that impact. Other APP efforts will likely face similar data quality challenges. So application of the model in practice will likely require some innovation around measurement in these areas.

    But here again, having a holistic model can help. Tyton highlights sustainability as the big reason to think about employer-pay business models. But if you happen to be thinking about the data problem while looking at the Tyton pillars graphic, it might occur to you that a direct relationship with employers can help with that problem. To start with, employers’ willingness to pay might be a good proxy for career progression benefits that students gain from their participation in the program. One would have to do the research, but it’s a plausible hypothesis. Second, the closer relationship with the employer makes outcomes data easier to get, perhaps in the form of anonymized aggregate data from the employers or by providing richer, longer term relationships with students that give them more incentive to provide the school with post-graduation follow-up data.

    What About Ed Tech?

    Nowhere does this framework explicitly address ed tech as such. And yet, ed tech decisions both large and small are often made by directly connecting a problem with a technological solution. A state wants to help reach more non-traditional students. Online learning can reach more students. So why not start an online learning program? A foundation or a college wants to help more first-generation college students make it through college. Adaptive learning programs seem to help some students in developmental math programs get past that stumbling block to their degree completion. So why not invest in adaptive learning programs?

    Absent of a richer analytic framework, these efforts are more likely to fail and less likely to be reproducible because they don’t start with either a holistic understanding of the needs of the students the efforts are trying to help or a clear understanding of how various aspects of the support ecosystem interact with each other. Without a theoretical framework, you can’t construct a clear hypothesis. Without a clear hypothesis, you can’t construct a proper experiment. Without a proper experiment, you can’t learn what works and what doesn’t.

    Very often, the best moment to think about ed tech is immediately after you have developed your program design and analyzed it for strengths and weaknesses. At this stage in the thinking, ed tech can potentially help by changing the laws of physics that underpin your model. OK, so low-income adult students need high-touch support which, when implemented in the traditional way, is resource intensive and therefore limits the number of students you can serve. Is there a way that technology can help provide that high-touch support at a lower resource cost? (By the way, the solution may not be to build robot advisors in the sky that can semi-read students’ minds. Instead, it might be saving advisors’ time spent doing other, non-student-facing work so that the same number of advisors have more time to serve students well.)

    I have no strong opinions about the answers that the Tyton paper arrives at, but I do believe that they are asking roughly the right questions in roughly the right order. As a field, we need more of this type of program- and policy-level research and analysis to inform a wide range of strategic decisions, including but not limited to use of ed tech. It is an exemplar of a genre of educational research that we should be looking to grow, propagate, and use to inform practical decision-making.

    This post is part of our Research in Translation series, which is funded in part by the Bill & Melinda Gates Foundation. The findings and conclusions (or views) contained within are those of the authors and do not necessarily reflect positions or policies of the Bill & Melinda Gates Foundation.

  • New e-Literate Genres and Grant-funded Coverage

    Most of our writing here at e-Literate is not directly funded, whether by ads, subscriptions, or sponsorships. We have received grants in the past to produce e-Literate TV videos, but generally not for blogging. We recently received a new grant which includes writing here on the web site as well as video production. It has both stimulated our thinking about what kind of coverage we could be doing on e-Literate and created a circumstance that was not covered by our previous disclosure policy. We’re outlining the scope of our planned coverage here, both because we’re excited to share our plans with you and because we want to update you on how we will handle grant-related disclosure on the blog going forward.

    We will be creating four content genres that will likely be ongoing here at e-Literate, whether grant-funded or not:

    1. Research in Translation: Educational research write-ups tend to be either academic journal articles, which are often hard to find and complex to evaluate for non-experts, or press coverage, which often don’t provide enough information for readers to understand and critically evaluate the scope and significance of the research. Research in Translation pieces will try to strike a middle ground by curating interesting studies, writing them up in less specialized language (or explaining essential specialized language), and providing some explanation of the scope and context of the research, methods, and the nature and strength of the results.
    2. From Mars to Venus: These will be interviews with ed tech product vendors that model the kinds of discussions educators should have when trying to understand the potential impact of the product or service. The goal is to have illuminating and useful conversation by asking questions that don’t require a lot of specialized knowledge to think of or to evaluate the answers.
    3. Learning Together: These case studies will examine ways in which college and university systems, consortia, professional associations, or other academic affinity groups are making conscious and coordinated efforts to share what they are learning with each other, test potential improvements to see how transferrable they are, and spread validated improvements beyond the little pockets in which they begin.
    4. Learning Bytes: These will be short pieces that explain key concepts necessary to understand important concepts relevant to larger issues like the ones covered in the previous three genres of coverage. For example during a Mars to Venus discussion with a vendor, we may be talking about implementation of an adaptive learning tool adapts specifically on the basis of science around getting information into the students’ long-term memory (as opposed to, for example, an adaptive learning product that focuses on identifying gaps students have in their foundational knowledge). We might produce a short Learning Bytes explainer video to accompany the main piece.

    We think these categories will be generally useful for our coverage well beyond the scope of the grant and aspire to use them broadly. But we were spurred to think them up as part of the most recent grant we received from the Bill & Melinda Gates Foundation.

    The broad mandate of the current grant is to cover ways in which technology-mediated education (or “digital education,” in the parlance of the foundation) are relevant to access-oriented colleges and universities that are working to close any achievement gaps. Much of our grant-funded coverage will address these issues directly, although some pieces will have broader relevance.

    Regular readers know we have a policy of always disclosing any relevant financial interests that could be perceived as influencing our coverage. We regularly revisit our disclosure approach as our business evolves to ensure we are maintaining appropriate transparency. And whenever we encounter a new situation that causes us to modify our disclosure policy, we let you know.

    For the duration of the grant, any posts in any of the genres described above will have the following text at the bottom:

    This post is part of our [name of genre] series, which is funded in part by the Bill & Melinda Gates Foundation. The findings and conclusions (or views) contained within are those of the authors and do not necessarily reflect positions or policies of the Bill & Melinda Gates Foundation.

    Should we receive other grants in the future, funded articles will always include a disclosure statement at the bottom. The specifics of that statement may vary based on the policies of the funder, but we will always disclose when a post is funded as well as who the funder is.

  • Fear Itself

    A little over a week ago, I wrote a “recommended reading” post pointing to a piece from Inside Higher ed and The Times Higher Ed called “Fear of Looking Stupid” about research from Carnegie Mellon University anthropologist Lauren Herckis about faculty resistance to “innovative” approaches. (I use quotes here not to imply a value judgement but to indicate that “innovative” was the word used in the article.) The title of the article gives you a pretty good sense of the angle taken by the author. The comment thread on that article was fascinating, as was John Warner’s terrific column in response.

    We are lucky to have a guest post from Dr. Herckis and her colleagues Richard Scheines and Joel Smith with some additional perspective and follow-up information. I took the liberty of adding a title to their post.

    – Michael

    We were delighted that Times Higher Education and Inside Higher Ed reported on the anthropological research being conducted at Carnegie Mellon on the roadblocks to implementation of demonstratively effective pedagogical innovations. We’d like to take the opportunity to expand the conversation.

    Our research exposes multiple factors behind faculty resistance to making changes to their teaching practice, including the institutional barriers encountered by faculty adopting evidence- or research-based practices and especially where technology is involved. There are, of course, many effective teaching practices in use, and current research helps to explain why they work when they do. But that research also points to many other tools and practices that increase teaching effectiveness.

    Our aim is to develop a detailed and actionable understanding of what impedes and what helps faculty adopting research-based practices.

    The idea that faculty are invested strongly in avoiding embarrassment, and are thus sometimes reluctant to adopt innovative tools or practices – as reported in the THE/IHE article – is true. But the story is richer.

    Faculty do not want to waste students’ time; they want to teach well. Using methods that they have honed is therefore important. Faculty learn to teach over years of practice, as most of us have little or no training in teaching. Tried and true methods are appealing because faculty have reason to think that they work.

    If students have seemed to enjoy the material and report learning from the course, why change? Methods that leave students feeling good about the course (and the professor) are appealing, both because they are validating (“yes, I AM a good professor!”) and because happy students provide good evaluations of teaching, which are vital for faculty job security.

    Until we change the incentives and provide alternative sources of personal identity affirmation, faculty will not be motivated to invest time and energy in changing their teaching to adopt practices shown by research to be more effective.

    Our research on implementation of research-based instruction shows that faculty care about their students, and want to ensure that students have a good experience. Yes, some faculty at Carnegie Mellon hesitate to use unfamiliar methods or technology because they don’t want to embarrass themselves in class. Few of us want that. But they also don’t want to waste students’ time if something goes awry, want the validation of satisfied students, take student satisfaction as a sign that things are going well, fear the professional consequences of poor teaching evaluations, don’t think alternatives are a good fit, are sceptical of literature that supports alternatives, and believe that institutional support for alternatives is lacking.

    Innovation for the sake of innovation doesn’t serve faculty or students. But the use of research-based, effective teaching methods does serve students, and it is in our interest to learn how to support faculty in adopting and sustaining the use of such methods.

    To do this, we need to step back, look at the big picture, and address the multiple contributing factors to success and failure in implementing evidence-based practice. Our research shows that systematically incorporating anthropological analysis is an important and rarely used tool for understanding roadblocks to, and enablers of, meaningful innovation in higher education. Without it, we are flying blind.

    The research results that we had time to report in our brief presentation at the Global Learning Council meeting  (and reported in THE/IHE) are only a small part of what we have learned about implementation of instructional innovation. A full report detailing our findings will be available in September 2017 at http://cmu.edu/simon.  Academic articles regarding methodology and results will be submitted for peer review in the coming months; these articles will be shared at http://www.cmu.edu/simon/projects/flagship-projects/barriers-to-tel.html as they are published

    Lauren Herckis is Simon Initiative research scientist and adjunct instructor, Richard Scheines is dean of Dietrich College of Humanities and Social Sciences, and Joel Smith is distinguished career teaching professor, all at Carnegie Mellon University.

  • Recommended Reading: Fear of Looking Stupid

    Inside Higher Ed has summarized some findings of a study by Carnegie Mellon University anthropologist Lauren Herckis about why faculty hesitate to try new teaching practices. (The article in IHE uses the word “innovative,” but I find that word loaded, and since I don’t know if or how it was used in the study itself, I’m going to avoid it for now.) The one that made the headline of the article is faculty being afraid of looking stupid in front of their students, but two other important ones were fear of punishment from bad course evaluations (or “smile sheets,” as they are sometimes derogatorily called) and deep “gut” convictions based on personal experience that they know what good teaching is and will prefer that instinct over the findings of a research article. Read the comment thread as well as the article; the discussion is fascinating (and tends to back up the latter two findings).

    We happened to interview Dr. Herckis for e-Literate TV a while back, and here she is making a related observation:

    I also recommend reading John Warner’s response in IHE about the necessity of being comfortable looking stupid. Great stuff.

    A couple of commenters on the original IHE article asked for a link to the original study. I don’t have one at the moment and am not sure it’s been published yet but will post it when it becomes available.