e-Literate

Present is Prologue

Tag: Research in Translation

  • Research in Translation: Cultural Limits of Self-Regulated Learning

    Research in Translation: Cultural Limits of Self-Regulated Learning

    We are currently facing two civilizational educational challenges. By “civilizational,” I mean that they go beyond country- or region-specific challenges. They are unprecedented in the history of humanity. The challenges I’m talking about are universal  access to quality education and universal lifelong learning, both which are almost certainly ones that you’re well aware of. But we don’t talk enough about how unprecedented they are, what we need to learn to do differently to meet them, and how we could go about learning what we need to learn.

    One plausible solution to both civilizational challenges is to get a lot better at teaching humans how to get better at learning. Again, this is probably not a new or shocking assertion to you. But the research paper I’m going to describe here—”Eight-minute self-regulation intervention raises educational attainment at scale in individualist but not collectivist cultures” by René F. Kizilcec and Geoffrey L. Cohen— suggests that doing so is going to be particularly hard because the effectiveness of different self-education strategies is heavily mediated by contextual factors like culture.

    At the very least, this means that our conversations about trendy approaches like adaptive learning and competency-based education need to become a lot more nuanced than they are right now. We’re not paying enough attention to the things we don’t know yet about the circumstances under which these strategies work.  But more profoundly, the findings of the paper raise questions about whether our fundamental approach to educational research is adequate to the task of learning how to meet these grand educational challenges of our age.

    The Challenges: Universal and Lifelong Education

    The first grand challenge is to give every human on the planet the access and support they need to achieve the highest level of education that suits their individual goals and abilities. No civilization has ever come close to achieving this before, particularly in large, heterogenous culture like the United States. And while the defunding of public college and university systems has increased the challenge, I’m not aware of any evidence that we could meet this challenge with our currently structured educational system at any economically feasible funding level.

    At the same time, the idea of a “terminal degree” being synonymous with the end of an individual’s education is going away. We’ve heard talk for the past few decades about how “knowledge workers” are the future of the economy and how everyone will need to be “lifelong learners” because the skills and knowledge they need will change quickly and constantly. But now we’re really seeing it show up all over the economy and, as a civilization, we haven’t yet developed strategies to deal with it. For example, as the coal mining industry dies, we don’t know how to help all of those miners learn the skills they need to find new careers. Our existing systems are not adequate for those kinds of large-scale educational transitions. So the second grand civilizational challenge is universal lifelong learning.

    These challenges are both partly ones of scale. We know how to help some humans achieve the highest level of education that suits their goals and abilities. We know how to help some miners learn what they need to make career transitions. We just don’t know how to do these things for everyone by growing our current system. Logistically, it’s something we’ve never done before, and economically, it’s not clear that we have the resources to give everybody access to the education they need, never mind at the level of quality that would eliminate any achievement gaps. At least, not with our current system.

    Whether consciously or not, most of the high-profile efforts and many of the discussions around how to address these challenges have been heavily influenced by research that has come to be known as Benjamin Bloom’s “two sigma problem.” Before we can understand the full implications René F. Kizilcec and Geoffrey L. Cohen’s paper, we need to understand the two sigma problem, including some of the limitations of the research and the ways in which it has framed educational reform discussions.

    Understanding the Two Sigma Problem

    Benjamin Bloom is most famous for three contributions to educational research. The first is Bloom’s Taxonomy, which is not relevant to this post. The second is the mastery learning approach, which we’ll get to shortly. And the third is the two sigma problem.

    A “sigma” or “standard deviation” is a statistical concept that has to do with measuring how far from a group’s “average” something is. So it’s a relative concept. Height for an adult human being doesn’t vary nearly as much as, say, hight for all primates. So one standard deviation, or sigma, from the average human height will be a smaller increment than one sigma for average primate height. In Bloom’s world, where he is measuring variation in grades in a class, a very rough proxy for one sigma is one letter grade in a final course grade, e.g., the difference between a C and a B.

    The popular interpretation Bloom’s two sigma experimental result is that students who get one-on-one tutors do better by roughly two sigma, or two final course grade letters, than students in a typical class who do not get one-on-one tutoring. But that’s not quite right. Or at least, it’s not the full picture.

    To start with, the two-sigma research is built on Bloom’s previous work on mastery learning. You may not be familiar with this term, but if you’re at all engaged with ed tech, then you’ve probably seen traces of its influence. The basic idea of mastery learning is that you break a subject up into small learning objectives that are properly sequenced, and students don’t move on to the next learning objective until they’ve demonstrated “mastery”—often defined in terms like “answering 90% or better of assessment questions correctly.” Bloom and his colleagues found that they could achieve a one-sigma improvement for students who were taught using mastery learning techniques over similar students who were not. Students who were taught using mastery learning and received one-on-one tutoring achieved a two-sigma improvement relative to the control groups.

    Textbook publishers love this result because it gives them a direction for product development. They know how to break course subjects down into small, sequenced learning objectives. They also know how to set thresholds on assessments that unlock the next bit of content. Whether educators choose to actually employ master learning techniques is out of their control, but at least they can develop products that are friendly to that approach. And if they could somehow automate the mastery learning pedagocial techniques—say, through adaptive learning, for example—then they could show that their products can improve students’ learning outcomes by two course grades over traditional teaching approaches.

    But this approach and Bloom’s “two sigma” framing are littered with caveats (as research tends to be). First, his control groups were school children in traditional courses. If you vary that environment significantly—say, by putting students in a MOOC or teaching adults pursuing non-degree career development knowledge—I’m not aware of robust findings that Bloom’s results still hold. Second, not all subjects lend themselves equally well to being broken up into small, discrete, and straightforwardly measurable learning objectives that can be determinatively sequenced. Third, as far as learning outcomes go, a “letter grade” difference is not a terribly authentic measure of progress.

    Lastly, and most relevant to the Kizilcec and Cohen paper, Bloom and his colleagues were never able to pin down exactly what it was about one-on-one tutoring that led to that second sigma of improvement. Bloom himself wrote,

    It should be pointed out that the need for corrective work under tutoring is very small.

    So what is it about one-on-one tutoring that made such a big difference? Bloom and his colleagues tried to isolate one or two variables that would account for it. They failed.

    What if they failed because there aren’t just one or two factors that account for the difference that human tutors make? What if there are many different factors that affect different students to different degrees in different contexts? What would this mean for the whole personalized learning enterprise? Even more broadly, if the impacts of teaching interventions vary dramatically, singly and in combination, across a wide range of contextual factors, what are the implications for making progress in educational research? What does a science of learning look like in a world where isolating a variable in a particular experimental condition doesn’t tell us a whole lot that can be generalized very far?

    These are the questions that ultimately interest me in the Kizilcec and Cohen paper. But as with all of these “Research in Translation” articles, we’re going to have to start by unpacking some of the discipline-specific knowledge that underpins the study itself. Some of it will probably be unfamiliar to you. For example, there’s a good chance that you haven’t run across the theory of cultural dimensions that the paper draws upon unless you’re a social psychologist or a sociologist. On the other hand, if you’re reading this blog, there’s a reasonable chance that you know at least a little bit about self-regulated learning. But even there we may find some aspects or implications that you’re not aware of.

    In fact, let’s start with self-regulated learning.

    Understanding Self-Regulation and Self-Regulated Learning

    Kizilcec and Cohen aren’t concerned with mastery learning but rather self-regulated learning. If we forget about the “learning” part for a moment and focus on the “self-regulation” part, the concept is familiar enough in our daily lives.

    Suppose that I want to lose 20 pounds. There are any number of strategies that I could employ try to get myself to where I want to be. Here are a few:

    • Plan a beach vacation six months from now and think about how I want to look in my bathing suit
    • Read scary articles about the consequences of being overweight
    • Promise myself I will buy something I really want if I achieve my goal
    • Commit to giving money to a cause I hate if I don’t achieve my goal
    • Do a 20-minute cardio workout four times a week
    • Increase my fiber intake
    • Fast 8 hours out of every day
    • Reduce my sugar and carbohydrate intake
    • Adopt the cabbage soup diet
    • Wear ear magnets

    Notice that there are two basic kinds of strategies on this list: motivation and implementation. Things that make me want to take action and actions that I can take.

    Let’s say I decide that I’m going to try to lose weight by wearing ear magnets. Every morning I put them on, and every day I weigh myself. After a month, I have not lost weight. Being the introspective person that I am, I come to the conclusion that ear magnets are not helping me achieve my goal.

    Next, I try reducing my sugar and carbohydrate intake. There’s only one problem: I can’t get myself to stick to the plan. Every morning, I reach for a bagel or muffin for breakfast. Whenever I’m out for dinner, I just can’t stop myself from ordering dessert. A month later, I still haven’t achieved my goal.

    So I decide I will promise myself that I will buy that remote controlled submarine drone I’ve been lusting after if I can lose weight by giving up those bagels and desserts. That will be my next experiment. And so on.

    That’s self-regulation in a nutshell:

    1. Set a goal
    2. Pick a strategy that might help you achieve your goal
    3. Monitor your progress toward your goal
    4. Reflect on whether your chosen strategy has resulted in satisfactory progress toward your goal
    5. Adjust your strategy (or stick with it) accordingly
    6. Rinse, repeat

    Apply this basic self-regulation approach to learning and you have…wait for it…self-regulated learning!

    cycle_self-regulated_learning.v2_456

    When I wrote at the top of the post that one plausible answer to our civilizational education challenges is to get a lot better at teaching humans how to get better at learning, that basically means getting a whole lot better at teaching students how to be self-regulated learners.

    SRL post-dates Bloom’s work on mastery learning and it reflects a different focus. Mastery learning is primarily focused on the content that is being learned. Any consideration of learner motivation is a means to an end. In contrast, SRL is focused on achieving the learner’s goals more effectively. The goal may very well be to master some coherent set of content. But any consideration of content is a means to an end. Specifically, the learner’s end.

    At least in theory. In practice, there are lots of attempts underway, with varying degrees of self-awareness, to marry mastery-based adaptive learning products with SRL techniques. For example, here’s an effort at Essex County College in Newark, NJ in which they recruited John Hudesman, a researcher in SRL, to marry the approach with an essentially mastery-based model:

    The basic idea is that maybe the SRL feedback loop can help achieve that elusive second sigma. It’s more sophisticated than Bloom’s hunt for the one or two factors that account for one-on-one tutoring’s effect on all students in the sense that each student can individualize and hopefully find for herself the factors that will enable her to reach her goals.

    But here, too, there are caveats.

    The Limits of SRL

    Let’s go back to that weight loss self-regulation problem. Suppose I’ve tried everything. Ear magnets. The cabbage soup diet. Exercise. Weightwatchers. Promises. Threats. Nothing works. In every case, either I can’t stick with the plan or I stick with it but don’t lose weight. Maybe, after many months of frustration, I talk to my doctor, who tells me that a side effect of my prescription medication is weight gain.

    Oh.

    Now I’m in uncharted territory. Can I take a different medicine? Is there some other way to counterbalance the effect of the medication? Or am I just stuck being 20 pounds overweight?

    There are limits to the power of self-regulation, including the power of self-regulated learning. If I’m a single parent working two jobs and driving for Uber in between to make ends meet, then no SRL strategy is going to give me time that I don’t have. If I have a learning disability, then I may need help finding SRL strategies that account for my particular situation.

    Here’s the tricky part that gets into the heart of the Kizilcec and Cohen paper: Barriers that impact the effectiveness of SRL strategies might be non-obvious and influenced by things like culture. In particular, the paper examines the differences of average effectiveness of a couple of different self-regulation strategies for people from individualist cultures versus collectivist cultures. This distinction will take a little unpacking in order to understand how it affects SRL.

    Suppose the week that I have a big assignment due, my uncle dies. I wasn’t particularly close to this person, but my extended family is generally close-knit. My relatives would be hurt if I didn’t come. I really value the closeness of my extended family, even if I didn’t really get to know my uncle well, and I care a lot about how they feel. It’s not really an option for me to skip the funeral. Which is halfway across the country. My sense of social obligation to my family makes it impossible for me to employ the self-regulation strategy of setting aside the time I need to complete my schoolwork.

    Imagine that your whole life were filled with these sorts of social obligations. Not just an occasional death in the family, but daily demands that you can’t predict and can’t ignore. In your world, there are many people who can ask you to change your plans at any time. Extended family members, neighbors, coworkers, or even strangers can place demands on you and, depending on the specifics, you really can’t say “no.” Because you live in a culture that places a high value on social bonds, and you have learned to place a high value on those bonds as well. The term of art for this kind of a culture is “collectivist,” and it contrasts with an “individualist” culture, where less emphasis is placed on social expectations and more on individual achievement.

    At the very least, you will find yourself in a similar problem to one of struggling to lose weight when you’re taking a medication that causes weight gain. Time management strategies don’t work because you’re really not in control of your time. But it may go even deeper than that. If your world is highly unpredictable because you can’t anticipate demands that you can’t reject and that come at you on a daily basis, then your fundamental idea of what it means to “manage time” may have to be different. You can’t just schedule certain nights of the week or reserve X hours to do your homework. You don’t have that power. Or, at least, it wouldn’t occur to you that exercising that power is a viable option, because you care deeply about what your willingness to fulfill your social obligations says about you as a person.

    This is exactly the hypothesis that Kizilcec and Cohen wanted to test. They wanted to see whether there are differences in how SRL works for students in collectivist versus individualist cultures.

    The stakes are high. Remember those two civilizational challenges: universal and lifelong education. To meet those challenges, we need to provide less traditional and formalized teacher support than the students in Bloom’s control groups got. We just don’t have enough teachers and classrooms to go around. This is a fundamentally different challenge than the one that Bloom was probing with his two sigma experiments. If it turns out that all kinds of factors, some as subtle as the the nature of the social ties in the culture you come from, can impact the effectiveness of our ability to teach students how to teach themselves, then how can we possibly meet our unprecedented civilizational goals? How can we tease out all the many possible factors, particularly when they interact with each other? How would we even begin to go about figuring that out in a rigorous, evidence-grounded way?

    Hold that thought. We’re going to return to it later in this post. First, though, we have to understand the experiment that Kizilcec and Cohen conducted to test whether this is even a problem. And to do that, we first need to understand one piece of social psychology.

    Understanding Cultural Dimensions Theory

    Kizilcec and Cohen wanted to figure out a way to test whether, on average, students from individualist cultures benefit more from being taught SRL techniques than students from collectivist cultures. The first thing they needed in order to do that is some way to define individualist versus collectivist cultures in a reasonably rigorous way.

    As you might expect, the researchers didn’t just pull the idea of individualist and collectivist cultures out of thin air. There is a body of research literature from which they were drawing. In particular, they drew on the research of a social psychologist named Geert Hofstede. During the late 1960s, Hofstede worked at IBM, where he founded and led the Personnel Research Department. This was at a time when globalization was really beginning to take hold and American-founded companies like IBM were learning how to run divisions in other countries with very different cultures. In the early days, these companies believed that they could train their new international employees on IBM-standard management practices and all would be well. But it soon became apparent that the challenge was more complicated than they thought. People in other countries responded to IBM’s management practices differently.

    At about this time, Hofstede stumbled upon a database of 117,000 attitude surveys from IBM employees all over the world. When analyzed for patterns on an individual level, the data were confusing. But when Hofstede grouped employees by nationality and looked for similarities and differences between national groups, some patterns began to emerge.

    As I have written about before, I worry that our generally low level of statistical literacy means that many of us are prone to misread statistical results and have little confidence in statistical analysis. So I am making a practice of providing some explanation of the statistical methods used when I write up these Research in Translation posts. In Hofstede’s case, he used a method called “factor analysis.”

    We can get a basic sense of the intuition that underlies that method through a common joke. You’ve probably seen lists with titles like “You might be a _________ if…”. The basic idea is that there are funny and non-obvious little traits and experiences that are shared by people of a certain type. When they are not mean, they are often inside jokes. For example, RallyPoint, a site that bills itself as “The Professional Military Network,” has an article entitled, “You might be a veteran if…” Item number five on the list is “You remember laughing at troops who thought 29% APR was good…” I don’t even know what that means, and I certainly wouldn’t think that an attitude about interest rates would be a marker of whether somebody is a veteran.

    Factor analysis starts with a collection of seemingly unrelated variables (like answers on an employee attitude survey) and looks for how closely correlated they are. If a group of variables is highly correlated, then it may be because they are all indications of a hidden or “latent” variable.

    “Oh, you answered ‘yes’ on eight out of these ten seemingly random questions. That suggests that you might be a veteran.”

    Hofstede applied factor analysis to national groups of employees and found evidence of four latent variables, which he calls “cultural dimensions.” (Subsequent research has reproduced Hofstede’s results, and the theory has been refined and expanded.) He called one of the latent variables “Individualism/Collectivism.” It purports to capture the degree to which each national culture has a strong web of the kinds of social obligations and expectations that I described in the previous section. Hofstede has used the cumulative research to create a country-by-country comparative index of his different dimensions, which you can play around with here.

    Kizilcec and Cohen used Hofstede’s country index of the Individualism/Collectivism dimension to provide some rigor to their question about cultural differences impacting the effectiveness SRL techniques.

    Understanding the Kizilcec and Cohen Paper

    The researchers conducted experiments on two MOOCs. In each case, students were given eight-minute tutorials on two SRL techniques: Mental Contrasting (MC) and Implementation Intentions (II). If you recall the weight loss strategy list from earlier in the posts, there were strategies that helped motivate me to do what I needed to do (like promising to buy myself a submarine drone if I meet my weight loss goal) and other strategies to actually accomplish my goal (like wearing ear magnets). MC and II fall into these two respective categories. MC is intended to help students self-motivate by (in the words of the authors)

    …vividly elaborating on positive outcomes associated with attaining a goal (e.g., learning a new skill) followed by vividly elaborating on central hindrances in the present that might interfere (e.g., a busy work schedule). By juxtaposing the desired future with current obstacles, MC can strengthen goal commitment and striving. Insofar as the obstacles to goal attainment are seen as surmountable, MC induces a sense that the desired future is within one’s reach, thereby increasing commitment and effortful goal striving.

    On the other side,

    [t]he II procedure helps people plan how to overcome obstacles and execute goal-directed actions. It encourages people to generate concrete if–then plans. Unlike unstructured planning, an II links a specific situation to a goal-directed action. An example of an II is, “If I feel too tired after work to watch the next lecture, then I will make myself coffee to stay awake.” Forming an II facilitates goal attainment because it increases the likelihood that people will respond efficiently and even automatically to regular obstacles that threaten the completion of their goals.

    Using their short MC and II tutorials, the researchers were able to increase MOOC completion by 32% over the control group in the first experiment and 15% in the second—for students from individualist countries. That’s a pretty remarkable result. If we’re trying to achieve these big civilizational goals of providing every human with higher education and lifelong learning advancement, then the possibility that we could increase completion rates in low-facilitation courses by up to 30% with an eight-minute lesson helping students get motivated and focused is pretty huge.

    That’s the good news. The bad news is that students from collectivist cultures showed no significant benefit. In fact, using India as a country on the collectivist end of the scale, the researchers found that

    relative to US respondents, Indian respondents reported that their social environment was more complex and that they shied away from forming if–then plans. Indian respondents listed more obstacles that could interfere with the goal of achieving a good grade in an online course than US respondents (India median = 4, US median = 3; Kruskal–Wallis X2 = 9.50, P = 0.002). They were also more likely to report that if– then plans oversimplify the complexity and ignore the uncertainty of real-life situations [t(192) = 3.12, P = 0.002, d = 0.45].

    In other words, Indian students were more likely to say that the II self-regulation strategy in general was unrealistic. It didn’t account for the complexity in their lives.

    There’s a lot more to this study than I’m going to cover here. For example, students from collectivist countries showed some benefits from MC when decoupled from II, which is interesting to think about. Generally speaking, I can’t unpack all the background in these Research in Translation posts and still have room to cover all the nuances of the papers themselves (although one of my goals is to give readers enough background that they can read and understand the papers themselves).

    But the headline here is provocative enough. When we think back to Bloom’s failure to find the one or two magic ingredients that tutors add which account for the second sigma of improvement, we can now see that the answer very likely is different from student to student. There is no silver bullet. In fact, there are all kinds of non-obvious factors that influence how well a given educational strategy works for a given student.

    So if we still want to address those two civilizational challenges—or even an intermediate challenge along the way, like closing achievement gaps—then Kizilcec and Cohen’s study raises a critical question:

    Now what?

    Implications

    Right about now, some of you are probably thinking, “So after making me read all of that, your grand conclusion is that students are individuals? Thanks for a whole lot of nothing, buddy.” Fair enough. But meeting these big educational challenges requires us to navigate between a rock and a hard place.

    On the one hand, we don’t want to fall for easy answers. As a culture, we tend to be a little schizophrenic about our attitudes toward education. Even smart people who believe in their hearts that every student is an individual human with different needs and goals can all too easily slip into over-generalizations and solutionism when the conversation turns from individual students to solving large-scale educational problems.

    On the other hand, if we’re committed to solving the big educational challenges, we can’t just shrug our shoulders and say, “It’s too hard. There’s no way to sort out all the factors.” We have to come up with a research approach that accounts for the fractal problem of student differences and how various combinations of those differences affect what works for different students in different learning contexts. The Kizilcec and Cohen paper is one model for what that kind of science could look like. But we need more. A lot more.

    In my view, we need what I call “empirical educators” and what Candace Thille calls “citizen scientists.” We need to crowd-source this problem by recruiting front-line classroom educators as field researchers that work in cooperation with researchers trained in learning sciences. For example, Kizilcec and Cohen’s experiments, however cleverly designed, can’t tell us how these results play out in courses that are not MOOCs. Or with students who are first-generation Americans whose families come from collectivist societies. Or whether other factors influence whether there are indentifiable factors that tell us which students from a collectivist country are most or least likely to be similar to their cultural norm in terms of SRL. Or what other SRL techniques might be more effective given any combination of these variables. The amount of research one can imagine being generated off the results of this one study alone is massive.

    We spend a lot of public and private money chasing silver bullets in education. I propose we would be better served by investing that money in providing educators with the training, support, and incentives to participate in the work of advancing the sciences of learning. At the very least, all professional educators should have a certain level of literacy on what we know about education, be able to read and understand the implications of a research paper, and believe that having this knowledge and these skills is a core part of their professional identity. Nobody should still be talking about learning styles, for example.

    Some educators may take this a step further and learn how to make the classroom experiments that they intuitively conduct on a regular basis a little more rigorous. And some may actively collaborate with professional researchers or even design their own studies using the disciplinary research tools that they already know from their graduate training.

    There won’t be one answer for every educator, any more than there will be one answer for every student. My country doctor primary care physician has a different relationship to medical science than an oncologist working at Memorial Sloan Kettering Cancer Center. But they both believe that having some relationship to medical science is essential to doing their jobs. In education, where the fractal nature of the problems we are trying to understand requires us to run many experiments in many contexts, having all educators see themselves in some sense as citizen scientists is even more critical.

    This post is part of our Research in Translation series, which is funded in part by the Bill & Melinda Gates Foundation. The findings and conclusions (or views) contained within are those of the authors and do not necessarily reflect positions or policies of the Bill & Melinda Gates Foundation.

  • “Alternative Pathways:” How to Rethink Vocational Education

    In Phil’s analysis of California Governor Jerry Brown’s directive for the California Community College System (CCCS) ((Disclosure: The Online Education Initiative from CCCS is a client of MindWires. The views in this and future posts represents my independent views and not OEI’s.)) to “take whatever steps necessary” to establish a fully online college, the punch line was as follows:

    What this points to is that for a new fully-online institution to get to some meaningful level of enrollment (let’s say 20,000) in the same ballpark as these comparison schools, I estimate it would take a full decade at the least….

    None of this analysis is to argue that CCCS should not try to establish a fully-online college. The goal of better serving nontraditional populations – adult students with and without jobs – is worth pursuing on its own merits.

    The numbers do argue, however, for a realistic view on the challenges the face:

    • Fighting against national demographic trends for adult students of community colleges;
    • Trying to avoid cannibalizing enrollment from existing California Community Colleges;
    • Having the patience to support the schools while it take years to grow to a size with meaningful enrollment levels; and
    • Accepting that best case this approach probably recovers less than 10% of the enrollment drop since 2009.

    Remember that the goal of this directive is to reach more non-traditional students. Community College Daily quotes CCCS Chancellor Eloy Oakley as saying

    We have literally tens of thousands of working adults with some college and no credentials and a couple of million working adults who are unemployed or underemployed,” Oakley said. “This is a wonderful opportunity to reach a population that really needs a community college to achieve economic mobility.

    So it’s worth looking at what other options may be available to address this goal. Fortuitously, Tyton Partners recently released a two-part report funded by The James Irvine Foundation called Path to Employment: Maximizing the Impact of Alternative Pathways Programs. [Registration required.] It provides a framework for analyzing the potential and critical success factors of shorter, non-degree and non-certification programs. There are various trendy Sillycon Valley buzzphrases associated with these sorts of programs, like “code academies,” “boot camps,” and “micro-credentials,” but they all fall under the broader heading of a term that has irrationally negative connotations in the United States: “vocational education.” For the purpose of this blog post, I am going to use vocational education and Tyton’s preferred term—Alternative Pathway Programs (APPs)—interchangeably.

    Tyton’s report provides an interesting general framework looking at how to think about these types of programs’ abilities to address the needs of non-traditional students, with a special focus on the state of California. It’s worth taking some time to examine aspects of the report in detail.

    Defining the Problem and the Goal

    First, let’s make sure we’re all talking about the same people, problems, and goals. Tyton’s report doesn’t talk about “non-traditional students” but rather “low-income adults,” which it defines as having the following attributes

    • May or may not be employed
    • Earn less than 200% of the federal poverty level
    • 18 years of age or older
    • Have limited or no exposure to postsecondary education

    I can’t say for certain whether that definition is one that Chancellor Oakley or Governor Brown would accept for the group of people they are trying to help, although I suspect that there is at least a strong overlap. For the remainder of this post, I will use the term “low-income adults” as defined by Tyton, since their analysis is built on that definition.

    There are approximately 7 million people who fit that definition in the state of California, “which accounts for nearly 37% of the state’s entire workforce,” according to Bureau of Labor Statistics numbers cited in the report.

    That’s a big number. How many of those people are touched by various educational programs?

    That grey space represents all the low-income adults in California who are receiving…nothing. But it’s actually worse than that in several ways, as Tyton points out in their report. First, not all of the 2.1 million California community college students fit the definition of “low-income adults.” Second, the average community college graduation rate is low. Nation-wide, it’s less than 30%. ((I chose to refer to the nation-wide graduation rate rather than California’s, largely because the nation-wide number is the one that the report uses.)) Particularly for low-income adults, going to school, incurring debt, and not getting a degree is worse than nothing.

    This brings up a second data fidelity problem, since one does not necessarily have to complete a degree in order to gain economic benefit from coursework. A worker could go back to school for a couple of accounting classes that enable her to get a better job, for example. One person’s degree non-completion is another person’s alternative pathway program. So the numbers Tyton uses as proxies for impact potentially both overestimate the number of low-income adults that California community colleges reach and underestimating the percentage of those it reaches that it economically impacts in a net positive way, although they are accurate enough for Tyton’s goal of painting a broad-brush picture of the magnitude of unmet need. More on the data issues later.

    So we have a severe problem of scaling access, even in a state that is historically known for its heavy investment in education. Even if a new online community campus were created and, under Phil’s most optimistic scenario, added another 20,000 enrollments over the next 10 years, that’s a drop in the bucket even before you consider that not all of those students would fit the “low-income adult” definition and it’s possible that considerably less than half of them would graduate. And this gap persists in spite of California spending “roughly 2.5% of the state’s entire annual budget” on educational programs for low-income students, according to Tyton’s figures.

    None of this is to cast aspersions on either the community college system or the idea of an online campus. Rather, the point is that the challenge is enormous. As Tyton puts it,

    [E]ven if all the spots within [community colleges and the two other identified types of] programs were allocated to low-income adults seeking to enhance their employment prospects, current capacity would support less than a third of the potential annual demand. Expanding the number of successful models that can support education-to-employment pathways for adults is imperative, both from within the current ecosystem of institutions and workforce programs and through new, innovative program models.

    The idea that Tyton’s report explores, which is not positioned as an alternative to existing programs but rather another tool in the toolbox, is what they call APP:

    An Alternative Pathways Program (APP) is defined as one that:

    • Focuses on education and training for specific job and career pathways

    • Maintains close alignment with employers and industries to facilitate job placement for participants completing the program

    • Does not offer a traditional postsecondary degree or certificate

    Most APPs focus on recruiting and serving participants directly, similar to colleges and universities, but they vary widely in their training model and program length, among other attributes. Some program models connect participants directly with employment opportunities. For example, the high-profile technology, design, and data “boot camps” o er a career-fair job connections model and last less than 12 weeks on average, while experiential learning programs such as apprenticeship programs prepare participants over a longer period (most are 6–12 months long) for a specific career path. Other programs do not connect participants with jobs, but they still offer training for specific career pathway and offer a certification or credential upon completion. Another set of programs focuses on general education as a baseline for specific career pathways, including programs or courses that help participants gain postsecondary credentials or learn general skills.

    Tyton defines the common goal that programs under this broad umbrella could solve as creating “a path for improving the economic and employment opportunities and outcomes for low-income adults.” So these are focused, vocational education programs aimed at helping adults stuck at the bottom rung of the income latter achieve enough education to begin climbing to higher rungs. Obviously, this is not the only set of goals one could have for post-secondary education. One could be concerned about developing more informed citizens, enriching people’s intellectual lives, giving them access to career paths that they didn’t know existed, cultivating a common national culture, among others. It is both reasonable and important to talk about different educational goals that we should aspire to. But these sorts of conversations tend to take on a one-goal-fits-all tone. Take a look again at the grey space in that last graphic:

    The people in that space are not currently being served by programs that address any of those goals. Tyton asks two basic questions. First, to what degree might APPs—many of which do not currently serve low-income adults—be recruited to fill in some of that grey space? And second, what are the characteristics of an APP that would make it most likely to achieve this goal? The first question can only be answered in a fairly general way, since we have no systematic tests of it yet. Tyton spends the bulk of their report on the second question by analyzing patterns across over 125 existing APPs.

    I am not going to provide a detailed critique of Tyton’s analysis framework in this post. Rather, I’ll be examining the ways in which having such a framework enables more productive and nuanced discussions addressing big educational problems like helping low-income adults using tools ranging from policy to educational technologies.

    The Six Pillars

    Much of Tyton’s analysis rests on what they call the “six pillars” of alternative pathways programs:

    • Enrollment policies: Processes and guidelines for admitting participants
    • Participant support: Resources and methods that support participants in overcoming life challenges
    • Labor market alignment: Level of program fit with the needs of employers and the local/regional economy
    • Connections: Extent to which program connects participants with employers and other job search resources
    • Training mix: Balance of curriculum emphasis on soft skills vs. academic and technical skills
    • Financial model: Ability to generate revenue and achieve organizational stability

    In and of themselves, there’s nothing earth-shaking about these categories. But stating them explicitly as part of the analytic framework enables us to do all kinds of additional important work.

    First, it enables us to ask, “Is this a complete and plausible list of critical success factors for a vocational program?” For example, the work of Vincent Tinto and others connects students’ sense of belonging in campus community to their likelihood of completing their degrees. Is that principle also operative in vocational programs for low-income adults? If so, how much of an effect does it have? And can it be subsumed under “participant support,” or is it distinct and important enough to merit its own pillar?

    This line of questioning brings up a second advantage of having such a framework, which is that it points to a research agenda. What sorts of participant support are most effective for helping low-income adults to complete APPs? Does the answer to that question vary by context? If so, which sorts of contextual factors matter the most? Tyton has proposed a set of preliminary hypotheses for the optimal way to address each pillar based on the research that they conducted:

    The point of the hypothesis is to have a truth proposition that can be tested. For example, Tyton has listed “employer as payer” as the “optimal model” for vocational programs. Going back to Governor Brown’s online campus proposition mentioned at the top of this post, California has funding mechanisms at its disposal that many of the start-up programs examined by Tyton do not. One might float an alternative hypothesis for that pillar and then test that hypothesis through various means.

    The framework also enables program designers to think about trade-offs more clearly. A classic trade-off is between enrollment policies and participant support. Tyton articulates several alternative models for each pillar. Here’s the selection of models they examine for support:

    But these models don’t exist independently of the other pillars. If your program has an enrollment process that is extremely good at identifying students who come to the program already possessing many of the skills and life circumstances that would enable them to succeed if only they were given the opportunity, then you may need less investment in participant support. Conversely, if the program’s goal is to help those students who have to face the most daunting obstacles to their education, then investment in support becomes more important.

    This example illustrates the point that we should interpret the word “optimal” loosely here. That large swath of grey in the earlier diagram representing all the low-income adults who are getting no education is not one homogenous blob of unmet need but rather a collection of millions of people with different strengths, needs, and circumstances. People in different situations will likely need different program designs in order to be successful. As a sector, we are thankfully past the peak of the idea that we can teach everything to everybody by posting video lectures online and calling them “MOOCs.” ((Not very far past, but still. Baby steps.))

    This brings us to one of the most deeply divisive terms in education: scale.

    The Tyton report examines two types of scale: inputs and outputs. Access and outcomes. Let’s imagine that California were somehow able to get 100% of low-income adults enrolled in community college degree programs. That would be good, right? Well, maybe. Let’s also suppose that, in doing so, California’s degree completion rate settles at the national average of 30% for community colleges. That would mean 70% of California’s low-income adults would end up still without a degree and still (likely) without improving their economic prospects. To the contrary, many of them would have increased levels of debt they would have to pay out of their unchanged salaries. So no, scaling access is not inherently good.

    For each of Tyton’s hypothesized optimal strategies in each of their hypothesized pillars, they analyze the trade-off the strategy makes between scale of access and scale of outcomes:

    If a chosen strategy for a pillar has a downward-pointing red arrow, then it presents a challenge to scaling access. On-site support to students is harder to provide to many students than providing students with no support. It’s harder to scale enrollments—access—for a program that commits to high-touch student support. Then why do it? Hopefully because it improves the percentage of degree completions and other positive outcomes.

    Taken together, the elements of Tyton’s framework give program designers and policy makers both a set of knobs they can turn in an effort to tune a particular program to the needs of a particular student population as well as a lens through which they can examine, test, and refine their assumptions in ways that will improve the effectiveness of those knobs and our knowledge of how the various settings interact with each other.

    Of course, having knobs to twiddle is good, but being able to measure the impact of your knob twiddling is critical. As I mentioned earlier, the numbers Tyton cites for community college impact on low-income adults are less than perfect measures. While adequate for the purpose of assessing the order of magnitude of unmet need in a large state like California, the two proxies the report uses for scale or access and outcomes—number of students served and graduation rate—are not great for measuring impact at a more granular level. And it will be tough to develop better ones. Going back to the example of the person who takes a couple of accounting classes at a community college and gets a better job as a result, how would one capture that? And yet, that is precisely what we would need to do to get clearer sense of the degree to which community colleges are currently serving the needs of low-income adults and the degree to which tweaking community college programs in various ways might increase that impact. Other APP efforts will likely face similar data quality challenges. So application of the model in practice will likely require some innovation around measurement in these areas.

    But here again, having a holistic model can help. Tyton highlights sustainability as the big reason to think about employer-pay business models. But if you happen to be thinking about the data problem while looking at the Tyton pillars graphic, it might occur to you that a direct relationship with employers can help with that problem. To start with, employers’ willingness to pay might be a good proxy for career progression benefits that students gain from their participation in the program. One would have to do the research, but it’s a plausible hypothesis. Second, the closer relationship with the employer makes outcomes data easier to get, perhaps in the form of anonymized aggregate data from the employers or by providing richer, longer term relationships with students that give them more incentive to provide the school with post-graduation follow-up data.

    What About Ed Tech?

    Nowhere does this framework explicitly address ed tech as such. And yet, ed tech decisions both large and small are often made by directly connecting a problem with a technological solution. A state wants to help reach more non-traditional students. Online learning can reach more students. So why not start an online learning program? A foundation or a college wants to help more first-generation college students make it through college. Adaptive learning programs seem to help some students in developmental math programs get past that stumbling block to their degree completion. So why not invest in adaptive learning programs?

    Absent of a richer analytic framework, these efforts are more likely to fail and less likely to be reproducible because they don’t start with either a holistic understanding of the needs of the students the efforts are trying to help or a clear understanding of how various aspects of the support ecosystem interact with each other. Without a theoretical framework, you can’t construct a clear hypothesis. Without a clear hypothesis, you can’t construct a proper experiment. Without a proper experiment, you can’t learn what works and what doesn’t.

    Very often, the best moment to think about ed tech is immediately after you have developed your program design and analyzed it for strengths and weaknesses. At this stage in the thinking, ed tech can potentially help by changing the laws of physics that underpin your model. OK, so low-income adult students need high-touch support which, when implemented in the traditional way, is resource intensive and therefore limits the number of students you can serve. Is there a way that technology can help provide that high-touch support at a lower resource cost? (By the way, the solution may not be to build robot advisors in the sky that can semi-read students’ minds. Instead, it might be saving advisors’ time spent doing other, non-student-facing work so that the same number of advisors have more time to serve students well.)

    I have no strong opinions about the answers that the Tyton paper arrives at, but I do believe that they are asking roughly the right questions in roughly the right order. As a field, we need more of this type of program- and policy-level research and analysis to inform a wide range of strategic decisions, including but not limited to use of ed tech. It is an exemplar of a genre of educational research that we should be looking to grow, propagate, and use to inform practical decision-making.

    This post is part of our Research in Translation series, which is funded in part by the Bill & Melinda Gates Foundation. The findings and conclusions (or views) contained within are those of the authors and do not necessarily reflect positions or policies of the Bill & Melinda Gates Foundation.