e-Literate

Present is Prologue

Tag: Tyton Partners

  • During #Covid19, Every Student is At-Risk

    One aspect of the current crisis that I’ve not seen a lot written about is retention. Oh, there’s plenty from a hand-wringing financial perspective. Will students take a gap year? If so, will they come back? Will they move to different institutions with better online learning offerings?

    What I haven’t seen a lot of yet is how to address the increased risk that traditionally successful students are more likely to drop out due to financial problems, the stress of working from home, other stresses in their lives, struggles to adapt to online education, and so on. If this list sounds familiar, it’s because students that we have thought of as “at risk” in the past—post-traditional students, including first-generation and older, working students—often face some or all of these challenges. But now, most or all students face them.

    Every student should be considered an at-risk student in the current environment.

    So we should be looking closely at the strategies that have been developed to address the needs of at-risk students and consider scaling them up to support every student. Particularly at higher end institutions whose populations have not been traditionally at-risk, it’s time to take a closer look at the supports you have in place for your non-traditional cohort and to learn from access-oriented colleges and universities that have been dealing with this problem at scale for a long time.

    I recently ran a Standard of Proof webinar with Tyton Partners, which recently released the latest iteration of their Driving Towards a Degree study of this strategy. There’s a lot that’s of interest in their work, including their methodology, which is quite clever. But today I want to highlight one of the major findings of the study, which is that colleges that collaborate across departmental lines on their retention initiatives achieve better retention. We all understand what the phrase “falling through the cracks” means, but Tyton has actually found a way to quantify how many more students fall through the cracks when college departments let those cracks form by not working together to create a seamless experience:

    Tyton’s findings on the impact of inter-departmental collaboration on retention

    [https://youtu.be/8At8Er_JXMY]

    Collaboration is particularly hard right now. Everybody is just trying to stay afloat. But that includes the students. Tyton was very careful not to generalize their pre-COVID study results to the current COVID situation—as they should be—but I would be money that next year’s results will be more dramatic than this year’s because the stresses on students are higher and there are more students under stress. Universities need to work harder than ever to knit their student support efforts into a coherent safety net for their students, and to anticipate many more students needing that support.

    Watch the whole webinar here.

  • Standard of Proof Webinars

    I have some announcements and updates to share regarding the Standard of Proof webinars. All good stuff, which I’m sure we could all use more of these days. As a general reminder, Standard of Proof webinars are always about contributions that vendors have made to our general knowledge or capabilities for supporting students—i.e., contributions that don’t just help their own customers—and they always include academics who were meaningful partners in the work. Companies featured in the Standard of Proof webinars are all sponsors of the Empirical Educator Project (EEP), but sponsorship does not guarantee a webinar. Only a qualifying contribution to the educational commons, as judged by me, earns them a webinar. This is not just promotional fluff. So I’m pleased to share the latest with you, which you’ll see represents an uptick in our tempo.

    First, the video archive of the webinar on the LXPathways learning architect and instructional technologist competency programs from iDesign is now live on YouTube. Or you can just watch it here, if you have the hour:

    iDesign LX Pathways Standard of Proof webinar

    Second, a reminder that we have a webinar tomorrow with Macmillan on their method for studying the effectiveness of their products early in the product development cycle. I serve as a (paid) member of the Macmillan Impact Research Advisory Council (IRAC), so I have had both a ring-side seat and some input on this process. The transparency they are attempting to provide regarding what they know about product impact, how they know it, and how confident they are in the results at any given moment is commendable. I’m looking forward to that conversation tomorrow, i.e., Wednesday, March 10th at 2 PM ET. Register for the Macmillan efficacy methodology webinar here.

    Third, you may have seen an announcement go out earlier today about our webinar for next Wednesday, March 18th at 2 PM ET for our Driving Toward a Degree webinar with Tyton Partners. We still have a lot to learn about increasing graduation rates, particularly among first-generation, post-traditional, and underrepresented students. Tyton has made an important contribution with their finding of the value of a coordinated response that far too few colleges and universities even attempt to achieve for this sort of effort. Their methodological approach to studying the impact of different approaches is also really interesting. Register for the Driving Toward a Degree webinar here.

    And finally, I have a hold-the-date pre-announcement for a follow-up webinar on the research AdmitHub has been doing. (You can see the video of our webinar with them on summer melt here.) This research has broader implications for helping students get to graduation. I’ll hold the details until we have the description nailed down with the participants—this is new research, so I always want to get the characterization right—but for now, save the date for a webinar on Thursday, April 9th at 2 PM.

  • “Alternative Pathways:” How to Rethink Vocational Education

    In Phil’s analysis of California Governor Jerry Brown’s directive for the California Community College System (CCCS) ((Disclosure: The Online Education Initiative from CCCS is a client of MindWires. The views in this and future posts represents my independent views and not OEI’s.)) to “take whatever steps necessary” to establish a fully online college, the punch line was as follows:

    What this points to is that for a new fully-online institution to get to some meaningful level of enrollment (let’s say 20,000) in the same ballpark as these comparison schools, I estimate it would take a full decade at the least….

    None of this analysis is to argue that CCCS should not try to establish a fully-online college. The goal of better serving nontraditional populations – adult students with and without jobs – is worth pursuing on its own merits.

    The numbers do argue, however, for a realistic view on the challenges the face:

    • Fighting against national demographic trends for adult students of community colleges;
    • Trying to avoid cannibalizing enrollment from existing California Community Colleges;
    • Having the patience to support the schools while it take years to grow to a size with meaningful enrollment levels; and
    • Accepting that best case this approach probably recovers less than 10% of the enrollment drop since 2009.

    Remember that the goal of this directive is to reach more non-traditional students. Community College Daily quotes CCCS Chancellor Eloy Oakley as saying

    We have literally tens of thousands of working adults with some college and no credentials and a couple of million working adults who are unemployed or underemployed,” Oakley said. “This is a wonderful opportunity to reach a population that really needs a community college to achieve economic mobility.

    So it’s worth looking at what other options may be available to address this goal. Fortuitously, Tyton Partners recently released a two-part report funded by The James Irvine Foundation called Path to Employment: Maximizing the Impact of Alternative Pathways Programs. [Registration required.] It provides a framework for analyzing the potential and critical success factors of shorter, non-degree and non-certification programs. There are various trendy Sillycon Valley buzzphrases associated with these sorts of programs, like “code academies,” “boot camps,” and “micro-credentials,” but they all fall under the broader heading of a term that has irrationally negative connotations in the United States: “vocational education.” For the purpose of this blog post, I am going to use vocational education and Tyton’s preferred term—Alternative Pathway Programs (APPs)—interchangeably.

    Tyton’s report provides an interesting general framework looking at how to think about these types of programs’ abilities to address the needs of non-traditional students, with a special focus on the state of California. It’s worth taking some time to examine aspects of the report in detail.

    Defining the Problem and the Goal

    First, let’s make sure we’re all talking about the same people, problems, and goals. Tyton’s report doesn’t talk about “non-traditional students” but rather “low-income adults,” which it defines as having the following attributes

    • May or may not be employed
    • Earn less than 200% of the federal poverty level
    • 18 years of age or older
    • Have limited or no exposure to postsecondary education

    I can’t say for certain whether that definition is one that Chancellor Oakley or Governor Brown would accept for the group of people they are trying to help, although I suspect that there is at least a strong overlap. For the remainder of this post, I will use the term “low-income adults” as defined by Tyton, since their analysis is built on that definition.

    There are approximately 7 million people who fit that definition in the state of California, “which accounts for nearly 37% of the state’s entire workforce,” according to Bureau of Labor Statistics numbers cited in the report.

    That’s a big number. How many of those people are touched by various educational programs?

    That grey space represents all the low-income adults in California who are receiving…nothing. But it’s actually worse than that in several ways, as Tyton points out in their report. First, not all of the 2.1 million California community college students fit the definition of “low-income adults.” Second, the average community college graduation rate is low. Nation-wide, it’s less than 30%. ((I chose to refer to the nation-wide graduation rate rather than California’s, largely because the nation-wide number is the one that the report uses.)) Particularly for low-income adults, going to school, incurring debt, and not getting a degree is worse than nothing.

    This brings up a second data fidelity problem, since one does not necessarily have to complete a degree in order to gain economic benefit from coursework. A worker could go back to school for a couple of accounting classes that enable her to get a better job, for example. One person’s degree non-completion is another person’s alternative pathway program. So the numbers Tyton uses as proxies for impact potentially both overestimate the number of low-income adults that California community colleges reach and underestimating the percentage of those it reaches that it economically impacts in a net positive way, although they are accurate enough for Tyton’s goal of painting a broad-brush picture of the magnitude of unmet need. More on the data issues later.

    So we have a severe problem of scaling access, even in a state that is historically known for its heavy investment in education. Even if a new online community campus were created and, under Phil’s most optimistic scenario, added another 20,000 enrollments over the next 10 years, that’s a drop in the bucket even before you consider that not all of those students would fit the “low-income adult” definition and it’s possible that considerably less than half of them would graduate. And this gap persists in spite of California spending “roughly 2.5% of the state’s entire annual budget” on educational programs for low-income students, according to Tyton’s figures.

    None of this is to cast aspersions on either the community college system or the idea of an online campus. Rather, the point is that the challenge is enormous. As Tyton puts it,

    [E]ven if all the spots within [community colleges and the two other identified types of] programs were allocated to low-income adults seeking to enhance their employment prospects, current capacity would support less than a third of the potential annual demand. Expanding the number of successful models that can support education-to-employment pathways for adults is imperative, both from within the current ecosystem of institutions and workforce programs and through new, innovative program models.

    The idea that Tyton’s report explores, which is not positioned as an alternative to existing programs but rather another tool in the toolbox, is what they call APP:

    An Alternative Pathways Program (APP) is defined as one that:

    • Focuses on education and training for specific job and career pathways

    • Maintains close alignment with employers and industries to facilitate job placement for participants completing the program

    • Does not offer a traditional postsecondary degree or certificate

    Most APPs focus on recruiting and serving participants directly, similar to colleges and universities, but they vary widely in their training model and program length, among other attributes. Some program models connect participants directly with employment opportunities. For example, the high-profile technology, design, and data “boot camps” o er a career-fair job connections model and last less than 12 weeks on average, while experiential learning programs such as apprenticeship programs prepare participants over a longer period (most are 6–12 months long) for a specific career path. Other programs do not connect participants with jobs, but they still offer training for specific career pathway and offer a certification or credential upon completion. Another set of programs focuses on general education as a baseline for specific career pathways, including programs or courses that help participants gain postsecondary credentials or learn general skills.

    Tyton defines the common goal that programs under this broad umbrella could solve as creating “a path for improving the economic and employment opportunities and outcomes for low-income adults.” So these are focused, vocational education programs aimed at helping adults stuck at the bottom rung of the income latter achieve enough education to begin climbing to higher rungs. Obviously, this is not the only set of goals one could have for post-secondary education. One could be concerned about developing more informed citizens, enriching people’s intellectual lives, giving them access to career paths that they didn’t know existed, cultivating a common national culture, among others. It is both reasonable and important to talk about different educational goals that we should aspire to. But these sorts of conversations tend to take on a one-goal-fits-all tone. Take a look again at the grey space in that last graphic:

    The people in that space are not currently being served by programs that address any of those goals. Tyton asks two basic questions. First, to what degree might APPs—many of which do not currently serve low-income adults—be recruited to fill in some of that grey space? And second, what are the characteristics of an APP that would make it most likely to achieve this goal? The first question can only be answered in a fairly general way, since we have no systematic tests of it yet. Tyton spends the bulk of their report on the second question by analyzing patterns across over 125 existing APPs.

    I am not going to provide a detailed critique of Tyton’s analysis framework in this post. Rather, I’ll be examining the ways in which having such a framework enables more productive and nuanced discussions addressing big educational problems like helping low-income adults using tools ranging from policy to educational technologies.

    The Six Pillars

    Much of Tyton’s analysis rests on what they call the “six pillars” of alternative pathways programs:

    • Enrollment policies: Processes and guidelines for admitting participants
    • Participant support: Resources and methods that support participants in overcoming life challenges
    • Labor market alignment: Level of program fit with the needs of employers and the local/regional economy
    • Connections: Extent to which program connects participants with employers and other job search resources
    • Training mix: Balance of curriculum emphasis on soft skills vs. academic and technical skills
    • Financial model: Ability to generate revenue and achieve organizational stability

    In and of themselves, there’s nothing earth-shaking about these categories. But stating them explicitly as part of the analytic framework enables us to do all kinds of additional important work.

    First, it enables us to ask, “Is this a complete and plausible list of critical success factors for a vocational program?” For example, the work of Vincent Tinto and others connects students’ sense of belonging in campus community to their likelihood of completing their degrees. Is that principle also operative in vocational programs for low-income adults? If so, how much of an effect does it have? And can it be subsumed under “participant support,” or is it distinct and important enough to merit its own pillar?

    This line of questioning brings up a second advantage of having such a framework, which is that it points to a research agenda. What sorts of participant support are most effective for helping low-income adults to complete APPs? Does the answer to that question vary by context? If so, which sorts of contextual factors matter the most? Tyton has proposed a set of preliminary hypotheses for the optimal way to address each pillar based on the research that they conducted:

    The point of the hypothesis is to have a truth proposition that can be tested. For example, Tyton has listed “employer as payer” as the “optimal model” for vocational programs. Going back to Governor Brown’s online campus proposition mentioned at the top of this post, California has funding mechanisms at its disposal that many of the start-up programs examined by Tyton do not. One might float an alternative hypothesis for that pillar and then test that hypothesis through various means.

    The framework also enables program designers to think about trade-offs more clearly. A classic trade-off is between enrollment policies and participant support. Tyton articulates several alternative models for each pillar. Here’s the selection of models they examine for support:

    But these models don’t exist independently of the other pillars. If your program has an enrollment process that is extremely good at identifying students who come to the program already possessing many of the skills and life circumstances that would enable them to succeed if only they were given the opportunity, then you may need less investment in participant support. Conversely, if the program’s goal is to help those students who have to face the most daunting obstacles to their education, then investment in support becomes more important.

    This example illustrates the point that we should interpret the word “optimal” loosely here. That large swath of grey in the earlier diagram representing all the low-income adults who are getting no education is not one homogenous blob of unmet need but rather a collection of millions of people with different strengths, needs, and circumstances. People in different situations will likely need different program designs in order to be successful. As a sector, we are thankfully past the peak of the idea that we can teach everything to everybody by posting video lectures online and calling them “MOOCs.” ((Not very far past, but still. Baby steps.))

    This brings us to one of the most deeply divisive terms in education: scale.

    The Tyton report examines two types of scale: inputs and outputs. Access and outcomes. Let’s imagine that California were somehow able to get 100% of low-income adults enrolled in community college degree programs. That would be good, right? Well, maybe. Let’s also suppose that, in doing so, California’s degree completion rate settles at the national average of 30% for community colleges. That would mean 70% of California’s low-income adults would end up still without a degree and still (likely) without improving their economic prospects. To the contrary, many of them would have increased levels of debt they would have to pay out of their unchanged salaries. So no, scaling access is not inherently good.

    For each of Tyton’s hypothesized optimal strategies in each of their hypothesized pillars, they analyze the trade-off the strategy makes between scale of access and scale of outcomes:

    If a chosen strategy for a pillar has a downward-pointing red arrow, then it presents a challenge to scaling access. On-site support to students is harder to provide to many students than providing students with no support. It’s harder to scale enrollments—access—for a program that commits to high-touch student support. Then why do it? Hopefully because it improves the percentage of degree completions and other positive outcomes.

    Taken together, the elements of Tyton’s framework give program designers and policy makers both a set of knobs they can turn in an effort to tune a particular program to the needs of a particular student population as well as a lens through which they can examine, test, and refine their assumptions in ways that will improve the effectiveness of those knobs and our knowledge of how the various settings interact with each other.

    Of course, having knobs to twiddle is good, but being able to measure the impact of your knob twiddling is critical. As I mentioned earlier, the numbers Tyton cites for community college impact on low-income adults are less than perfect measures. While adequate for the purpose of assessing the order of magnitude of unmet need in a large state like California, the two proxies the report uses for scale or access and outcomes—number of students served and graduation rate—are not great for measuring impact at a more granular level. And it will be tough to develop better ones. Going back to the example of the person who takes a couple of accounting classes at a community college and gets a better job as a result, how would one capture that? And yet, that is precisely what we would need to do to get clearer sense of the degree to which community colleges are currently serving the needs of low-income adults and the degree to which tweaking community college programs in various ways might increase that impact. Other APP efforts will likely face similar data quality challenges. So application of the model in practice will likely require some innovation around measurement in these areas.

    But here again, having a holistic model can help. Tyton highlights sustainability as the big reason to think about employer-pay business models. But if you happen to be thinking about the data problem while looking at the Tyton pillars graphic, it might occur to you that a direct relationship with employers can help with that problem. To start with, employers’ willingness to pay might be a good proxy for career progression benefits that students gain from their participation in the program. One would have to do the research, but it’s a plausible hypothesis. Second, the closer relationship with the employer makes outcomes data easier to get, perhaps in the form of anonymized aggregate data from the employers or by providing richer, longer term relationships with students that give them more incentive to provide the school with post-graduation follow-up data.

    What About Ed Tech?

    Nowhere does this framework explicitly address ed tech as such. And yet, ed tech decisions both large and small are often made by directly connecting a problem with a technological solution. A state wants to help reach more non-traditional students. Online learning can reach more students. So why not start an online learning program? A foundation or a college wants to help more first-generation college students make it through college. Adaptive learning programs seem to help some students in developmental math programs get past that stumbling block to their degree completion. So why not invest in adaptive learning programs?

    Absent of a richer analytic framework, these efforts are more likely to fail and less likely to be reproducible because they don’t start with either a holistic understanding of the needs of the students the efforts are trying to help or a clear understanding of how various aspects of the support ecosystem interact with each other. Without a theoretical framework, you can’t construct a clear hypothesis. Without a clear hypothesis, you can’t construct a proper experiment. Without a proper experiment, you can’t learn what works and what doesn’t.

    Very often, the best moment to think about ed tech is immediately after you have developed your program design and analyzed it for strengths and weaknesses. At this stage in the thinking, ed tech can potentially help by changing the laws of physics that underpin your model. OK, so low-income adult students need high-touch support which, when implemented in the traditional way, is resource intensive and therefore limits the number of students you can serve. Is there a way that technology can help provide that high-touch support at a lower resource cost? (By the way, the solution may not be to build robot advisors in the sky that can semi-read students’ minds. Instead, it might be saving advisors’ time spent doing other, non-student-facing work so that the same number of advisors have more time to serve students well.)

    I have no strong opinions about the answers that the Tyton paper arrives at, but I do believe that they are asking roughly the right questions in roughly the right order. As a field, we need more of this type of program- and policy-level research and analysis to inform a wide range of strategic decisions, including but not limited to use of ed tech. It is an exemplar of a genre of educational research that we should be looking to grow, propagate, and use to inform practical decision-making.

    This post is part of our Research in Translation series, which is funded in part by the Bill & Melinda Gates Foundation. The findings and conclusions (or views) contained within are those of the authors and do not necessarily reflect positions or policies of the Bill & Melinda Gates Foundation.