e-Literate

Present is Prologue

Category: Policy

  • Welcome Change: OpenStax using more accurate data on student textbook expenditures

    Welcome Change: OpenStax using more accurate data on student textbook expenditures

    Last week OpenStax, the Rice University-based publisher of open educational resource (OER) materials, announced that according to their data more than 2.2 million students at 48% of colleges in the US and 1,150 outside the US are using OpenStax free textbooks, saving an estimated $177 million.

    This is compelling data in its own right, and we are working on analysis around this organization and its model, but somewhat buried in the press release is another significant statement around what students currently spend on textbooks and what savings are possible with OER.

    “Our community is creating a movement that will make a big impact on college affordability. The success of open textbooks like OpenStax have ignited competition in the textbook market, and textbook prices are actually falling for the first time in 50 years.”

    As a result of the unprecedented downward shift in textbook prices, OpenStax will be decreasing its estimated student savings figure from $98.57 to $79.37 based on federal data. The U.S. Department of Education’s National Center for Education Statistics published a study in May stating the average undergraduate student spent $555.60 on required course materials for the academic year. Dividing that number by seven courses (the undergraduate average, according to enrollment data) comes out to $79.37 in savings for each student using an OpenStax book.

    I have long argued that OER groups and others arguing for making college more affordable should use baseline numbers based on what students actually pay for textbooks, rather than the all-too-common $1,220 – $1,420 per year numbers from a misuse of College Board budget numbers (see chart at top of page 10 in this document). With OpenStax moving to new federal data showing $556 average expenditures, we should start to see more reliable estimates of student savings. Kudos to them.

    However, this level of student spending should not be a surprise to anyone following the curricular materials market.

    Our 2015 post “How Much Do College Students Actually Pay For Textbooks?”, as well as a follow-up post, show in detail that we have had data for years showing that students roughly $600 per year on textbooks and related course materials, and that that number has been falling since at least 2008. Using data from the National Association of College Stores (NACS), we knew three years ago about the rough level of spending and the multi-year decline. NACS has continued to release annual updates, with the most recent public release from last summer:

    NACS data showing course material expenditure

    What OpenStax refers to, however, is the new National Postsecondary Student Aid Study (NPSAS) restricted-use data from the US Department of Education’s National Center on Education Statistics, showing $555.60 average student expenditures per year. Which is right in line with the NACS data.

    We plan to explore the NPSAS data in more detail, as it provides rich data for crosstabs and exploration of student expenses. But for now, kudos to OpenStax for this change in student savings estimates, even if it is years overdue. I would hope that other OER advocates would follow their lead.

  • Hawai’i Senate OER Bill Update: Amended language saves the day

    Hawai’i Senate OER Bill Update: Amended language saves the day

    On Friday I reported about SB2328, a bill that passed (with amendments) the Hawai’i Senate Committee on Higher Education and would have mandated open educational resources (OER) for all courses at all 10 University of Hawai’i campuses. And if there were no adequate OER materials for a course? “. . . the faculty member or lecturer responsible with providing instruction for the course shall create the instructional materials and offer those materials free of charge to students through open educational resources.”

    This bill was a disaster in the making. Not only would it have been unworkable in terms of funding and intellectual property ownership, it would also have set back the OER movement by associating OER with unfunded faculty mandates and reduction of academic freedom. All this from good intentions but apparently shallow understanding.

    As mentioned in an update to Friday’s posts, the amendments that resulted from committee hearings removed the mandates. We now have the amended language, and it is a completely different bill.

    Updated language:

    The purpose of this Act is to:

    (1) Establish the University of Hawai‘i open educational resources task force to conduct a comprehensive analysis and evaluation on all general education courses and high attendance courses taught at the University of Hawai‘i system to identify open educational resources for those courses;

    (2) Establish and appropriate funds for an open educational resources pilot project grant program to incentivize faculty that adopt, develop, and implement open educational resources; and

    (3) Require the University of Hawai‘i open educational resources task force to report its findings and recommendation initiatives for supporting and expanding the use of open educational resources at the University of Hawai‘i to the Legislature prior to the Regular Session of 2019.

    During the hearings there were several dozen testimonies shared, and all but two opposed the bill (and one of those two changed positions to oppose). Leading the opposition was the University of Hawai’i Professional Assembly, the local faculty union.

    In short, the bill amendments removed mandates, creates a task force charged with a one-year evaluation of high-enrollment and general ed courses, and creates a $50,000 grant fund to incentivize faculty adoption.

    It is not clear whether the bill will make it through remaining hurdles to become state law, but if it does, we will have a fairly significant move in the state dealing with the costs of curricular materials and OER adoption.

    Billy Meinke, OER Technologist and UH Manoa and a key player within the system (let’s call him Kane OER), was unaware of SB2328 before it came out. This gets to the heart of the problem – the original bill appears to have been written without any input from the people already working on OER adoption within the University of Hawai’i.

    I still have a problem with the preamble of the bill that uses the misleading claim that the “average cost for books and supplies for the same academic year at public colleges averaged $1,250.” Students actually pay about half this amount, and this false setup will lead to erroneous estimates of how much any such bill could save for its students. But for now, crisis averted.

  • Hawai’i Senate Bill: Would mandate OER material for all U Hawai’i system courses

    Hawai’i Senate Bill: Would mandate OER material for all U Hawai’i system courses

    Thanks for update from Brent Auernheimer, I found out that the Hawai’i Senate Committee on Higher Education recently debated a bill regarding Open Educational Resources (OER) usage at the University of Hawai’i system of 10 campuses. Introduced on January 19th, SB2328 states:

    Beginning with the 2020-2021 school year, all courses at all campuses within the University of Hawai‘i system that require the use of instructional materials, including textbooks, shall use instructional materials from the open educational resources at the University of Hawai‘i; provided that the use of instructional materials, including textbooks, that requires a student to purchase or pay a subscription for the materials shall be prohibited; provided further that if open educational resources does not have relevant instructional materials available for a course, the faculty member or lecturer responsible with providing instruction for the course shall create the instructional materials and offer those materials free of charge to students through open educational resources.

    Read that carefully – OER for all courses, no commercial services around OER allowed, and if appropriate OER does not exist, the faculty member must create the material themselves, all mandated from the state legislature.

    When I first saw this news, I assumed it was a either a misguided effort that would quietly be killed in committee or a political statement. Predictably, and appropriately, the University of Hawai’i Professional Assembly actively opposed this bill, calling it “legislative overreach” and “infringement on academic judgement”, while also calling out the costs and support needed for faculty to create such materials.

    On January 30 hearings, the vast majority of testimony – much of it from faculty members – opposed the measure with only two statements supporting. Yet on February 6, the Senate Committee unanimously passed the bill on to the full Senate.

    The committee(s) on HRE recommend(s) that the measure be PASSED, WITH AMENDMENTS. The votes in HRE were as follows: 5 Aye(s): Senator(s) K. Kahele, Kim, S. Chang, Keith-Agaran, Kidani; Aye(s) with reservations: none ; 0 No(es): none; and 0 Excused: none.

    I have not been able to determine what the amendments are for the bill (or if that refers to future amendments coming from floor debate), and I also do not know how likely it is to pass the full senate or to become state law. I’ll keep looking for more information.

    Unless I’m missing something, this could be a jump-the-shark moment for portions of the OER movement. Comments appreciated.

    Update: From Twitter stream (sounds like some good changes):

    https://twitter.com/billymeinke/status/962127447171907584

  • WGU Is Not Off the Hook

    WGU Is Not Off the Hook

    In Phil’s first piece on the Department of Education’s Office of the Inspector General (OIG) finding the Western Governors University (WGU) should be considered a correspondence provider rather than a distance education provider, he wrote,

    This audit is a travesty in my opinion. Even though it is likely to be rejected by the ED itself, it will have an impact, and the internal review of the audit will likely take years.

    The problem, in a nutshell, is that the OIG decided that WGU’s unbundled instructor’s role, with multiple staff roles supporting students in a (largely) self-paced environment, does not count as “regular and substantive interaction between students and teachers,” which is a requirement for classification as a distance learning provider.

    Phil believes that this assessment by the OIG was arbitrary and, based on my admittedly limited understanding of their assessment process, I tend to agree. But that doesn’t mean that the OIG is wrong. It means we don’t know whether the OIG is wrong. And the heart of the problem—the definition and test for “regular and substantive interaction between students and instructors”—is a real challenge. While feel fairly confident that the OIG applied too narrow an interpretation of a standard that problably needs to be revised anyway, coming up with a better evidence-based standard is tough. And if we don’t have one, we can’t tell if WGU’s programs should be considered equivalent to more traditional distance learning programs.

    There are two positions that one could take in arguing against the OIG finding: (1) that it is possible to deliver the equal of a traditional education without regular and substantive interaction between students and teachers, or (2) that this interaction is necessary but we need a different, perhaps more flexible definition of it.

    Let’s look at each of these in turn.

    Position 1: The Standard is Unnecessary

    The more radical of the two positions is that “regular and substantive interactions between students and teachers” is outdated in the sense that such interaction is not necessary for a quality distance education program. In this view, good design and good technology provide enough support for self-paced students. People who take this position tend to have a high opinion of the impact of technology, a low opinion of the impact of the average instructor, or both.

    I’m not aware of any research that definitively settles this particular debate and would be surprised if there were any. In fact, I’m not sure it’s possible to produce such evidence in principle, because there are too many contextual factors to come up with just one answer. Some students in some programs studying some subjects to some level of achievement may do as well (or better) in a self-paced, largely self-guided competency-based program as they would in a traditional instructor-led setting. There would need to be an enormous amount of research, including some foundational research that we don’t have yet, to sort out all of the many “ifs” that determine the circumstances under which such a program would be equivalent in effectiveness.

    I think it’s dangerous to assume that “regular and substantive interaction between students and instructors” is an obsolete standard, and I do think there are at least three strands of research with results that should give us pause about being too aggressive about taking human teachers out of the equation.

    First, there’s Benjamin Bloom’s research on the Two Sigma Problem. Since I recently wrote about this in some detail as part of a longer post, I’ll give you the short version. Bloom found that by using tutors in a mastery learning context, he could achieve two standard deviations of improvement over standard instruction. One could argue that WGU’s model of self-paced learning with periodic assessments and support from course mentors attempts to imitate Bloom’s approach (although one would have to look closely to see whether the degree to which they are actually doing so). The relevant detail for our current purpose is that Bloom could never isolate exactly what it was about the tutors that delivered that second sigma. Without understanding the reasons why having a human tutor involved improves student outcomes by as much as a full course grade, it seems imprudent to assume it can be removed or replaced.

    Second, there’s the research conducted by Gallup and Purdue University showing that college graduates were 1.7 times more likely to thrive in all five of Gallup’s measures of wellbeing—physical, financial, community, career, and social—if they agreed with the statement “My professors at [college] cared about me as a person.” They were 1.5 more likely to thrive on those measures if they answered agreed with the statement “I had at least one professor at [College] who made me excited about learning.” Those are pretty compelling results, and it’s hard to see how one would replicate them without some form of regular and substantial interaction between students and instructors. For more on this study, see my post on it.

    In a follow-up piece to that post I just referenced, I talked about the third strand of research from Vincent Tinto. He showed that students are more likely to persist at college if they feel a sense of belonging. “[S]Students have to come to see themselves as a member of a community of other students, faculty and staff who value their membership.” Yet again, there is evidence of impact for a human factor that argues in favor of regular and substantial interaction between students and teachers.

    To be clear, I’m not suggesting that one could not create an educational system that provides real value without such interaction. But it would probably be a different kind of education that provides different kinds and levels of value. The OIG is concerned with classification and equivalence: Is WGU providing educational value that’s similar enough to more traditional distance learning programs that it can be classified as the same type of degree? I don’t think we can let the university off the hook by dismissing the “regular and substantial interaction” requirement as obsolete.

    Position 2: The Standard Needs Revision

    The more conservative argument against the OIG’s evaluation of WGU’s courses is that we still need a standard for “regular and substantial interaction between students and teachers,” but that our interpretation of that standard should be more flexible than the one that the OIG applied. Ideally, there would be some sort of evidence-based test. Let’s see if we can imagine what such a test might look like, based on the three research strands I mentioned above.

    It would be hard (and probably pointless) to try to replicate Bloom’s highly controlled laboratory experiments which took place in a very different schooling context. But we might get something from the spirit of the experiment. Simply put, can we come up with some sort of rough measure of the impact of the instructors (or the various folks who individually or collectively fulfill the instructor’s function) on mastery of materials? Can we find evidence of impact? One place might be to look at variance in student performance between instructors teaching the same material. If there is substantial variance that can be reliably attributed to instructors, then WGU could argue that their courses have enough student/instructor interaction to make a difference.

    The Tinto and Gallup/Purdue research would be relatively straightforward to draw upon, since they both use student attitude surveys. But only relatively, because I haven’t seen studies applying any of these instruments specifically to distance learning programs. (If anybody knows of such research, please let me know.) One would need to establish a baseline. But that seems like a good idea anyway.

    So there are probably a number of ways that the OIG could establish an empirical test to find evidence of student/instructor interaction that is regular and substantive enough to pass an equivalence threshold. It would probably be crude, but a crude test is better than no test at all, which appears to be what we have now.

    I don’t know if the OIG assessment of WGU’s courses was wrong. I feel fairly confident that it was made arbitrarily. But the fact that we have no reason to believe that it is right is not the same as saying we have reason to believe that it is wrong. The reason that bears repeating is that, defined this way, the problem exists not only for WGU but for every assessment that the OIG makes. If the standard is completely subjective and therefore inherently arbitrary in its application, then it is meaningless.

  • WGU Audit: Likely impacts for fragile movement of competency-based education

    WGU Audit: Likely impacts for fragile movement of competency-based education

    One issue that almost all observers seem to agree upon is that the Department of Education is unlikely to accept the Office of the Inspector General’s (OIG’s) recommendations to declare Western Governors University (WGU) a provider of correspondence courses and to force the school to pay back more than $700 million in Title IV funds. It would be a mistake, however, to dismiss the audit findings that attempted to add new interpretations of distance education requirements for “regular and substantive interaction”, a topic I mentioned on Friday.

    One important variable in this equation is timing – how long will the ED take to review the audit findings and make official decisions on adopting or rejecting the recommendations? Based on history, the decision could take years, as described by Michael Goldstein, lawyer at Cooley LLP in a WCET post by Russ Poulin:

    The IG’s report and recommendations go to Federal Student Aid, which decides what, if any, action should be taken. (The “if any” is directly from the IG transmittal.) That involves a further, and often lengthy, review process. The ultimate decision authority is the Secretary.

    I’m not counting on institutional leaders using the argument “I’m not worried, Betsy DeVos has my back”. The longer the review process drags on, the bigger the impact.

    Another important variable is the extent to which the ED will reject the findings – will they reject the audit in its entirety and take no action, or will they accept some of the findings? Keep in mind that the argument about WGU providing self-paced courses and rejecting the institution’s claims to be a term-based institution are much stronger than the argument behind the OIG’s arbitrary interpretation of regular and substantive interaction.

    Impact on Big CBE Programs

    The impact on WGU itself is most likely a matter of whether perception drives enrollment down. WGU clearly spent a lot of time and effort this summer preparing their web site to deal with the fall out. The central theme is to attempt to reassure current and prospective students that WGU “students, graduates, and employers of our graduates can rest assured that WGU’s accreditation and financial aid eligibility are intact”.

    Students don’t follow all the details of political maneuvering and and even less of OIG audit reviews, but the perception that WGU’s status as a qualified distance education provider is at risk, therefore making the value of the degrees and likelihood of financial aid at risk, will add a barrier to enrollment decisions. I don’t suspect WGU can sweep this under the rug – they will have to go public as they have done with web site and hit this topic head on.

    In my estimation, however, this will be a matter of a reduction in WGU’s enrollment growth, or possibly a drop, and not an existential problem for them. WGU has a 20-year history, more than 83,000 students, full institutional commitment to fight this audit in public, and a lot of clout in Washington with bipartisan support of their model – a rarity these days.

    Southern New Hampshire University (SNHU) is also known for CBE, as their College for America got a lot of press over the past several years. What is not as well-known is that College for America, with its 8,000+ students, has been folded into the main operations of SNHU and its 100,000+ students, using the description of Workforce Partnerships. SNHU has less exposure than WGU to the audit for three reasons:

    • Obviously the audit was specifically on WGU and not SNHU.
    • CBE is all that WGU does, whereas SNHU has a fast-growing traditional online set of programs that provides the majority of its enrollment.
    • SNHU’s College for America is a business-to-business model, working directly with employers rather than being a consumer program targeting individual students as WGU does. It will be a lot easier to control the message and reassure partners in the B2B model.

    There could be some impact to SNHU and College for America, but again this is likely a manageable problem for these large schools.

    Impact on Everyone Else

    The bigger impact in my estimation will be on the other CBE programs in operation or in consideration, and this gets to the fragile movement comment.

    Both WGU and SNHU have gone all-in on CBE. They invested heavily in developing the models, they have worked with employers to understand needs, they already have thousands and thousands of students and are growing enrollment, and they are fully committed as an institution to the concept and implementation of full-fledged CBE and not just CBE-lite. But both schools are outliers in the broader CBE movement.

    A great deal of the perception of CBE is that there are “hundreds of schools” developing programs, often driven by a 2015 Public Agenda survey. What is happening in reality is that hundreds of schools are very cautiously dipping their toes in the CBE waters with no real commitment to make the model work, and they are only playing with courses and individual programs. Carl Straumsheim from Inside Higher Ed covered this situation in the spring, triggered by e-Literate news that Ellucian was dropping its CBE platform due to weak market demand:

    Last year, Ellucian partnered with the consulting and research firm Eduventures and the American Council on Education to survey 251 colleges on their competency-based education strategies. The survey identified one major reason why the competency-based education market may be a tricky one for vendors to build a profitable business model in: most colleges aren’t ready to go all in yet.

    The study found that only 7 percent of the colleges surveyed said they delivered most of their education using a competency-based model. Many more colleges said they were at the point of testing competency-based education in individual programs (18 percent) or courses (37 percent).

    For most of the CBE programs that I have seen, we’re dealing with dozens, or perhaps hundreds of students. The University of Wisconsin’s UW Flex program – one of the best known outside of WGU and SNHU – has only grown to 5 degree programs, 3 certificate programs, 1300 students in total, and $2.8 million in annual gross revenue. And the vast majority of programs are much smaller than that. Read the Eduventures 2016 report (the same one referenced by Ellucian in the IHE article) for additional perspective.

    Rather than a single, dominant version of CBE, our 2016 survey data reveals a diversity of practice across a spectrum of schools, each deploying and experimenting with CBE in order to meet specific institutional challenges. A portrait of CBE emerges as a menu of tools and practices, rather than a monolithic approach or linear path. These findings underscore the need for institutions to carefully weigh the pros and cons of CBE implementation, and to proactively select the CBE components that make the most sense for their students and mission.

    The challenge is that CBE calls for a new organizational model and a new pedagogical design in order to make it work, at least in a self-sustaining manner. It is great to see hundreds of schools experimenting with new methods to reach non-traditional students, but none of these programs will last if they do not get sufficient scale to justify the costs. But very few schools are even looking at how to get to thousands of students and determining what investment and organizational setup will be required to get there. This isn’t as simple as flipping a classroom or two.

    When you add in the WGU audit results, this challenge gets much harder to address. It will be much more difficult to justify investing in CBE programs or partnerships, or expanding beyond a pilot, when the audit provides marketing copy for those would resist CBE at traditional schools.

    California

    From the annals of bad timing comes this news from a week ago:

    The California Community Colleges’ Board of Governors today approved a new partnership with Western Governors University (WGU) that allows graduates of California’s 114 community colleges to transfer and seek their bachelor’s degree at a discount from the fully accredited, online institution.

    The agreement goes beyond pre-setting up transfers. It also is meant to encourage CCC students to move to WGU for their bachelor’s degree, where appropriate. There are terms for a 5% discount for CCC students, and there is a section on joint marketing.

    The CCCCO supports and will encourage CCCs to collaborate with WGU so that partnership
    information is available to students, faculty, and staff. WGU will also work in collaboration
    with the CCCCO to assist in dissemination of information to students, faculty, and staff at
    CCCs. CCCs will be encouraged to publicize locally the Chancellor’s Office support of this
    agreement, to inform students, faculty, and staff of the agreement’s benefits. Information will
    be encouraged through established internal CCC communication channels (such as
    student/employee newsletters, web sites pages, and listservs)

    This news comes at the same time as the active debate on the CCC system being directed to create a fully-online college serving non-traditional working adult students.

    These are dramatic proposals for the largest higher education system in the US, and there is bound to be plenty of opposition to such plans from faculty unions and other groups skeptical about changes in model.

    I suspect that the political climate in California just got a lot more difficult in terms of implementing both initiatives, and I suspect that the option lists available for setting up the new online college is shorter than it was last week.

    Reminder

    To be clear for those who have not read the first post, I consider the audit a travesty. The CBE movement and individual programs deserve scrutiny to ensure quality education for students, but this audit and its impact will do nothing useful to protect students or ensure quality.

    ((Full disclosure and Update: WGU is a past client of MindWires. Although we have no recent or pending relationship with WGU, since the topic was raised at Hack Education, I have added this disclosure. SNHU is a customer of our LMS market analysis service.))

  • WGU Audit Findings: Interpretations of “regular and substantive” and “self-paced”

    WGU Audit Findings: Interpretations of “regular and substantive” and “self-paced”

    The big news this week was the Office of the Inspector General (OIG) at the Department of Education (ED) finding that Western Governors University (WGU) should be considered a correspondence provider instead of a distance education provider, and the school should return more than $700 million in Title IV federal funding programs. ((Full disclosure and Update: WGU is a past client of MindWires. Although we have no recent or pending relationship with WGU, since the topic was raised at Hack Education, I have added this disclosure.)) In short, being ruled a correspondence provider would mean that most student loans and Pell grants would not apply for WGU students and would all but shut down the institution or make it irrelevant. While the ED itself is unlikely to follow these non-binding recommendations, this ruling will have a big impact for years. More on that in tomorrow’s post. For now, the title loans for college students are the best option right now.

    It would be useful to first review the actual audit findings, especially since most media reporting focused mostly or solely on the issue of “regular and substantive interaction”, but the findings are broader and also encompass issues are self-paced vs. credit-hour / term-based education.

    The audit started over four years ago and primarily focused on 102 courses (out of 980) offered in the 2013 – 14 academic year. No one seems to know why the OIG started this audit, but the audit report itself makes it clear that quality was not the issue (page 6):

    we did not assess whether the school’s model was improving educational quality or expanding access to higher education.

    WGU’s regional accreditor,  Northwest Commission on Colleges and Universities, has accredited WGU as a term-based distance education provider, including reaffirming the accreditation in February of this year. The Department of Education explicitly allowed WGU to be classified as a distance education provider as part of the Distance Education Demonstration Program from 1999 – 2005 and granted further waivers and agreements in April 2005.

    By my reading, the audit is a model of hyper-literal translation of ambiguous regulations, leading to three findings.

    • Finding 1) Course Offerings Met the Title IV Definition of a Correspondence Course, Not the Title IV Definition of Distance Education – This finding was centered on reviews of course materials for 69 courses as well as a mapping of WGU’s unbundled faculty role to traditional instructor definitions. The OIG found that using the courses did not meet the interpretation of regular and substantive interaction required of distance education courses.
    • Finding 2) Western Governors University Disbursed Title IV Funds to Students Before the Students Were Eligible to Receive the Funds – This finding was based on additional mapping of WGU’s CBE model to traditional term-based model. WGU itself decided to consider itself a term-based institution, mapping one competency unit to one credit hour over a 26-week academic term, and the ED recognized this classification in the 2005 agreement. The OIG, however, found that WGU should have been classified as a non-term school using self-paced programs. There are a different set of regulations for non-term programs.
    • Finding 3) Western Governors University Did Not Always Comply With the Requirements Governing the Return of Title IV Funds – This finding is important, but it deals with detailed bureaucratic rules upon student withdrawals. I’ll let someone else look at this finding.

    It is the combination of findings 1) and 2) that are important not just to WGU but to any school developing an online or hybrid non-lecture-based approach.

    Regular and Substantive Interaction

    WGU has been at the forefront of breaking apart the traditional faculty role, instead using mentors, evaluators, and other interdependent roles. The audit acknowledged that (page 15):

    Northwest Commission recognized Western Governors University’s student mentors, course mentors, evaluators, product managers, and council members as members of the school’s faculty. The accrediting agency also distinguished between the roles of student mentors and course mentors, characterizing student mentors as serving in academic advisory roles and course mentors serving in instructional roles.

    The OIG used a binary role-based approach (you are an instructor or you are not) leading to conclusion that only course mentors and evaluators could be considered as instructors, however. The basis of this determination was an instructor must “provide instruction on course content” – clearly a content-dissemination view that rejects alternative pedagogies. And this interpretation that the OIG treats as unambiguous is not based on law, regulations, or commonly-accepted educational terminology.

    The OIG looked at the ambiguous regulations and chose their own, very literal, interpretations (page 14):

    Because the HEA and Title IV regulations did not define instructor, substantive, or regular, we considered the ordinary meaning of those terms when assessing whether the school designed the 102 courses to offer regular and substantive interaction between students and instructors. We reviewed the school’s course design materials for evidence of interaction that was not primarily initiated by the student and was (1) with someone who instructs or provides knowledge about the subject matter of the course (instructor), (2) relevant to the subject matter (substantive), and (3) occurring with some reasonable frequency considering the school-suggested length of the course (regular).

    This is why I call the audit methodology as hyper-literal. Somehow the OIG thinks they can determine – without any disagreement or ambiguity – the “ordinary meaning of those terms” based on their own interpretations.

    Also note that the determination was entirely based on course design materials – think syllabus and course outlines. The OIG did not look at interactions arising during the course of actual course work, just whether there were pre-defined webinars, meetings, and student-instructor interactions. The OIG did eliminate many interactions as not being “substantive” (page 16):

    After identifying the employees who could reasonably be considered instructors, we determined what type of interactions could reasonably be considered substantive. We considered an interaction to be substantive if the course design materials described student interaction with a course mentor or required an individual submission of a performance task for which an evaluator provided the student feedback. We did not consider the following to be instances of substantive interactions between students and instructors:

    • Objective assessments that students submitted for evaluation because feedback on objective assessments was computer-generated, was not provided by instructors, and did not facilitate synchronous or asynchronous interaction between students and instructors.
    • Recorded webinars, videos, and reading materials if the course design materials did not require the students to watch the webinars or videos and then interact with an instructor. Many course outlines stated only that course mentors were available to students for assistance if the student wanted to contact the course mentor. Had the course design materials indicated that the recorded webinars, videos, and reading materials facilitated synchronous or asynchronous interactions, such as requiring the student to contact an instructor or participate in an online discussion moderated by an instructor, we would have considered those instances to be substantive interaction.
    • Contact with student mentors because the accrediting agency’s recognition, the school’s description of the student mentor’s role, and our interviews with six student mentors disclosed that student mentors did not provide instruction on the subject matter of the courses that students were taking.

    To be “regular”, the OIG required that all interactions be pre-planned, with the right people, in the course design materials (page 16).

    We did not find any evidence in the course design materials for 69 courses that would provide a reasonable basis for concluding that planned student interactions with course mentors and evaluators could be considered as occurring with some reasonable frequency (regular). The only evidence of regular interaction was student contact with student mentors. However, student mentors did not provide instruction.

    In its comment on the draft findings, WGU complained that OIG (page 20 and 26):

    did not count, as regular and substantive interaction, significant interactions not described in course outlines.

    OIG’s response basically agreed with this complaint:

    We considered all events described in course outlines and pacing guides, along with calendars of live events referenced in those materials. If substantive interactions were not described in any of these course design materials, we had no reasonable basis to conclude that such interactions were part of the design of the courses and did not consider them as planned course requirements. [snip]

    Course mentors might have identified students who were struggling, and many course outlines instructed students to contact course mentors if the students needed assistance. However, if course design materials did not describe the interaction, there was no reasonable assurance that students had any regular and substantive interaction with course mentors.

    These views essentially reject not just WGU’s approach to CBE but also the broader movement of faculty from “sage on the stage to guide on the side”. Instructors, from the OIG view, must provide instruction on course content and interactions must be pre-planned in the course design materials, at least for online courses.

    Self-Paced

    While I assume there were good reasons for WGU to want to avoid being classified as a non-term school, their 2005 declaration that they were not self-paced but rather term-based now appears to be a self-inflicted wound (page 34).

    Western Governors University Comments
    Western Governors University stated that its courses were not self-paced. Students were given a pacing guide and were expected to complete a certain number of competency units each term. In addition, a key role of the student mentor was to guide the pace of academic progress with individual students to ensure course completion by a certain date. Students had some flexibility in the pacing and moved through the content at different rates to allow for their individual competency development. However, that does not mean that the courses were self-paced.

    OIG Response
    Western Governors University’s statement that courses were not self-paced is contrary to its advertising materials, pacing guide descriptions, and statements from school officials we interviewed. According to the school’s web site, students could complete a degree program as soon as they successfully completed all of the necessary assessments. Students who completed assessments quicker could complete their degree quicker. The school’s web site also stated that students could complete assessments as soon as they were ready; if they were already competent in a subject area, they could prove it faster and complete their degree faster. If students completed their programs in less time, they paid only the tuition for the number of student terms in which they needed to enroll to complete the program. Course outlines stated the following about the pacing guides: “The pacing guide suggests a weekly structure to pace your completion of learning activities. It is provided as a suggestion and does not represent a mandatory schedule.” Western Governors University’s Program Development Operations Manager, Director of Assessment Design and Development, and Associate Provost for Academic Services confirmed to us during the audit that courses were self-paced.

    Ouch. It is far different for OIG to arbitrarily pick their own interpretations of regular and substantive interactions than it is for OIG to use WGU’s own descriptions and interviews. This seems to be a strong argument by OIG.

    WGU Response and Web Site

    WGU’s response was dated May 22 of this year, so they have had time to prepare for the audit findings. They have a web site that explains the situation, answers basic questions, and highlights their arguments against the findings.

    WGU strongly disagrees with the Inspector General’s audit report, which challenges our innovative, results-proven faculty model.

    With this key arguments:

    WGU has complied with the higher education laws and Department of Education guidance since our founding 20 years ago. Students, graduates, and employers of our graduates can rest assured that WGU’s accreditation and financial aid eligibility are intact.

    Our accreditor, the Northwest Commission on Colleges and Universities, has approved our faculty model and reaffirmed our accreditation in February 2017. Accreditors are responsible for determining whether a university is eligible for federal financial aid.

    Fixes Needed

    I agree with both Russ Poulin and Amy Laitinen regarding the need to fix but not remove the “regular and substantive interaction” regulations. From Inside Higher Ed’s coverage:

    Russell Poulin, director of policy and analysis at the WICHE Cooperative for Educational Technologies, said the department has done a “horrible” job of informing colleges about its expectations of how to comply with the regular-and-substantive requirements, which he said have changed over time.

    In addition, he criticized the inspector general’s decision to base its compliance position on disagreement about the mode of teaching at WGU when there is no evidence of any harm to students.

    “I totally agree with the intention of proponents of the ‘regular-and-substantive interaction’ rule, which is to avoid fraud. But it is an outdated method of reaching that goal,” he said via email, comparing it to a hypothetical decision by regulators to remove all ATM card readers because of the risk of credit card skimmers. [snip]

    Amy Laitinen, director of higher education policy for the group and a former Obama administration Education Department official, said the law was a response to rampant fraud and abuse.

    “We need to carefully fix (not gut) the now-outdated law to ensure that students are getting the academic and other supports that they need,” she said via email. “If we don’t do it carefully, it will be a fast race to the bottom, which would be bad for students and bad for the competency-based education community.”

    The Worst Part

    To me the worst part of the audit is the language used by the OIG that tries to make any disagreement with a hyper-literal translation of ambiguous regulations seem to be invalid (page 3).

    None of these 69 courses could reasonably be considered as providing regular and substantive interaction between students and instructors.

    Who holds these “unreasonable” views? The Department of Education and the Northwest Commission are two groups, among others. It’s bad enough that the OIG took the out-of-context, hyper-literal approach to the audit, but to not acknowledge the ambiguity and lack of clear guidance about this requirement is disingenuous.

    This audit is a travesty in my opinion. Even though it is likely to be rejected by the ED itself, it will have an impact, and the internal review of the audit will likely take years. I’ll write more about potential impact of the audit tomorrow.

    Additional Reading

  • “Alternative Pathways:” How to Rethink Vocational Education

    In Phil’s analysis of California Governor Jerry Brown’s directive for the California Community College System (CCCS) ((Disclosure: The Online Education Initiative from CCCS is a client of MindWires. The views in this and future posts represents my independent views and not OEI’s.)) to “take whatever steps necessary” to establish a fully online college, the punch line was as follows:

    What this points to is that for a new fully-online institution to get to some meaningful level of enrollment (let’s say 20,000) in the same ballpark as these comparison schools, I estimate it would take a full decade at the least….

    None of this analysis is to argue that CCCS should not try to establish a fully-online college. The goal of better serving nontraditional populations – adult students with and without jobs – is worth pursuing on its own merits.

    The numbers do argue, however, for a realistic view on the challenges the face:

    • Fighting against national demographic trends for adult students of community colleges;
    • Trying to avoid cannibalizing enrollment from existing California Community Colleges;
    • Having the patience to support the schools while it take years to grow to a size with meaningful enrollment levels; and
    • Accepting that best case this approach probably recovers less than 10% of the enrollment drop since 2009.

    Remember that the goal of this directive is to reach more non-traditional students. Community College Daily quotes CCCS Chancellor Eloy Oakley as saying

    We have literally tens of thousands of working adults with some college and no credentials and a couple of million working adults who are unemployed or underemployed,” Oakley said. “This is a wonderful opportunity to reach a population that really needs a community college to achieve economic mobility.

    So it’s worth looking at what other options may be available to address this goal. Fortuitously, Tyton Partners recently released a two-part report funded by The James Irvine Foundation called Path to Employment: Maximizing the Impact of Alternative Pathways Programs. [Registration required.] It provides a framework for analyzing the potential and critical success factors of shorter, non-degree and non-certification programs. There are various trendy Sillycon Valley buzzphrases associated with these sorts of programs, like “code academies,” “boot camps,” and “micro-credentials,” but they all fall under the broader heading of a term that has irrationally negative connotations in the United States: “vocational education.” For the purpose of this blog post, I am going to use vocational education and Tyton’s preferred term—Alternative Pathway Programs (APPs)—interchangeably.

    Tyton’s report provides an interesting general framework looking at how to think about these types of programs’ abilities to address the needs of non-traditional students, with a special focus on the state of California. It’s worth taking some time to examine aspects of the report in detail.

    Defining the Problem and the Goal

    First, let’s make sure we’re all talking about the same people, problems, and goals. Tyton’s report doesn’t talk about “non-traditional students” but rather “low-income adults,” which it defines as having the following attributes

    • May or may not be employed
    • Earn less than 200% of the federal poverty level
    • 18 years of age or older
    • Have limited or no exposure to postsecondary education

    I can’t say for certain whether that definition is one that Chancellor Oakley or Governor Brown would accept for the group of people they are trying to help, although I suspect that there is at least a strong overlap. For the remainder of this post, I will use the term “low-income adults” as defined by Tyton, since their analysis is built on that definition.

    There are approximately 7 million people who fit that definition in the state of California, “which accounts for nearly 37% of the state’s entire workforce,” according to Bureau of Labor Statistics numbers cited in the report.

    That’s a big number. How many of those people are touched by various educational programs?

    That grey space represents all the low-income adults in California who are receiving…nothing. But it’s actually worse than that in several ways, as Tyton points out in their report. First, not all of the 2.1 million California community college students fit the definition of “low-income adults.” Second, the average community college graduation rate is low. Nation-wide, it’s less than 30%. ((I chose to refer to the nation-wide graduation rate rather than California’s, largely because the nation-wide number is the one that the report uses.)) Particularly for low-income adults, going to school, incurring debt, and not getting a degree is worse than nothing.

    This brings up a second data fidelity problem, since one does not necessarily have to complete a degree in order to gain economic benefit from coursework. A worker could go back to school for a couple of accounting classes that enable her to get a better job, for example. One person’s degree non-completion is another person’s alternative pathway program. So the numbers Tyton uses as proxies for impact potentially both overestimate the number of low-income adults that California community colleges reach and underestimating the percentage of those it reaches that it economically impacts in a net positive way, although they are accurate enough for Tyton’s goal of painting a broad-brush picture of the magnitude of unmet need. More on the data issues later.

    So we have a severe problem of scaling access, even in a state that is historically known for its heavy investment in education. Even if a new online community campus were created and, under Phil’s most optimistic scenario, added another 20,000 enrollments over the next 10 years, that’s a drop in the bucket even before you consider that not all of those students would fit the “low-income adult” definition and it’s possible that considerably less than half of them would graduate. And this gap persists in spite of California spending “roughly 2.5% of the state’s entire annual budget” on educational programs for low-income students, according to Tyton’s figures.

    None of this is to cast aspersions on either the community college system or the idea of an online campus. Rather, the point is that the challenge is enormous. As Tyton puts it,

    [E]ven if all the spots within [community colleges and the two other identified types of] programs were allocated to low-income adults seeking to enhance their employment prospects, current capacity would support less than a third of the potential annual demand. Expanding the number of successful models that can support education-to-employment pathways for adults is imperative, both from within the current ecosystem of institutions and workforce programs and through new, innovative program models.

    The idea that Tyton’s report explores, which is not positioned as an alternative to existing programs but rather another tool in the toolbox, is what they call APP:

    An Alternative Pathways Program (APP) is defined as one that:

    • Focuses on education and training for specific job and career pathways

    • Maintains close alignment with employers and industries to facilitate job placement for participants completing the program

    • Does not offer a traditional postsecondary degree or certificate

    Most APPs focus on recruiting and serving participants directly, similar to colleges and universities, but they vary widely in their training model and program length, among other attributes. Some program models connect participants directly with employment opportunities. For example, the high-profile technology, design, and data “boot camps” o er a career-fair job connections model and last less than 12 weeks on average, while experiential learning programs such as apprenticeship programs prepare participants over a longer period (most are 6–12 months long) for a specific career path. Other programs do not connect participants with jobs, but they still offer training for specific career pathway and offer a certification or credential upon completion. Another set of programs focuses on general education as a baseline for specific career pathways, including programs or courses that help participants gain postsecondary credentials or learn general skills.

    Tyton defines the common goal that programs under this broad umbrella could solve as creating “a path for improving the economic and employment opportunities and outcomes for low-income adults.” So these are focused, vocational education programs aimed at helping adults stuck at the bottom rung of the income latter achieve enough education to begin climbing to higher rungs. Obviously, this is not the only set of goals one could have for post-secondary education. One could be concerned about developing more informed citizens, enriching people’s intellectual lives, giving them access to career paths that they didn’t know existed, cultivating a common national culture, among others. It is both reasonable and important to talk about different educational goals that we should aspire to. But these sorts of conversations tend to take on a one-goal-fits-all tone. Take a look again at the grey space in that last graphic:

    The people in that space are not currently being served by programs that address any of those goals. Tyton asks two basic questions. First, to what degree might APPs—many of which do not currently serve low-income adults—be recruited to fill in some of that grey space? And second, what are the characteristics of an APP that would make it most likely to achieve this goal? The first question can only be answered in a fairly general way, since we have no systematic tests of it yet. Tyton spends the bulk of their report on the second question by analyzing patterns across over 125 existing APPs.

    I am not going to provide a detailed critique of Tyton’s analysis framework in this post. Rather, I’ll be examining the ways in which having such a framework enables more productive and nuanced discussions addressing big educational problems like helping low-income adults using tools ranging from policy to educational technologies.

    The Six Pillars

    Much of Tyton’s analysis rests on what they call the “six pillars” of alternative pathways programs:

    • Enrollment policies: Processes and guidelines for admitting participants
    • Participant support: Resources and methods that support participants in overcoming life challenges
    • Labor market alignment: Level of program fit with the needs of employers and the local/regional economy
    • Connections: Extent to which program connects participants with employers and other job search resources
    • Training mix: Balance of curriculum emphasis on soft skills vs. academic and technical skills
    • Financial model: Ability to generate revenue and achieve organizational stability

    In and of themselves, there’s nothing earth-shaking about these categories. But stating them explicitly as part of the analytic framework enables us to do all kinds of additional important work.

    First, it enables us to ask, “Is this a complete and plausible list of critical success factors for a vocational program?” For example, the work of Vincent Tinto and others connects students’ sense of belonging in campus community to their likelihood of completing their degrees. Is that principle also operative in vocational programs for low-income adults? If so, how much of an effect does it have? And can it be subsumed under “participant support,” or is it distinct and important enough to merit its own pillar?

    This line of questioning brings up a second advantage of having such a framework, which is that it points to a research agenda. What sorts of participant support are most effective for helping low-income adults to complete APPs? Does the answer to that question vary by context? If so, which sorts of contextual factors matter the most? Tyton has proposed a set of preliminary hypotheses for the optimal way to address each pillar based on the research that they conducted:

    The point of the hypothesis is to have a truth proposition that can be tested. For example, Tyton has listed “employer as payer” as the “optimal model” for vocational programs. Going back to Governor Brown’s online campus proposition mentioned at the top of this post, California has funding mechanisms at its disposal that many of the start-up programs examined by Tyton do not. One might float an alternative hypothesis for that pillar and then test that hypothesis through various means.

    The framework also enables program designers to think about trade-offs more clearly. A classic trade-off is between enrollment policies and participant support. Tyton articulates several alternative models for each pillar. Here’s the selection of models they examine for support:

    But these models don’t exist independently of the other pillars. If your program has an enrollment process that is extremely good at identifying students who come to the program already possessing many of the skills and life circumstances that would enable them to succeed if only they were given the opportunity, then you may need less investment in participant support. Conversely, if the program’s goal is to help those students who have to face the most daunting obstacles to their education, then investment in support becomes more important.

    This example illustrates the point that we should interpret the word “optimal” loosely here. That large swath of grey in the earlier diagram representing all the low-income adults who are getting no education is not one homogenous blob of unmet need but rather a collection of millions of people with different strengths, needs, and circumstances. People in different situations will likely need different program designs in order to be successful. As a sector, we are thankfully past the peak of the idea that we can teach everything to everybody by posting video lectures online and calling them “MOOCs.” ((Not very far past, but still. Baby steps.))

    This brings us to one of the most deeply divisive terms in education: scale.

    The Tyton report examines two types of scale: inputs and outputs. Access and outcomes. Let’s imagine that California were somehow able to get 100% of low-income adults enrolled in community college degree programs. That would be good, right? Well, maybe. Let’s also suppose that, in doing so, California’s degree completion rate settles at the national average of 30% for community colleges. That would mean 70% of California’s low-income adults would end up still without a degree and still (likely) without improving their economic prospects. To the contrary, many of them would have increased levels of debt they would have to pay out of their unchanged salaries. So no, scaling access is not inherently good.

    For each of Tyton’s hypothesized optimal strategies in each of their hypothesized pillars, they analyze the trade-off the strategy makes between scale of access and scale of outcomes:

    If a chosen strategy for a pillar has a downward-pointing red arrow, then it presents a challenge to scaling access. On-site support to students is harder to provide to many students than providing students with no support. It’s harder to scale enrollments—access—for a program that commits to high-touch student support. Then why do it? Hopefully because it improves the percentage of degree completions and other positive outcomes.

    Taken together, the elements of Tyton’s framework give program designers and policy makers both a set of knobs they can turn in an effort to tune a particular program to the needs of a particular student population as well as a lens through which they can examine, test, and refine their assumptions in ways that will improve the effectiveness of those knobs and our knowledge of how the various settings interact with each other.

    Of course, having knobs to twiddle is good, but being able to measure the impact of your knob twiddling is critical. As I mentioned earlier, the numbers Tyton cites for community college impact on low-income adults are less than perfect measures. While adequate for the purpose of assessing the order of magnitude of unmet need in a large state like California, the two proxies the report uses for scale or access and outcomes—number of students served and graduation rate—are not great for measuring impact at a more granular level. And it will be tough to develop better ones. Going back to the example of the person who takes a couple of accounting classes at a community college and gets a better job as a result, how would one capture that? And yet, that is precisely what we would need to do to get clearer sense of the degree to which community colleges are currently serving the needs of low-income adults and the degree to which tweaking community college programs in various ways might increase that impact. Other APP efforts will likely face similar data quality challenges. So application of the model in practice will likely require some innovation around measurement in these areas.

    But here again, having a holistic model can help. Tyton highlights sustainability as the big reason to think about employer-pay business models. But if you happen to be thinking about the data problem while looking at the Tyton pillars graphic, it might occur to you that a direct relationship with employers can help with that problem. To start with, employers’ willingness to pay might be a good proxy for career progression benefits that students gain from their participation in the program. One would have to do the research, but it’s a plausible hypothesis. Second, the closer relationship with the employer makes outcomes data easier to get, perhaps in the form of anonymized aggregate data from the employers or by providing richer, longer term relationships with students that give them more incentive to provide the school with post-graduation follow-up data.

    What About Ed Tech?

    Nowhere does this framework explicitly address ed tech as such. And yet, ed tech decisions both large and small are often made by directly connecting a problem with a technological solution. A state wants to help reach more non-traditional students. Online learning can reach more students. So why not start an online learning program? A foundation or a college wants to help more first-generation college students make it through college. Adaptive learning programs seem to help some students in developmental math programs get past that stumbling block to their degree completion. So why not invest in adaptive learning programs?

    Absent of a richer analytic framework, these efforts are more likely to fail and less likely to be reproducible because they don’t start with either a holistic understanding of the needs of the students the efforts are trying to help or a clear understanding of how various aspects of the support ecosystem interact with each other. Without a theoretical framework, you can’t construct a clear hypothesis. Without a clear hypothesis, you can’t construct a proper experiment. Without a proper experiment, you can’t learn what works and what doesn’t.

    Very often, the best moment to think about ed tech is immediately after you have developed your program design and analyzed it for strengths and weaknesses. At this stage in the thinking, ed tech can potentially help by changing the laws of physics that underpin your model. OK, so low-income adult students need high-touch support which, when implemented in the traditional way, is resource intensive and therefore limits the number of students you can serve. Is there a way that technology can help provide that high-touch support at a lower resource cost? (By the way, the solution may not be to build robot advisors in the sky that can semi-read students’ minds. Instead, it might be saving advisors’ time spent doing other, non-student-facing work so that the same number of advisors have more time to serve students well.)

    I have no strong opinions about the answers that the Tyton paper arrives at, but I do believe that they are asking roughly the right questions in roughly the right order. As a field, we need more of this type of program- and policy-level research and analysis to inform a wide range of strategic decisions, including but not limited to use of ed tech. It is an exemplar of a genre of educational research that we should be looking to grow, propagate, and use to inform practical decision-making.

    This post is part of our Research in Translation series, which is funded in part by the Bill & Melinda Gates Foundation. The findings and conclusions (or views) contained within are those of the authors and do not necessarily reflect positions or policies of the Bill & Melinda Gates Foundation.