e-Literate

Present is Prologue

Tag: Gates Foundation

  • Postscript on Rio Salado Coverage: Clarity about different outcome types

    Postscript on Rio Salado Coverage: Clarity about different outcome types

    After my two posts two weeks ago calling into question whether Rio Salado College has demonstrated clear student outcomes that justify its usage as an exemplar institution, college officials have responded through the WCET blog and in an article from Paul Fain at Inside Higher Ed. I recommend reading the IHE article in particular to get an understanding of the challenging an nuanced question of what is “good enough” in terms of college completion, and for even more detail read Rio’s written response. Unfortunately, the lengthy explanations on graduation rates from Rio end up obscuring a critical distinction between awarding degrees versus certificates of completion. The reason this matters is that Rio appears to be doing a better than average job supporting student transfers to four-year degree programs, a worse than average job of helping students complete two-year degrees, and an unknown job of providing students with non-degree stand-alone certificates.

    Before dealing with this issue, I’d like to point out how useful this dialog has been and how open Rio has been in sharing internal metrics. One example that I wish other institutions used, and that can help us clarify how to understand completion data, is based on student goals.

    Rio Student Intentions

    Fully 76% of Rio students have no intention to get a degree or certificate, and of this group 33% (11,628) indicated earning transferable credit as the key reason to enroll. As Sally Johnstone said in the IHE article, “It’s a feeder school for Arizona State University”. This transfer of credit mission can be good or bad – it depends on how well these students perform in getting an ASU degree. In the written response, but not in WCET or IHE articles, Rio shares an interesting but incomplete metric.

    Additionally, as part of the work RSC does related to accountability, we track the success of our transfer students to the three public universities in Arizona (The 2010/11 – 2015/16 5 year trend shows a 77.10% increase in bachelor degrees awarded to students that had completed a minimum of 12 credit hours (up to 60 or more hours) at Rio Salado College).

    Increase to what and over what? It is not clear if this is comparing to Arizona university students who do not transfer any credits from Rio, or those who transfer less than 12, or those who transfer from other colleges. Nevertheless, this is an indicator of the college helping students obtain four-year degrees through credit transfers.

    Once you get beyond non-degree / non-certificate seeking students, the numbers are murkier, and this is the area that needs clarity. One out of four Rio students are there to get an award – how successful are they? Rio recommends we look at the Voluntary Framework for Accountability (VFA) measures, instead of IPEDS, with the following results.

    • 2-year graduation rate is higher than peer institutions: Rio 13%; Peer Institutions 10%
    • 6-year graduation rate for credential-seeking cohort is 36.2%
    • The credential seeking cohort far outperformed sister institutions in completion: Rio 42%; Peer Institutions 16%

    I do not know why Rio uses the term graduation rate, as VFA instead describes Percent Students Completed a Formal Award, but from the numbers used we can see where the data comes from. The first and third bullets come from the VFA Two-Year Progress measures (blue column under Completed). ((The VFA definition of the cohorts: Main Cohort = fall entering, first time at reporting college; “all students”; Credential Seeking = earned 12 credits by end of year two.))

    What is key is that the Completed metric includes those who obtained an associate’s degree or official certificate. Unlike degrees, certificates of completion have no standards across schools, and their usage is very uneven. As an example of Rio’s usage:

    Notably, since 2010 Rio has defined and awarded more and more of these <1 year certificates, based on IPEDS data.

    We can see is that the large majority of completions, which Rio labels under graduation rate, comes from the nonstandard certificates rather than degrees.

    One of the strengths of VFA is that under the Six-Year Outcomes measures, you can separate out the various types of award. Below is the data for Rio – pay attention to the Credential-Seeking Cohort, as it removes the majority of non-degree / non-certificate seeking students at Rio only looking to transfer a small number of credits. ((For degree or certificate-seeking cohorts, I prefer the Rio definition based on student self-reporting, but the VFA definitions get at the same idea.)) Also note that VFA outcomes are mutually-exclusive and hierarchical in the order shown. If a student gets a certificate and a degree, they are listed under the degree outcome.

    For the Credential-Seeking Cohort, 22% of Rio students get a certificate and do not transfer to a four-year school, 7% get a certificate and transfer, 3% get an associate’s degree and do not transfer, and 4% get an associate’s degree and transfer. A large majority of Rio Salado completions are for certificates, which is consistent with the IPEDS data.

    How does Rio compare to other large community colleges? While we have not analyzed all 200+ colleges in VFA, spot checking with some peer schools indicates that Rio is well below others in awarding degrees but well above others in awarding certificates.

    The resulting data provide the basis of my comment in IHE:

    “This is not a bad situation, per se, as long as students are gaining value in the workplace for these official certificates,” he said via email. “But certificates are not useful in terms of comparing apples to apples, especially when one school uses them liberally, as does Rio, and most other schools do not.”

    Hill also noted that Rio Salado’s VFA completion rate for associate degree programs is a “troublesome” 7 percent.

    While I appreciate the valuable sharing of information from Rio Salado in this process, I’ll stick with my original conclusion:

    At best, this is a school with mixed results that should not simply be labeled a success without caveats or explanations.

    Rio has shown that they can cut costs and still be a successful transfer school, but not that they can be a successful degree school. This matters a great deal in terms of how transferrable the lessons are. It’s fine to focus on transfers when you’re right next door to ASU. But what if you’re Adirondack Community College? There are places where degree completion matter more or less than at Rio. And what about those students going to Rio who *are* seeking degrees? Does this model promote some educational outcomes at the expense of others? And if so, shouldn’t students know about that before enrolling?

  • Rio Salado College As Exemplar: A critical external view

    In yesterday’s post I described how a review of two courses at Rio Salado College indicated reasons to question the use of this school in the ASU / BCG case study report on Digital Learning ((Disclosure: Our e-Literate TV series was funded in part by the Bill & Melinda Gates Foundation who also funded the ASU / BCG study.)). The report, titled “Making Digital Learning Work: Success Strategies From Six Leading Universities and Community Colleges”, has the following description [emphasis added]:

    How can the use of digital technologies in postsecondary education impact students’ access to education, student outcomes, and the return on investment for students and institutions? What are the biggest challenges for an institution seeking to implement high-quality digital learning opportunities? What promising practices enable an institution to achieve impact at a larger scale? [snip]

    The answers, at least in part, lie in case studies of six colleges and universities: Arizona State University, the University of Central Florida, Georgia State University, Houston Community College, Kentucky Community and Technical College System, and Rio Salado Community College. The first three institutions in this list are public research universities, representing different geographic populations and access missions. The other three institutions include two community colleges and a state-wide community college system.

    These six institutions have a strong track record of using digital learning to serve large, socioeconomically diverse student populations, and each has been a pioneer in innovating to expand access to postsecondary education, improve student outcomes, and provide higher education at an affordable cost.

    Let’s look at the track record of Rio Salado College in terms of aggregate academic student outcomes to see how appropriate it is to include them as an exemplar in such a case study-based report. For much of my analysis, I looked at the 862 public 2-year colleges fully reporting data in the Fall 2016 IPEDS data set.

    Reduced Expenditures

    The primary claim made in the ASU / BCG report is that Rio Salado is has reduced costs.

    This claim does hold up to scrutiny, as Rio’s $89 of instructional delivery costs per student credit hour and $16 of student services costs are 10th and 9th lowest in the country for the 862 colleges I reviewed.

    The primary outcomes claims made in the case study quickly brush off the standard metrics for Rio.

    Rio Salado’s success is not defined solely by improvements in graduation and retention rates. In particular, the college has a high transfer-out rate (32% compared to an average of 19% for other MCCCD colleges [ed. – the other schools in the Maricopa County Community College District]), and the students who transfer to Arizona universities from Rio Salado have a 74% four-year graduation rate—3 percentage points higher than the average for other MCCCD transfer students. At Rio Salado itself, students’ course-level success rates have slowly been improving over time, to about 64% in 2016.

    Graduation and Retention Rates

    It is all well and good to point out the importance of transfer students, but we should not ignore graduation and retention rates. And we should explore whether the data supports the phrase “improvements in graduation and retention rates”.

    The report describes Rio’s investment in a suite of advising tools.

    Obtaining these advisory tools required an initial investment of $1 million, but the tools pay for themselves by increasing Rio Salado’s term-to-term retention by 7%, increasing ROI through improved student progress and the additional tuition dollars that the institution receives from retained students.

    Looking at IPEDS data, we can see Year 1 to Year 2 retention rates (not quite the same as term-to-term), and it shows improvement since 2013. Unfortunately, Rio’s retention rates for full-time and part-time students are both in the bottom 10% of all community colleges at 33% and 27%, respectively. And the full-time rate is lower than it was prior to 2011.

    Rio Salado Year 2 Retention Rates

    While the ASU / BCG report does not mention graduation rates directly, there have been several claims made about Rio’s numbers in other publications.

    The problem is that Rio Salado’s 4-year graduation rate for first-time full-time students as reported in IPEDS is 5% – the second lowest of any public 2-year college in the nation. How can we resolve this discrepancy?

    The key to understanding the claims in these four articles is to follow the two links, which both point to a customized IPEDS Feedback Report from 2013 for Rio against their selection of 100 peer institutions.

    Figures 10 and 11 from IPEDS report

    This measure captures any award – degree or certificate – for a small cohort of entering full-time students at Rio from 2009. If you trace that data through IPEDS, you see that this measure is for a 150-student cohort (in 2012) out of the 45,000+ students at Rio.

    Using this specific metric, you can find the 42% graduation rate in 2011, and the “four times greater” rate of 27% in 2012, both as the rate plummets from 64% in 2008 to 3% in 2016.

    Using broader, up-to-date metrics for graduation and retention rates as well as the new IPEDS outcomes measures (which allows comparison outside of just full-time first-time cohorts), we see that Rio Salado College has some of the lowest student outcomes measures in the country.

    For all but two measures (6-year First-Time Full-Time Awards and 6-year Non-First-Time Full-Time Awards), the performance is in the bottom 10% of all colleges in the study. For those two awards measures, the performance is in the bottom half of the country and represents just 11% of the student body.

    Transfer Rates

    What we are left with are reports about high transfer rates and subsequent success rates 3% higher than other transfers out of the same district. The IPEDS data set now includes transfer rates in its new Outcomes Measures section, and it is true that Rio Salado College does transfer out a significant portion of students. For all cohorts defined below, Rio is in top 20% of public 2-year institutions.

    • First-Time Full-Time Students: 39% transfer rate, Rank 170 out of 862
    • First-Time Part-Time Students: 47% transfer rate, Rank 112 out of 862
    • Non-First-Time Full-Time Students: 46% transfer rate, Rank 196 out of 862
    • Non-First-Time Part-Time Students: 59% transfer rate, Rank 121 out of 862

    Note that this data indicates that students transferred to another institution and are still enrolled there. This data does not indicate what portion of the transfers were planned versus students deciding to move on for other reasons. Nevertheless, for a community college, especially one with very close ties to ASU (I have been told that a large percentage of Rio students are ASU students trying to fill out their schedule based on saving money and more convenient schedules), this is a favorable metric.

    I cannot provide independent confirmation on the subsequent success rates, so we’ll accept that claim at face value.

    Case Study Questions

    I should note that Rio Salado College does provide accurate information on the metrics mentioned in this post, both through IPEDS reporting and through their Research & Planning section of the web site. The picture that emerges from the aggregate outcomes data is of a large school that:

    • transfers out greater than average percentage of students;
    • is just below average for awarding certificates for full-time students;
    • is among the poorest performing in the country for retaining students;
    • is among the poorest performing in the country for awarding degrees; and
    • is among the poorest performing in the country for awarding certificates for part-time students.

    At best, this is a school with mixed results that should not simply be labeled a success without caveats or explanations.

    One question we should ask is whether it is appropriate to hold up a school with some of the lowest student outcomes measures in the country as an exemplar. Yes, Rio Salado has found a way to spend as little as possible on instruction and student support services, and yes, a lot of students transfer out, but that is not enough. We need greater evidence of student success if we are to use them as a case study for others to emulate. And we also need a more robust genre of a case study that looks across, at a minimum, a relatively standard set of publicly-available information, and deeper dives where appropriate, to understand educational practices and their impacts on students. These case studies should present information in context – the good and the bad – since education is complex and challenging, and a clear focus on evidence will benefit all parties in the end.

  • Rio Salado College As Exemplar: A critical internal view

    Rio Salado College As Exemplar: A critical internal view

    10 days ago Arizona State University (ASU) and the Boston Consulting Group (BCG) released a report, supported by the Bill & Melinda Gates Foundation (BMGF) ((Disclosure: Our e-Literate TV series was funded in part by the Bill & Melinda Gates Foundation.)), titled “Making Digital Learning Work: Success Strategies From Six Leading Universities and Community Colleges”. The basic idea [emphasis added]:

    How can the use of digital technologies in postsecondary education impact students’ access to education, student outcomes, and the return on investment for students and institutions? What are the biggest challenges for an institution seeking to implement high-quality digital learning opportunities? What promising practices enable an institution to achieve impact at a larger scale? [snip]

    The answers, at least in part, lie in case studies of six colleges and universities: Arizona State University, the University of Central Florida, Georgia State University, Houston Community College, Kentucky Community and Technical College System, and Rio Salado Community College. The first three institutions in this list are public research universities, representing different geographic populations and access missions. The other three institutions include two community colleges and a state-wide community college system.

    These six institutions have a strong track record of using digital learning to serve large, socioeconomically diverse student populations, and each has been a pioneer in innovating to expand access to postsecondary education, improve student outcomes, and provide higher education at an affordable cost.

    The methodology is a case study of each school, and the report then takes a stance on what other institutions should do.

    Now is the time for leaders to champion the potential of digital learning to open the doors of higher education wider and to improve student outcomes, while operating more efficiently and at lower cost. The journey of each college or university will be unique, but the set of promising practices described in this report may serve as a useful guide for all institutions.

    There are multiple methods to evaluating a school; it turns out that some are more meaningful than others, and it is always helpful to start with what students experience in actual courses. The ASU / BCG report provides useful context on course design.

    At Rio Salado, 22 full-time faculty chairs develop courses with the support of a central team that includes subject-matter experts, instructional designers, media support staff, and production staff. About 1,500 adjunct faculty members teach the courses, which they can personalize by adding an introductory message or video for each module.

    It turns out that I have access to two courses – ENG102 (First Year Composition) and EED200 (Foundations of Early Childhood Education) – and this gives an opportunity to better understand the student experience. What these courses indicate, however, is a troubling lack of meaningful interaction between faculty and students.

    Looking at ENG102, the course materials is Rio Salado developed courseware embedded in their custom course management system, RioLearn. The materials appear to be quite extensive, and all materials are available at the start of the term in 14 lessons. To find actual due dates, students have to go to the gradebook, as no specific dates are included in the courseware.

    Course Welcome Message

    The instructor sent out several announcements at the beginning of class, a welcome letter, and then one reminder or update message at the beginning of each month. That’s it for instructor-initiated interactions.

    Course announcements

    Sample Message

    There were no student-to-student discussions, as the discussion board was unused for the course. There was one peer-review activity, but otherwise no interactions between students. I have been a frequent critic of threaded discussion boards, but it is certainly better to have something instead of nothing.

    Discussion Tool

    The rest of the interactions came in two categories: grading of assignments and responses to student-initiated messages. The primary feedback method in these two courses was faculty usage of the custom-developed Feedback Tool, which uses rubrics to grade assignments.

    Lesson Feedback

    There were several assignments that included annotated mark-ups of the submitted papers, which appear to be the most useful feedback from instructors.

    A review of EED200 shows the same course structure – few instructor-initiated interactions, use of rubric grading as feedback on assignments, and specific responses when students send questions in through the Message Center.

    For both courses, the instructors typically responded to student messages within a day or two.

    This approach is troubling, as both courses appear to not meet the “regular and substantive interaction” regulation for credit-bearing online courses. I have been critical about the vague standards and the egregious application by the Department of Education’s Office of Inspector General audit of Western Governors University, but these complaints do not mean that the regulation has no point. The idea is that course design and facilitation should be implemented to ensure that students are not left to figure out static course materials and to be responsible for initiating most forms of interaction. As stated in a 2014 Dear Colleague Letter on the subject of regular and substantive interaction based on competency-based education (CBE) approaches [emphasis added]:

    We do not consider interaction that is wholly optional or initiated primarily by the student to be regular and substantive interaction between students and instructors. Interaction that occurs only upon the request of the student (either electronically or otherwise) would not be considered regular and substantive interaction.

    Some institutions design their CBE programs using a faculty model where no single faculty member is responsible for all aspects of a given course or competency. In these models, different instructors might perform different roles: for example, some working with students to develop and implement an academic action plan, others evaluating assessments and providing substantive feedback (merely grading a test or paper would not be substantive interaction), and still others responding to content questions.

    The problem with case studies is that the selection of cases to study may not be representative or appropriate to prove one’s thesis. It is possible that I happened to look in detail at two courses that are aberrations. But even so, this view is troubling, as these are centrally-developed courses (not subject to the whim of individual instructors), and both courses have little meaningful provisions for faculty-student or student-student interactions. The instructors were responsive to messages, but that is not enough to back up claims of high-quality courses. The dissonance between these two courses and what is described in the ASU / BCG report led me to take a critical look at the data.

    For the next post, I’ll take the external view and look at aggregate public data on Rio Salado College on academic outcomes.

  • SRI’s Study on Gates Personalized Learning Grants Is Out

    This is almost old news now, but we just haven’t been able to dig into it yet. As part of its Adaptive Learning Market Acceleration Program (ALMAP) program, the Gates Foundation funded SRI to do a study of the results of the grants after two years. I hope to finally clear some time to parse through it this week, but two high-level points jump out at me at first glance (neither of which is a big surprise):

    1. There are no conclusive wins here. This is not a robot-tutor-in-the-sky moment. A few programs did well here and there. A handful produced promising incremental gains. But this is not a report that screams, “Wow, adaptive courseware works!” The most you can say is that adaptive learning looks like it could be another arrow in the quiver that helps out in some situations—like developmental math courses at two-year colleges, for example.
    2. Large-scale educational research is incredibly hard and may actually be impossible to do rigorously for certain kinds of questions. I’ll probably get into this more when I’m ready to really write about the report, but one reason the conclusions are murky is because there so many variables in each class—not just each course subject, not just each course at one university, but even with each section of each class taught by one teacher—that really matter, some of which are impossible to control and others of which are unethical to control. It would be a mistake to overinterpret the study as showing that adaptive learning doesn’t help much. I’ve seen studies with narrower focus that have gotten clearer gains. This brings us back to the arrow-in-the-quiver theory. The wider the scope of the research focus, the more that pockets of real benefit will be obscured by noise and uncontrolled variables.

    The report is here.

  • Release of University of California at Davis Case Study on e-Literate TV

    Today we are thrilled to release the fifth and final case study in our new e-Literate TV series on “personalized learning”. In this series, we examine how that term, which is heavily marketed but poorly defined, is implemented on the ground at a variety of colleges and universities. We plan to cap off this series with two analysis episodes looking at themes across the case studies.

    We are adding three episodes from the University of California at Davis (UC Davis), a large research university that has a strong emphasis in science, technology, engineering, and math or STEM fields. The school has determined that the biggest opportunity to improve STEM education is to improve the success rates in introductory sciences classes – the ones typically taught in large lecture format at universities of their size. Can you personalize this most impersonal of academic experiences? What opportunities and barriers do institutions face when they try to extend personalized learning approaches?

    You can see all the case studies (either 2 or 3 per case study) at the series link, and you can access individual episodes below. (more…)

  • Release of Empire State College Case Study on e-Literate TV

    Today we are thrilled to release the fourth case study in our new e-Literate TV series on “personalized learning”. In this series, we examine how that term, which is heavily marketed but poorly defined, is implemented on the ground at a variety of colleges and universities.

    We are adding two episodes from Empire State College (ESC), a school that was founded in 1971 as part of the State University of New York. Through a lot of one-on-one, student-faculty interactions, the school was designed to serve the needs of students who don’t do well at traditional colleges. What problems are they trying to solve? How do students view some of the changes? What role does the practice of granting prior-learning assessments (PLA) play in non-traditional students’ education?

    You can see all the case studies (either 2 or 3 per case study) at the series link, and you can access individual episodes below. (more…)

  • Personalized Learning Changes: Effect on instructors and coaches

    Kate Bowles left an interesting comment at my previous post about an ASU episode on e-Literate TV, where I argued that there is a profound change in the instructor role. Her comment:

    Phil, I’m interested to know if you found anything out about the pay rates for coaches v TAs. I’m also interested in what coaches were actually paid to do — how the parameters of their employable hours fit what they ended up doing. Academics are rarely encouraged to think of their work in terms of billable increments, because this would sink the ship. But still I’m curious. Did ASU really just hike up their staffing costs in moving to personalised learning, or was there some other cost efficiency here? If the overall increase in students paid off, how did this happen? I’m grappling with how this worked for ASU in budgetary terms, as the pedagogical gain is so clear.

    This comment happened to coincide with my participation in WCET’s Leadership Summit on Adaptive Learning, where similar subjects were being discussed. For the purposes of this blog post, we’ll use the “personalized learning” language, which includes use of adaptive software as a subset. Let’s first address the ASU-specific questions. (more…)