Present is Prologue

Category: Strategy & Change Management

Terminology is Key to Understanding Blackboard Learn Prospects
There are three observations from Blackboard’s users conference that we feel are important to share before we pull together our thoughts for more comprehensive posts, and all three issues build off of Blackboard’s strong focus at the conference on Learn Ultra as the future of their LMS product line.

Learn Ultra “In Production” or “Using Ultra”

The first issue is terminology around Learn Ultra. Blackboard ((Disclosure: Blackboard is a client of the e-Literate LMS Market Analysis service and a participating sponsor in our Empirical Educator Project.)) is pushing the metric that there are 61 or 62 Learn Ultra customers “in production” or “using Ultra”, yet we have found very few that use, or even plan to use, Learn Ultra as their primary, institution-wide LMS. What gives? What became quite clear at the conference is that when Blackboard says in production, what they mean is that the LMS administrator has enabled the Ultra global navigation, which uses the new Ultra user experience framework as the landing page / dashboard with activity feed that users see before entering a specific course. The company calls this Base Navigation, but at this point every course can be configured to be in the Original Experience or the Ultra Experience. Thus, enabling the possibility of running a course in Ultra counts as in production.

Once a school has enabled Learn Ultra Base Navigation, they could choose to move exclusively to Ultra (e.g. the University of Phoenix, Northwest Florida State College, and a few others), or they could choose to keep all courses in Original (e.g. Northeastern State University), or they could choose to have some courses in Ultra and some in Original (used by the majority of schools investigating Ultra). This last mode is known as Dual Course mode, and even Blackboard executives are surprised to find out that the vast majority of schools putting Ultra in production are in fact running in Dual Course. For many of these schools, there are no definitive plans to ever move exclusively to Ultra.

This distinction is important, as the majority of functionality for an LMS occurs within a course, even if we did hear a few schools present that they saw some end-user benefits to having the landing page itself. We will add more commentary on this subject in future posts, but for now the takeaway is that Ultra in production numbers from Blackboard do not necessarily mean that any or most courses are in the Ultra Experience.

Product Variations Resulting in Three Dates for New Features

The second issue is that Blackboard Learn has two experiences (Ultra and Original), three deployments (self-hosting, managed-hosting by Blackboard, and SaaS-hosting at using AWS), often resulting in three different dates for full delivery of new features.

Keeping in mind that the Original Experience is available on all three deployment models but Ultra is available only in SaaS, here is the view of the final delivery groupings for the courses:
- Learn Original on self-hosting and Learn Original on managed-hosting, which the bulk of their customers use, is the first case.
- Learn Original on SaaS-hosting, which represents the bulk of their 383 customers on Learn SaaS announced at the conference, is the second case.
- Learn Ultra, which represents some subset of the 62 customers announced as “Ultra in production”, is the third case.
Of course there is significant overlap in terms of common code, such as the micro-services running in the SaaS environment for both experiences, or shared source code between deployment options. But from a feature delivery perspective, the full release often has three different delivery dates – Original SH & MH, Original SaaS, and Ultra.

This is good news or bad news, depending on your perspective. For Blackboard’s customers, it means more options without forced migrations. Blackboard staff stated several times in presentations and in hallway conversations that none of the deployment options or experiences have any plans to go end-of-life or even into maintenance mode. But on the other hand, Blackboard will not achieve many of the benefits of becoming a cloud product company until they can move the majority of development purely onto SaaS.

SaaS Better Indicator Than Ultra of Client Retention

The third issue, which is related to the first two, is that we believe ((OK, OK. I admit that Michael should get credit for seeing this issue more clearly than I did. Not many people read footnotes, so I feel comfortable with this admission.)) that the migration to Learn SaaS might be a better indicator – at least in the short run – than Ultra adoption of whether a school plans to stick with Blackboard. Whether or not the school enables Ultra base navigation or any courses in the Ultra Experience.

When a school moves to Learn SaaS, they tend to sign contract extensions for 1 – 3 years to cover the new services. And the migration to Learn SaaS does not suffer from the vague terminology issues – a school either uses Learn deployed on SaaS (through AWS) or they don’t.

For now, just treat this as clarification on the complexities of Blackboard Learn LMS options that became more apparent at the conference.

Update 7/23: Changed header and some text in second section to focus on dates of feature delivery.
July 22, 2018
Revisiting 2012 Post on Barriers That MOOCs Would Face
Inside Higher Ed published an article today, titled “Free MOOCs Face The Music”, about edX quietly adding support fees for many of their courses. Dhawal Shah and I both commented that we were not surprised by the move.

Writing about the introduction of the fee, Dhawal Shah, founder and CEO of Class Central, a review site for online courses, said the announcement was the latest in a phenomenon he termed “the shrinking of free.” Regardless of MOOC provider — be it edX, Coursera, Udacity or FutureLearn — “all have cut back on what was originally free in MOOCs.”

Phil Hill, co-founder of Mindwires Consulting and an author of the e-Literate blog, agreed that the edX announcement was not surprising. Early MOOC providers like edX thought they would be able to “get really big for free,” said Hill. “Magic didn’t happen, and now they’re facing reality.”

There’s more information in the article worth reading, but I would like to revisit a post here at e-Literate from 2012 to help explain the point I made. In “Four Barriers That MOOCs Must Overcome To Build a Sustainable Model”, I noted:
The current generation of courses has proven the feasibility of massive online enrollments, but the Kolowich article reveals that the result is based on a form of adult continuing education. The majority of students in the Udacity and Coursera courses analyzed were professionals in the software industry – hardly the target audience for those seeking a change in how we educate postsecondary students. The current MOOCs provide a nice proof-of-concept, but hardly solve significant educational problems.

So what are the barriers that must be overcome for the MOOC concept (in future generations) to become self-sustaining? To me the most obvious barriers are:
- Developing revenue models to make the concept self-sustaining;
- Delivering valuable signifiers of completion such as credentials, badges or acceptance into accredited programs;
- Providing an experience and perceived value that enables higher course completion rates (most today have less than 10% of registered students actually completing the course); and
- Authenticating students in a manner to satisfy accrediting institutions or hiring companies that the student identify is actually known.
Given this short timeline and the nature of investment-backed educational experiments, I think the real focus should be on whether and how MOOCs or successor models build on current scalability and openness while overcoming these four barriers.

What have we seen since 2012?
- Revenue Models: Coursera, FutureLearn, and edX moving towards an OPM business model, and Udacity focusing on corporate education;
- Credentialing: All MOOCs offering some sort of verified certificates, and in the OPM cases offering actual degrees through their partner institutions;
- Course Completion: MOOCs realizing that the two issues above lead to higher completion rates; and
- Authentication: Verified certificates and OPM models requiring student authentication through webcams and approaches similar to online proctoring companies, and even partnering with proctoring companies.
At this stage pretty much everyone recognizes a blatant ‘I told you so’ post written while Michael is on vacation and unable to talk me out of it, so I’ll move along and cut off further commentary.
June 14, 2018
UF Online’s New Corporate Partner: Discover Financial joins Walmart with Online Education benefit

Last week Walmart announced a new education benefit for its employers, subsidizing undergraduate college degrees at select online programs at the University of Florida (through UF Online), Brandman University, or Bellevue University.

The company said it estimates as many as 68,000 of its employees initially could sign up for the new college program. Walmart employs 1.5 million in the U.S. The company declined to comment on how much this initiative will cost it.

Employees will contribute $1 per day, for 365 days every year, toward their education, so long as they’re enrolled. Walmart will cover the rest of the tab. Workers will be able to choose from the three nonprofit schools and have the option of taking classes online with the flexibility to study during the evenings or on weekends.

Today UF Online announced another partner: Discover Financial Services, issuers of the Discover Card and Diners Club International, with its 14,000 employees.

Discover announced a significant new education assistance benefit that provides all eligible U.S.-based employees the opportunity to earn a bachelor’s degree online from an accredited public or non-profit private university at no cost.

Known as The Discover College Commitment, the benefit covers tuition and required fees, books and supplies needed to complete select online degrees at one of three schools – the University of Florida (via UF Online), Wilmington University and Brandman University. The benefit has no tenure requirement so employees can start participating as soon as they want regardless of how long they have been with the company, including new employees on their first day.

The two partnerships are part of what UF Online calls their Employer Pathways, and in the process seem to be defining an alternate method of marketing and enrollment management, in contrast with most Online Program Management (OPM) assumptions.

Employees from Walmart and Discover Financial Services have expanded opportunities to earn a bachelor’s degree from the University of Florida (UF) thanks to a new and significant investment in their education from their employers. This remarkable step by these and other large employers demonstrates the value of a UF degree and the accessibility of UF Online, UF’s online undergraduate experience. Employers will provide a robust support package covering tuition, fees, plus access to life coaching and college readiness programs.

Discover is offering this new benefit to select programs at UF Online, Wilmington University, and Brandman University.

Discover is covering 100% of cost for select bachelor’s degrees. Discover is proud to offer “The Discover College Commitment” program – an innovative full-ride college education benefit, providing US-based employees the opportunity to participate in one of several select high-quality, fully paid, online bachelor’s degrees.

Both packages of education benefits is managed by Guild Education, “a tuition reimbursement and education platform that helps large employers extend education benefits, including tuition reimbursement, to workers”.

I think these moves are more significant than just individual benefits. What we are seeing is UF Online, along with a handful of others, defining a different approach to marketing and finding potential online students, at least for undergraduate degrees. Historically, there is a common assumption that to enable a scalable online program there is a need for traditional digital marketing as the primary approach – Google AdWords, call centers, social media campaigns – with a partnership or two thrown in on the side. The origin of the OPM market is centered on providing these services in exchange for a percentage of tuition revenue, and for the majority of cases, the OPM’s spending on this marketing and enrollment management category is the most expensive service in the package. The Employer Pathways approach by UF Online has the potential to flip the student acquisition assumptions – primarily driven by employer partnerships with traditional digital marketing channels as a secondary approach.

Walmart and Discover already have potential students identified, and the companies have the incentive to internally market these education benefits to employees. UF Online needs to support that internal marketing and feed applicants into the same application process, but they have a reduced need for general-purpose marketing. UF Online started out with Pearson Online Learning Services (formerly known as Embanet) to be their OPM partner, but in late 2015 they pushed Pearson out and decided to take over this management internally.

I talked to Evangeline (Evie) Cummings, Associate Provost and Director of UF Online, at WCET’s Leadership Summit last week, and she described how UF Online is trying to reduce their marketing spend and instead invest more in faculty and course development. Part of that move is to spend less trying to get out-of-state students, which may pay higher tuition but still cost much more per student in acquisition costs. But the other move is to decrease the need to spend on traditional marketing channels.

We can think of Guild Education in this case as a derivative of the unbundling version of OPM, enabling the front-end services with an alternative approach and what appears to be a lower revenue share.

Rather than charge a transaction fee per student to the employer, Guild takes a cut of the tuition revenue from the universities it works with. This revenue-share model is an “elegant” solution for institutions that want to grow their enrollment online but don’t want to spend more on marketing, said Carlson. It’s also an attractive proposition for employers, who don’t have to pay any additional charges on top of the contribution they make to their employees’ tuition. The tuition fees are not discounted for the employers and will be charged at in-state or out-of-state rates depending on the location of the student. Neither Guild nor the three universities involved in the Walmart offer would disclose what percentage of tuition revenue Guild will take.

The bulk of the OPM market serves master’s programs, and UF Online is centered on bachelor’s degrees with a handful of master’s, so this approach might not work across the board. But I’ll bet that there will be plenty of undergraduate online initiatives that will be looking to UF Online and wondering if they should develop similar degree pathway partnerships as the centerpiece of their student acquisition plans. The online education space is maturing and becoming a lot more interesting.

June 12, 2018
Top Hat’s OER Announcement: Doubling down on faculty engagement
Several months ago I wrote a post looking at the Top Hat’s push into digital curricular materials through their Textbook product and Marketplace for digital course content. Leading up to that post, I had been planning to cover the Open Educational Resources (OER) angle, as the Marketplace included a number of openly-licensed material, much of it from OpenStax, and Top Hat had already begun marketing itself as an OER provider. At the time, the OER strategy seemed a work in progress. In fact, I found that some of my questions for company staff about OER basics – the role of Creative Commons licenses, community dynamics exhibited at the OpenEd conference, etc – led to a lack of answers, and at the time there was no export capability to get OER out of the platform.

To Fee or Not to Fee

The situation has changed since January, and with last month’s announcement of Top Hat’s Open Content Initiative the company is taking a stand on whether it is appropriate to charge for platform access. The idea of hosting and modifying OER on a fee-based platform became a big topic last year. Lumen Learning pioneered the Red Hat type model in 2014, and last year there was a big movement with Cengage, Knewton, OpenStax, Macmillan, and Top Hat all offering OER within their platforms. In many cases, the OER content itself has been redesigned from traditional textbook-in-PDF format to learning objective-driven content with aligned assessments. ((Disclosure: Lumen is a client of MindWires, and I recently gave a paid keynote at a Top Hat user’s conference.))

Top Hat has now removed the student platform fees and added an export-to-epub feature. As evidenced in a company blog post by CEO Mike Silagadze, they are not shy about it either.

At Top Hat, we’ve been working on making education more effective and affordable since 2009. Now, we’re happy to make a move that delivers on both fronts. Beginning April 12, with the launch of our Open Content Initiative, we’re offering completely free access to thousands of textbooks and other Open Educational Resources (OER)—freely accessible and openly licensed learning materials—on the Top Hat Marketplace.

It’s about time. Students have been forced to weigh the pros and cons of emptying their wallets and draining their financial aid to buy textbooks for far too long. [snip] Just as bad, digital publishing platforms and e-readers have been charging a toll to students—disguised as a platform fee—to access free, openly licensed OER.

Enough is enough.

I asked Silagadze about a point I noted when describing Cengage’s OpenNow product:

For each course [VP of Content Strategy] Constantini estimates that the modifications take $50k – $100k of internal work, including verifying of licenses for embedded elements. I would note a certain irony here in that OpenStax produces more-or-less traditional digital textbooks requiring publishers or OER services companies like Lumen to break apart and realign to competencies or outcomes.

Silagadze brushed off this description and stated that the modifications made by other providers were far smaller and easier to make than is being claimed. We now have a third variation in the OER market, with the provision of wrap-around platform and a clear argument that these platforms not only will be free on Top Hat, but that they should be free as a matter of principle.
- Free content, not dependent on specific platform
- Free and modified content, available on a paid platform, content available for export
- Free and lightly modified content, available on a free platform, content available for export
While this is a marketing position by one of the competitors in a new field, this move by Top Hat is a further sign of the OER movement breaking into different branches. From our perspective, the fractures in the OER community have been widening for the past several years, but to a degree this is a sign of success. Openly-licensed content usage is becoming more and more common in education, and even traditional publishers mostly accept the value of OER.

Faculty Engagement

Perhaps more significantly, at least in terms of understanding Top Hat as a company, is that the OER initiative doubles down on their bet on faculty engagement. A well-known issue with OER (and even with non-open content) is that few faculty end up taking advantage when given the ability to modify the course materials in any significant manner. In theory many people talk about open pedagogy in terms of faculty modification and collaboration on content, but in practice this rarely happens. Top Hat’s view is that the barrier has been flat content and cumbersome platforms, as best described in an eLearning Inside interview.

“We think the promise of OER has fallen down,” said Nina Bilimoria Angelo, VP of product and customer marketing at Top Hat. The Toronto-based company has created a platform to house, customize, and share OER and other educational resources. It has been used by over 2.8 million students to date.

“The promise was there’s a community that continues to build on open materials,” Angelo said. “But when those materials are trapped behind static PDFs, and then people are making changes to it on their own without a mechanism to share it back, that’s where things fall flat. We really wanted to create a system where things can be improved in real time, not over a 3 or 4 year cycle like with traditional publishers.”

“Discoverability is a challenge with OER,” Angelo said. “Quality can be perceived as uneven which is probably why OER adoption has stalled at the 5-10% level for instructors. There’s a lot of skepticism amongst higher educators. We’re trying to make sure all the high quality material is available in the Marketplace. Once it’s adopted, it’s really customizable. But then those customizations – this is the magic – those customizations can be shared back with the author and the team so that they can improve upon what they’ve created.”

This is the best way to interpret Top Hat’s OER move, in my opinion, and you can see more details in my January post about the Marketplace to better understand the customization and sharing capabilities of the platform.

According to an internal Top Hat survey of users, 89% of adopters make changes to digital textbooks that they adopt, with 22% reporting “lots of customization”. If this internal data is representative, there may be some indicators that faculty can be more involved in modifying and sharing content. Top Hat is betting on faculty engaging with content, modifying it, sharing it, updating it. And they are betting that this model will drive faculty adoption decisions.

There are a lot of unknowns about faculty adoption and modification of OER content through the Top Hat Open Content Initiative, but it is clear that the company is positioning itself differently than other providers. The transformation of digital curricular materials continues.
May 20, 2018
Postscript on Rio Salado Coverage: Clarity about different outcome types

After my two posts two weeks ago calling into question whether Rio Salado College has demonstrated clear student outcomes that justify its usage as an exemplar institution, college officials have responded through the WCET blog and in an article from Paul Fain at Inside Higher Ed. I recommend reading the IHE article in particular to get an understanding of the challenging an nuanced question of what is “good enough” in terms of college completion, and for even more detail read Rio’s written response. Unfortunately, the lengthy explanations on graduation rates from Rio end up obscuring a critical distinction between awarding degrees versus certificates of completion. The reason this matters is that Rio appears to be doing a better than average job supporting student transfers to four-year degree programs, a worse than average job of helping students complete two-year degrees, and an unknown job of providing students with non-degree stand-alone certificates.

Before dealing with this issue, I’d like to point out how useful this dialog has been and how open Rio has been in sharing internal metrics. One example that I wish other institutions used, and that can help us clarify how to understand completion data, is based on student goals.

Fully 76% of Rio students have no intention to get a degree or certificate, and of this group 33% (11,628) indicated earning transferable credit as the key reason to enroll. As Sally Johnstone said in the IHE article, “It’s a feeder school for Arizona State University”. This transfer of credit mission can be good or bad – it depends on how well these students perform in getting an ASU degree. In the written response, but not in WCET or IHE articles, Rio shares an interesting but incomplete metric.

Additionally, as part of the work RSC does related to accountability, we track the success of our transfer students to the three public universities in Arizona (The 2010/11 – 2015/16 5 year trend shows a 77.10% increase in bachelor degrees awarded to students that had completed a minimum of 12 credit hours (up to 60 or more hours) at Rio Salado College).

Increase to what and over what? It is not clear if this is comparing to Arizona university students who do not transfer any credits from Rio, or those who transfer less than 12, or those who transfer from other colleges. Nevertheless, this is an indicator of the college helping students obtain four-year degrees through credit transfers.

Once you get beyond non-degree / non-certificate seeking students, the numbers are murkier, and this is the area that needs clarity. One out of four Rio students are there to get an award – how successful are they? Rio recommends we look at the Voluntary Framework for Accountability (VFA) measures, instead of IPEDS, with the following results.

• 2-year graduation rate is higher than peer institutions: Rio 13%; Peer Institutions 10%
• 6-year graduation rate for credential-seeking cohort is 36.2%
• The credential seeking cohort far outperformed sister institutions in completion: Rio 42%; Peer Institutions 16%

I do not know why Rio uses the term graduation rate, as VFA instead describes Percent Students Completed a Formal Award, but from the numbers used we can see where the data comes from. The first and third bullets come from the VFA Two-Year Progress measures (blue column under Completed). ((The VFA definition of the cohorts: Main Cohort = fall entering, first time at reporting college; “all students”; Credential Seeking = earned 12 credits by end of year two.))

What is key is that the Completed metric includes those who obtained an associate’s degree or official certificate. Unlike degrees, certificates of completion have no standards across schools, and their usage is very uneven. As an example of Rio’s usage:

Notably, since 2010 Rio has defined and awarded more and more of these <1 year certificates, based on IPEDS data.

We can see is that the large majority of completions, which Rio labels under graduation rate, comes from the nonstandard certificates rather than degrees.

One of the strengths of VFA is that under the Six-Year Outcomes measures, you can separate out the various types of award. Below is the data for Rio – pay attention to the Credential-Seeking Cohort, as it removes the majority of non-degree / non-certificate seeking students at Rio only looking to transfer a small number of credits. ((For degree or certificate-seeking cohorts, I prefer the Rio definition based on student self-reporting, but the VFA definitions get at the same idea.)) Also note that VFA outcomes are mutually-exclusive and hierarchical in the order shown. If a student gets a certificate and a degree, they are listed under the degree outcome.

For the Credential-Seeking Cohort, 22% of Rio students get a certificate and do not transfer to a four-year school, 7% get a certificate and transfer, 3% get an associate’s degree and do not transfer, and 4% get an associate’s degree and transfer. A large majority of Rio Salado completions are for certificates, which is consistent with the IPEDS data.

How does Rio compare to other large community colleges? While we have not analyzed all 200+ colleges in VFA, spot checking with some peer schools indicates that Rio is well below others in awarding degrees but well above others in awarding certificates.

The resulting data provide the basis of my comment in IHE:

“This is not a bad situation, per se, as long as students are gaining value in the workplace for these official certificates,” he said via email. “But certificates are not useful in terms of comparing apples to apples, especially when one school uses them liberally, as does Rio, and most other schools do not.”

Hill also noted that Rio Salado’s VFA completion rate for associate degree programs is a “troublesome” 7 percent.

While I appreciate the valuable sharing of information from Rio Salado in this process, I’ll stick with my original conclusion:

At best, this is a school with mixed results that should not simply be labeled a success without caveats or explanations.

Rio has shown that they can cut costs and still be a successful transfer school, but not that they can be a successful degree school. This matters a great deal in terms of how transferrable the lessons are. It’s fine to focus on transfers when you’re right next door to ASU. But what if you’re Adirondack Community College? There are places where degree completion matter more or less than at Rio. And what about those students going to Rio who *are* seeking degrees? Does this model promote some educational outcomes at the expense of others? And if so, shouldn’t students know about that before enrolling?

May 10, 2018
Rio Salado College As Exemplar: A critical external view
In yesterday’s post I described how a review of two courses at Rio Salado College indicated reasons to question the use of this school in the ASU / BCG case study report on Digital Learning ((Disclosure: Our e-Literate TV series was funded in part by the Bill & Melinda Gates Foundation who also funded the ASU / BCG study.)). The report, titled “Making Digital Learning Work: Success Strategies From Six Leading Universities and Community Colleges”, has the following description [emphasis added]:

How can the use of digital technologies in postsecondary education impact students’ access to education, student outcomes, and the return on investment for students and institutions? What are the biggest challenges for an institution seeking to implement high-quality digital learning opportunities? What promising practices enable an institution to achieve impact at a larger scale? [snip]

The answers, at least in part, lie in case studies of six colleges and universities: Arizona State University, the University of Central Florida, Georgia State University, Houston Community College, Kentucky Community and Technical College System, and Rio Salado Community College. The first three institutions in this list are public research universities, representing different geographic populations and access missions. The other three institutions include two community colleges and a state-wide community college system.

These six institutions have a strong track record of using digital learning to serve large, socioeconomically diverse student populations, and each has been a pioneer in innovating to expand access to postsecondary education, improve student outcomes, and provide higher education at an affordable cost.

Let’s look at the track record of Rio Salado College in terms of aggregate academic student outcomes to see how appropriate it is to include them as an exemplar in such a case study-based report. For much of my analysis, I looked at the 862 public 2-year colleges fully reporting data in the Fall 2016 IPEDS data set.

Reduced Expenditures

The primary claim made in the ASU / BCG report is that Rio Salado is has reduced costs.

This claim does hold up to scrutiny, as Rio’s $89 of instructional delivery costs per student credit hour and $16 of student services costs are 10th and 9th lowest in the country for the 862 colleges I reviewed.

The primary outcomes claims made in the case study quickly brush off the standard metrics for Rio.

Rio Salado’s success is not defined solely by improvements in graduation and retention rates. In particular, the college has a high transfer-out rate (32% compared to an average of 19% for other MCCCD colleges [ed. – the other schools in the Maricopa County Community College District]), and the students who transfer to Arizona universities from Rio Salado have a 74% four-year graduation rate—3 percentage points higher than the average for other MCCCD transfer students. At Rio Salado itself, students’ course-level success rates have slowly been improving over time, to about 64% in 2016.

Graduation and Retention Rates

It is all well and good to point out the importance of transfer students, but we should not ignore graduation and retention rates. And we should explore whether the data supports the phrase “improvements in graduation and retention rates”.

The report describes Rio’s investment in a suite of advising tools.

Obtaining these advisory tools required an initial investment of $1 million, but the tools pay for themselves by increasing Rio Salado’s term-to-term retention by 7%, increasing ROI through improved student progress and the additional tuition dollars that the institution receives from retained students.

Looking at IPEDS data, we can see Year 1 to Year 2 retention rates (not quite the same as term-to-term), and it shows improvement since 2013. Unfortunately, Rio’s retention rates for full-time and part-time students are both in the bottom 10% of all community colleges at 33% and 27%, respectively. And the full-time rate is lower than it was prior to 2011.

While the ASU / BCG report does not mention graduation rates directly, there have been several claims made about Rio’s numbers in other publications.
- American Enterprise Institute, 2012: “Rio Salado boasts a 42 percent graduation rate, about twice the national average for community colleges.”
- The Hechinger Report, 2014: “Thanks to a barrage of support, however, Rio Salado boasts a graduation rate four times that of comparable schools, U.S. Department of Education figures show.”
- Community College Review, 2017: “The greatest success seen at Rio Salado thus far is their incredibly high graduation rate, which currently stands at four times that of comparable institutions!”
- Bill & Melinda Gates Foundation, 2017: “Rio Salado’s two-year graduation rate—that is, the percentage of students who finish an associate’s degree in the normal two years—is four times higher than the average for 100 peer institutions.”
The problem is that Rio Salado’s 4-year graduation rate for first-time full-time students as reported in IPEDS is 5% – the second lowest of any public 2-year college in the nation. How can we resolve this discrepancy?

The key to understanding the claims in these four articles is to follow the two links, which both point to a customized IPEDS Feedback Report from 2013 for Rio against their selection of 100 peer institutions.

This measure captures any award – degree or certificate – for a small cohort of entering full-time students at Rio from 2009. If you trace that data through IPEDS, you see that this measure is for a 150-student cohort (in 2012) out of the 45,000+ students at Rio.

Using this specific metric, you can find the 42% graduation rate in 2011, and the “four times greater” rate of 27% in 2012, both as the rate plummets from 64% in 2008 to 3% in 2016.

Using broader, up-to-date metrics for graduation and retention rates as well as the new IPEDS outcomes measures (which allows comparison outside of just full-time first-time cohorts), we see that Rio Salado College has some of the lowest student outcomes measures in the country.

For all but two measures (6-year First-Time Full-Time Awards and 6-year Non-First-Time Full-Time Awards), the performance is in the bottom 10% of all colleges in the study. For those two awards measures, the performance is in the bottom half of the country and represents just 11% of the student body.

Transfer Rates

What we are left with are reports about high transfer rates and subsequent success rates 3% higher than other transfers out of the same district. The IPEDS data set now includes transfer rates in its new Outcomes Measures section, and it is true that Rio Salado College does transfer out a significant portion of students. For all cohorts defined below, Rio is in top 20% of public 2-year institutions.
- First-Time Full-Time Students: 39% transfer rate, Rank 170 out of 862
- First-Time Part-Time Students: 47% transfer rate, Rank 112 out of 862
- Non-First-Time Full-Time Students: 46% transfer rate, Rank 196 out of 862
- Non-First-Time Part-Time Students: 59% transfer rate, Rank 121 out of 862
Note that this data indicates that students transferred to another institution and are still enrolled there. This data does not indicate what portion of the transfers were planned versus students deciding to move on for other reasons. Nevertheless, for a community college, especially one with very close ties to ASU (I have been told that a large percentage of Rio students are ASU students trying to fill out their schedule based on saving money and more convenient schedules), this is a favorable metric.

I cannot provide independent confirmation on the subsequent success rates, so we’ll accept that claim at face value.

Case Study Questions

I should note that Rio Salado College does provide accurate information on the metrics mentioned in this post, both through IPEDS reporting and through their Research & Planning section of the web site. The picture that emerges from the aggregate outcomes data is of a large school that:
- transfers out greater than average percentage of students;
- is just below average for awarding certificates for full-time students;
- is among the poorest performing in the country for retaining students;
- is among the poorest performing in the country for awarding degrees; and
- is among the poorest performing in the country for awarding certificates for part-time students.
At best, this is a school with mixed results that should not simply be labeled a success without caveats or explanations.

One question we should ask is whether it is appropriate to hold up a school with some of the lowest student outcomes measures in the country as an exemplar. Yes, Rio Salado has found a way to spend as little as possible on instruction and student support services, and yes, a lot of students transfer out, but that is not enough. We need greater evidence of student success if we are to use them as a case study for others to emulate. And we also need a more robust genre of a case study that looks across, at a minimum, a relatively standard set of publicly-available information, and deeper dives where appropriate, to understand educational practices and their impacts on students. These case studies should present information in context – the good and the bad – since education is complex and challenging, and a clear focus on evidence will benefit all parties in the end.
April 23, 2018
Rio Salado College As Exemplar: A critical internal view

10 days ago Arizona State University (ASU) and the Boston Consulting Group (BCG) released a report, supported by the Bill & Melinda Gates Foundation (BMGF) ((Disclosure: Our e-Literate TV series was funded in part by the Bill & Melinda Gates Foundation.)), titled “Making Digital Learning Work: Success Strategies From Six Leading Universities and Community Colleges”. The basic idea [emphasis added]:

How can the use of digital technologies in postsecondary education impact students’ access to education, student outcomes, and the return on investment for students and institutions? What are the biggest challenges for an institution seeking to implement high-quality digital learning opportunities? What promising practices enable an institution to achieve impact at a larger scale? [snip]

The answers, at least in part, lie in case studies of six colleges and universities: Arizona State University, the University of Central Florida, Georgia State University, Houston Community College, Kentucky Community and Technical College System, and Rio Salado Community College. The first three institutions in this list are public research universities, representing different geographic populations and access missions. The other three institutions include two community colleges and a state-wide community college system.

These six institutions have a strong track record of using digital learning to serve large, socioeconomically diverse student populations, and each has been a pioneer in innovating to expand access to postsecondary education, improve student outcomes, and provide higher education at an affordable cost.

The methodology is a case study of each school, and the report then takes a stance on what other institutions should do.

Now is the time for leaders to champion the potential of digital learning to open the doors of higher education wider and to improve student outcomes, while operating more efficiently and at lower cost. The journey of each college or university will be unique, but the set of promising practices described in this report may serve as a useful guide for all institutions.

There are multiple methods to evaluating a school; it turns out that some are more meaningful than others, and it is always helpful to start with what students experience in actual courses. The ASU / BCG report provides useful context on course design.

At Rio Salado, 22 full-time faculty chairs develop courses with the support of a central team that includes subject-matter experts, instructional designers, media support staff, and production staff. About 1,500 adjunct faculty members teach the courses, which they can personalize by adding an introductory message or video for each module.

It turns out that I have access to two courses – ENG102 (First Year Composition) and EED200 (Foundations of Early Childhood Education) – and this gives an opportunity to better understand the student experience. What these courses indicate, however, is a troubling lack of meaningful interaction between faculty and students.

Looking at ENG102, the course materials is Rio Salado developed courseware embedded in their custom course management system, RioLearn. The materials appear to be quite extensive, and all materials are available at the start of the term in 14 lessons. To find actual due dates, students have to go to the gradebook, as no specific dates are included in the courseware.

The instructor sent out several announcements at the beginning of class, a welcome letter, and then one reminder or update message at the beginning of each month. That’s it for instructor-initiated interactions.

There were no student-to-student discussions, as the discussion board was unused for the course. There was one peer-review activity, but otherwise no interactions between students. I have been a frequent critic of threaded discussion boards, but it is certainly better to have something instead of nothing.

The rest of the interactions came in two categories: grading of assignments and responses to student-initiated messages. The primary feedback method in these two courses was faculty usage of the custom-developed Feedback Tool, which uses rubrics to grade assignments.

There were several assignments that included annotated mark-ups of the submitted papers, which appear to be the most useful feedback from instructors.

A review of EED200 shows the same course structure – few instructor-initiated interactions, use of rubric grading as feedback on assignments, and specific responses when students send questions in through the Message Center.

For both courses, the instructors typically responded to student messages within a day or two.

This approach is troubling, as both courses appear to not meet the “regular and substantive interaction” regulation for credit-bearing online courses. I have been critical about the vague standards and the egregious application by the Department of Education’s Office of Inspector General audit of Western Governors University, but these complaints do not mean that the regulation has no point. The idea is that course design and facilitation should be implemented to ensure that students are not left to figure out static course materials and to be responsible for initiating most forms of interaction. As stated in a 2014 Dear Colleague Letter on the subject of regular and substantive interaction based on competency-based education (CBE) approaches [emphasis added]:

We do not consider interaction that is wholly optional or initiated primarily by the student to be regular and substantive interaction between students and instructors. Interaction that occurs only upon the request of the student (either electronically or otherwise) would not be considered regular and substantive interaction.

Some institutions design their CBE programs using a faculty model where no single faculty member is responsible for all aspects of a given course or competency. In these models, different instructors might perform different roles: for example, some working with students to develop and implement an academic action plan, others evaluating assessments and providing substantive feedback (merely grading a test or paper would not be substantive interaction), and still others responding to content questions.

The problem with case studies is that the selection of cases to study may not be representative or appropriate to prove one’s thesis. It is possible that I happened to look in detail at two courses that are aberrations. But even so, this view is troubling, as these are centrally-developed courses (not subject to the whim of individual instructors), and both courses have little meaningful provisions for faculty-student or student-student interactions. The instructors were responsive to messages, but that is not enough to back up claims of high-quality courses. The dissonance between these two courses and what is described in the ASU / BCG report led me to take a critical look at the data.

For the next post, I’ll take the external view and look at aggregate public data on Rio Salado College on academic outcomes.

April 22, 2018