Outcomes-Based Education and the Conservative Radicalism of the AAC&U

I have been invited to participate on the Digital Working Group of the Association of American Colleges & Universities (AAC&U’s) General Education Maps and Markers (GEMs) program. (As we will see, AAC&U loves its acronyms.) GEMs is a really interesting project made even more interesting because of who is doing it. AAC&U is a fundamentally conservative organization in the sense that their purview is to help conserve that which is valuable and distinctive about the American college education. GEMs is part of a larger alphabet soup of AAC&U initiatives that are clearly about outcomes-based education and arguably about competency-based education (CBE). (For more on the distinction between these two, see Phil’s earlier blog post on the subject.) The organization’s leadership is frank about the fact that these have been at least partly defensive moves; given all the pressure for increasing accountability, the Association felt that academics should lead the charge on defining what that should mean before others define it for them. (GEMs is the latest in a series of related efforts to that end.) But the trail they are blazing has the potential to be pretty radical. Maybe even more radical than the organization has fully come to grips with.

I’m going to be doing some of my homework for the working group here on the blog in future posts, but before I do that, I’d like to take the time in this one to outline what I understand to be the goal and context of GEMs. Fair warning: I’m not sure that my understanding is correct. I am new to AAC&U and, to be honest, they were not terribly good at articulating the goals for GEMs at the first summit. (I am not the only participant who had trouble parsing the goals and priorities.) With any luck, some AAC&U veterans will read this and correct any mistakes I make along the way.

Pre-History

In the beginning, there was LEAP (Liberal Education & America’s Promise).1 Started in 2005, it is the precursor to the other programs I will be talking about. I’m not going to spend a lot of time parsing that pre-history (either for you or for myself), but I find this definition from the vision paper to be emblematic of all these related efforts:

Liberal Education: An approach to college learning that seeks to empower individuals and prepare them to deal with complexity, diversity, and change. This approach emphasizes broad knowledge of the wider world (e.g., science, culture, and society) as well as in-depth achievement in at least one specific field of study. It helps students develop a sense of social responsibility, strong cross-disciplinary intellectual and practical skills (e.g., communication, analytical and problem-solving skills), and a demonstrated ability to apply knowledge and skills in real-world settings.2

This, I think, is the heart of the matter. This is what they view themselves as defending. In an important sense, LEAP, GEMs, and the other initiatives I will write about here are all efforts to adapt the means of achieving a liberal education to the changing times in order to preserve the core values.

Several initiatives flowed out of LEAP, the first of which is the Degree Qualifications Profile (DQP). On the one hand, DQP appears to conform to the principles of Competency-Based Education that SPT Malan articulated in his article describing the CBE (and referenced in the aforementioned post by Phil):

• Explicit learning outcomes with respect to the required skills and concomitant proficiency (standards for assessment)
• A flexible time frame to master these skills
• A variety of  instructional activities to facilitate learning
• Criterion-referenced testing of the required outcomes
• Certification based on demonstrated learning outcomes
• Adaptable programs to ensure optimum learner guidance

Some of these points are fairly radical relative to the traditional structure of an undergraduate liberal arts degree, and I want to be clear that AAC&U appears to embrace these radicalisms. For example, they approvingly distributed a copy of Paul LeBlanc’s Inside Higher Ed essay advocating for CBE, including an explicit call for “a fundamental change at the core of our higher education ‘system’: making learning non-negotiable and the claims for learning clear while making time variable” [emphasis in original]. The two areas where they appear to want to push back against the current popular interpretations of CBE are (1) they have a more expansive notion of the educational goals of CBE which includes moral education and a grounding in academic disciplinarity, and (2) they advocate for rich and authentic (and, implicitly, often human-graded) assessment. They also view the core value proposition for students somewhat differently. The typical reason people embrace CBE is to support educational access. The idea is that students should be able to test out of competencies that they already know (or can learn through non-tuition-costing experiences) and thus reduce the time and expense of a college degree. While there is nothing I’ve seen in the AAC&U documents that precludes this notion, their emphasis is on educational quality. One of the uses of DQP is as a kind of nutritional labeling for a degree program. DQP focuses on five dimensions:  applied learning, civic learning, intellectual skills, broad integrative knowledge, and specialized knowledge:

This graph can be used as the basis for defining the distinctive characteristics of a particular school or degree program, like so:

The Oregon University System is doing these “spiderweb graphs” for programs across all of their colleges. Which brings me to another goal for DQP; namely, it functions as a tool around which campuses can build conversations regarding common educational goals. As I said in my post about Pearson’s efficacy framework, the magic of any rubric is in the norming conversation.

As you might imagine, these competencies are defined at a very high level in order to be useful across a broad range of institutions. For example, one goal (objective? competency?) under the heading of “Broad, Integrative Knowledge” is “Produces an investigative, creative or practical work that draws on specific theories, tools, and methods from at least two core fields of study.” This is a perfect example of the balancing act that the AAC&U is attempting to perform. On the one hand, they need to be specific enough that the criteria are meaningful and enable at least first-approximation comparisons of the similarities and differences among and between institutions. (AAC&U seems understandably cautious about embracing comparisons between institutions too whole-heartedly, but that’s really what the spiderweb graph above enables.) On the other hand, they need to provide enough flexibility in the framework to support the kind of faculty-, discipline- and academic-centric vision for higher education that they are fighting to preserve. (One of the unresolved and unspoken tensions in this work, which I’ll touch on again later, is that the AAC&U does support values like academic freedom that are near and dear to faculty but do not necessarily take a faculty-centric view of the world if you define “faculty-centric” from a labor perspective.) They need to provide enough specificity to enable campus communities to understand how to proceed yet enough flexibility to allow each campus to make the framework work within their own cultural and academic peculiarities.

VALUE and GEMs

Nowhere does this balancing act become more apparent than in assessment. It’s easy enough to create mom and apple pie goals that everybody can agree to, in part because everybody can pretend that they share the same definitions while traveling down divergent paths. The Valid Assessment of Learning in Undergraduate Education (VALUE) framework is therefore where the rubber meets the road in an important sense. VALUE is intended to be

a quality assurance framework to help institutions, systems, and accreditors determine to what degree students are making progress toward the expected competencies, and whether students meet graduation-level standards for demonstrated achievement.

At its heart, VALUE is a set of 16 rubrics across the following areas:

• Inquiry and analysis
• Critical thinking
• Creative thinking
• Written communication
• Oral communication
• Reading
• Quantitative literacy
• Information literacy
• Teamwork
• Problem solving

AAC&U claims that they “ensure a high level of reliability and validity of the results”, presumably based on inter-rater studies that they’ve done. (Like I said, the magic of the rubric is in the norming conversation.) But again, the organization works hard to strike a balance between specificity and flexibility. Here is an example of a criterion from their “Integrative Learning” rubric:

This is pretty specific without being totally rigid. You can see where this level of specificity is likely to get fairly high inter-rater agreement among colleagues at the same institution or a peer institution while still allowing for some differences of interpretation between, say, an Ivy League school and a community college.

Now, having defined a set of competencies and a set of rubrics for assessing those competencies, there is the slight problem that the overwhelming majority of schools currently implement neither, have the curricular and support structure for neither, and are missing the tools to get students from one to the other. That’s where General Education Maps and Markers (GEMs)3 comes in. It is intended to be the connective tissue between DQP and VALUE.

The heart of this work seems to be in the Design Working Group, which is charged with coming up with case studies and exemplar practices of how institutions can build gen ed programs that are compatible with DQP and VALUE. I don’t have a clear sense of what their final product will look like, other than a collection of examples, but maybe a collection of examples is a good enough place to start. Their work is supported by the Equity Working Group and the Digital Working Group.

Wanted: A Theory of Change

One of the questions asked by a colleague on the Digital Working Group, after hearing a few hours of background presentation the first day of the first meeting, was “What is your theory of change?” That’s my concern too. At first blush, I very much like what I see in DQP and VALUE. In fact, I think these frameworks collectively represent the beginnings of the robust yet realistic vision for the future of liberal arts education that I’ve been yearning for so dearly. But I would like them to be more than a vision. So far, AAC&U can only talk about a handful of pilots and experiments with them. I’m not sure how they and GEMs will move into broader adoption. There are two dangers here. The first is that they will end the way the vast majority of grant-funded, association-created academic frameworks end up, if we’re being honest. In a drawer somewhere.4 The other danger is that they will get uptake but will run into one or more of several fairly obvious buzz saws. For example, if we are serious about the notion that competencies are rigorously enforced but time to completion is variable, that has deep implications for the academic labor force. Who is talking to the unions about this? Or the faculty senates? Another danger is based in the fact that there is an enormous amount of money being invested into a quite different vision of CBE. There is huge potential for confusion and co-optation. If AAC&U is to be successful, they will need to engage with the broader academic community early and robustly.

I worry because my first impression of the organization is that it is fairly insular with a relatively uneven sense of what the world outside is like or how to interact with it. For example, one of the AAC&U leaders was absolutely flabbergasted that I had never heard of the National Survey of Student Engagement (NSSE) and that I wasn’t embarrassed to admit it. And of course, I undeniably am a remedial student in some ways, being new to both the AAC&U and some of the broader conversations and efforts that intersect with theirs. But I am hardly the only person to have never heard of NSSE. Nor does the organization seem to know how to help people like me become more aware of their world, what goes on in it, and why we should care about it. There are sparse stubs of articles on Wikipedia about NSSE and AAC&U itself and nothing at all on DQP, VALUE, or GEMs. The descriptions of DQP, VALUE, and GEMs on their own web sites were disjointed, and it is hard to figure out exactly how they are connected to each other (which is one reason to encourage crowdsourcing of descriptions through something like Wikipedia). There was no hashtag for the GEMs meeting and no awareness of how social media could have been an aid to the work going on in the room. In order for me to download the 16 VALUE rubrics, I had to go through a complicated and poorly designed ecommerce checkout process even though the rubrics are free. I could go on. Within the world of AAC&U participants, they seem to do a pretty good job of building (non-technological) social networks. But if you’re outside the tent, then you’re waaay outside the tent. And there is a lot of world outside that tent.

I want to be careful not to overgeneralize here; I met any number of highly aware and networked people at the GEMs meeting, particularly on the Digital Working Group. But as an institution, the AAC&U does not seem to be wired for the kind of outreach that it will need to do if DQP, VALUE, and GEMs are to have the impact that they should.

And so, with the blessings of my working group chair Randy Bass (who, by the way, is fantastic), I will be blogging my exemplar homework for the GEMs Digital Working Group, and I will be inviting you to submit your own exemplars, as well as any comments, concerns, criticisms, or kudos you might want to add. I hope that my GEMs colleagues will also join in. Let’s see if we can create an exemplar of our own.

Stay tuned.