e-Literate

Present is Prologue

Author: Michael Feldstein

  • ChatGPT: Post-ASU+GSV Reflections on Generative AI

    ChatGPT: Post-ASU+GSV Reflections on Generative AI

    The one question I heard over and over again in hallway conversations at ASU+GSV was “Do you think there will be a single presentation that doesn’t mention ChatGPT, Large Langauge Models (LLMs), and generative AI?”

    Nobody I met said “yes.” AI seemed to be the only thing anybody talked about.

    And yet the discourse sounded a little bit like GPT-2 trying to explain the uses, strengths, and limitations of GPT-5. It was filled with a lot of empty words, peppered in equal parts with occasional startling insights and ghastly hallucinations. 

    That lack of clarity is not a reflection of the conference or its attendees. Rather, it underscores the magnitude of the change that is only beginning. Generative AI is at least as revolutionary as the graphical user interface, the personal computer, the touch screen, or even the internet. Of course we don’t understand the ramifications yet.

    Still, lessons learned from GPT-2 enabled the creation of GPT-3 and so on. So today, I reflect on some of the lessons I am learning so far regarding generative AI, particularly in EdTech.

    Generative AI will destroy so we can create

    Most conversations on the topic of generative AI have the words “ChatGPT” and “obsolete” in the same sentence. “ChatGPT will make writing obsolete.” “ChatGPT will make programmers obsolete.” “ChatGPT will make education obsolete.” “ChatGPT will make thinking and humans obsolete.” While some of these predictions will be wrong, the common theme behind them is right. Generative AI is a commoditizing force. It is a tsunami of creative destruction.

    Consider the textbook industry. As long-time e-Literate readers know, I’ve been thinking a lot about how its story will end. Because of its unusual economic moats, it is one of the last media product categories to be decimated or disrupted by the internet. But those moats have been drained one by one. Its army of sales reps physically knocking on campus doors? Gone. The value of those expensive print production and distribution capabilities? Gone. Brand reputation? Long gone. 

    Just a few days ago, Cengage announced a $500 million cash infusion from its private equity owner:

    “This investment is a strong affirmation of our performance and strategy by an investor who has deep knowledge of our industry and a track record of value creation,” said Michael E. Hansen, CEO, Cengage Group. “By replacing debt with equity capital from Apollo Funds, we are meaningfully reducing outstanding debt giving us optionality to invest in our portfolio of growing businesses.”Cengage Group Announces $500 Million Investment From Apollo Funds (prnewswire.com)

    That’s PR-speak for “our private equity owners decided it would be better to give us yet another cash infusion than to let us go through yet another bankruptcy.”

    What will happen to this tottering industry when professors, perhaps with the help of on-campus learning designers, can use an LLM to spit out their own textbooks tuned to the way they teach? What will happen when the big online universities decide they want to produce their own content that’s aligned with their competencies and is tied to assessments that they can track and tune themselves? 

    Don’t be fooled by the LLM hallucination fear. The technology doesn’t need to (and shouldn’t) produce a perfect, finished draft with zero human supervision. It just needs to lower the work required from expert humans enough that producing a finished, student-safe curricular product will be worth the effort. 

    How hard would it be for LLM-powered individual authors to replace the textbook industry? A recent contest challenged AI researchers to develop systems that match human judgment in scoring free text short-answer questions. “The winners were identified based on the accuracy of automated scores compared to human agreement and lack of bias observed in their predictions.” Six entrants met the challenge. All six were built on LLMs. 

    This is a harder test than generating anything in a typical textbook or courseware product today. 

    The textbook industry has received ongoing investment from private equity because of its slow rate of decay. Publishers threw off enough cash that the slum lords who owned them could milk their thirty-year-old platforms, twenty-year-old textbook franchises, and $75 PDFs for cash. As the Cengage announcement shows, that model is already starting to break down. 

    How long will it take before generative AI causes what’s left of this industry to visibly and rapidly disintegrate? I predict 24 months at most. 

    EdTech, like many industries, is filled with old product categories and business models that are like blighted city blocks of condemned buildings. They need to be torn down before something better can be built in their place. We will get a better sense of the new models that will rise as we see old models fall. Generative AI is a wrecking ball.

    “Chat” is conversation

    I pay $20/month for a subscription to ChatGPT Plus. I don’t just play with it. I use it as a tool every day. And I don’t treat it like a magic information answer machine. If you want a better version of a search engine, use Microsoft Bing Chat. To get real value out of ChatGPT, you have to treat it less like an all-knowing Oracle and more like a colleague. It knows some things that you don’t and vice versa. It’s smart but can be wrong. If you disagree with it or don’t understand its reasoning, you can challenge it or ask follow-up questions. Within limits, it is capable of “rethinking” its answer. And it can participate in a sustained conversation that leads somewhere. 

    For example, I wanted to learn how to tune an LLM so that it can generate high-quality rubrics by training it on a set of human-created rubrics. The first piece I needed to learn is how LLMs are tuned. What kind of magic computer programming incantations do I need to get somebody to write for me?

    As it turns out, the answer is none, at least generally speaking. LLMs are tuned using plain English. You give it multiple pairs of input that a user might type into the text box and desired output from the machine. For example, suppose you want to tune the LLM to provide cooking recipes. Your tuning “program” might look something like this:

    • Input: How do I make scrambled eggs?
    • Output: [Recipe]

    Obviously, the recipe output example you give would have a number of structured components, like an ingredient list and steps for cooking. Given enough examples, the LLM begins to identify patterns. You teach it how to respond to a type of question or a request by showing it examples of good answers. 

    I know this because ChatGPT explained it to me. It also explained that the GPT-4 model can’t be tuned this way yet but other LLMs, including earlier versions of GPT, can. With a little more conversation, I was able to learn how LLMs are tuned, which ones are tunable, and that I might even have the “programming” skills necessary to tune one of these beasts myself. 

    It’s a thrilling discovery for me. For each rubric, I can write the input. I can describe the kind of evaluation I want, including the important details I want it to address. I, Michael Feldstein, am capable of writing half the “program” needed to tune the algorithm for one of the most advanced AI programs on the planet. 

    But the output I want, a rubric, is usually expressed as a table. LLMs speak English. They can create tables but have to express their meaning in English and then translate that meaning into table format. Much like I do. This is a funny sort of conundrum. Normally, I can express what I want in English but don’t know how to get it into another format. This time I have to figure out how to express what the table means in English sentences.

    I have a conversation with ChatGPT about how to do this. First I ask it about what the finished product would look like. It explains how to express a table in plain English, using a rubric as an example. 

    OK! That makes sense. Once it gives me the example, I get it. Since I am a human and understand my goal while ChatGPT is just a language model—as it likes to remind me—I can see ways to fine-tune what it’s given me. But it taught me the basic concept.

    Now how do I convert many rubric tables? I don’t want to manually write all those sentences to describe the table columns, rows, and cells. I happen to know that, if I can get the table in a spreadsheet (as opposed to a word-processing document), I can export it as a CSV. Maybe that would help. I ask ChatGPT, “Could a computer program create those sentences from a CSV export?” 

    “Why yes! As long as the table has headings for each column, a program could generate these sentences from a CSV.” 

    “Could you write a program for me that does this?” 

    “Why, yes! If you give me the headings, I can write a Python program for you.” 

    It warns me that a human computer programmer should check its work. It always says that. 

    In this particular case, the program is simple enough that I’m not sure I would need that help. It also tells me, when I ask, that it can write a program that would import my examples into the GPT-3 model in bulk. And it again warns me that a human programmer should check its work. 

    ChatGPT taught me how I can tune an LLM to generate rubrics. By myself. Later, we discussed how to test and further improve the model, depending on how many rubrics I have as examples. How good would its results be? I don’t know yet. But I want to find out. 

    Don’t you?

    LLMs won’t replace the need for all knowledge and skills

    Notice that I needed both knowledge and skills in order to get what I needed from ChatGPT. I needed to understand rubrics, what a good one looks like, and how to describe the purpose of one. I needed to think through the problem of the table format far enough that I could ask the right questions. And I had to clarify several aspects of the goal and the needs throughout the conversation in order to get the answers I wanted. ChatGPT’s usefulness is shaped and limited by my capabilities and limitations as its operator. 

    This dynamic became more apparent when I explored with ChatGPT how to generate a courseware module. While this task may sound straightforward, it has several kinds of complexity to it. First, well-designed courseware modules have many interrelated parts from a learning design perspective. Learning objectives are related to assessments and specific content. Within even as simple an assessment as a multiple-choice question (MCQ), there are many interrelated parts. There’s the “stem,” or the question. There are “distractors,” which are wrong answers. Each answer may have feedback that is written in a certain way to support a pedagogical purpose. Each question may also have several successive hints, each of which is written in a particular way to support a particular pedagogical purpose. Getting these relationships—these semantic relationships—right will result in more effective teaching content. It will also contain structure that supports better learning analytics. 

    Importantly, many of these pedagogical concepts will be useful for generating a variety of different learning experiences. The relationships I’m trying to teach the LLM happen to come from courseware. But many of these learning design elements are necessary to design simulations and other types of learning experiences too. I’m not just teaching the LLM about courseware. I’m teaching it about teaching. 

    Anyway, feeding whole modules into an LLM as output examples wouldn’t guarantee that the software would catch all of these subtleties and relationships. ChatGPT didn’t know about some of the complexities involved in the task I want to accomplish. I had to explain them to it. Once it “understood,” we were able to have a conversation about the problem. Together, we came up with three different ways to slice and dice content examples into input-output pairs. In order to train the system to catch as many of the relationships and subtleties as possible, it would be best to feed the same content to the LLM all three ways.

    Most publicly available courseware modules are not consistently and explicitly designed in ways that would make this kind of slicing and dicing easy (or even possible). Luckily, I happen to know where can get my hands on some high-quality modules that are marked up in XML. Since I know just a little bit about XML and how these modules use it, I was able to have a conversation with ChatGPT about which XML to strip out, the pros and cons of converting the rest into English versus leaving them as XML, how to use the XML Document Type Definition (DTD) to teach the software about some of the explicit and implicit relationships among the module parts, and how to write the software that would do the work of converting the modules into input-output pairs. 

    By the end of the exploratory chat, it was clear that the work I want to accomplish requires more software programming skill than I have, even with ChatGPT’s help. But now I can estimate how much time I need from a programmer. I also know the level of skill the programmer needs. So I can estimate the cost of getting the work done. 

    To get this result, I had to draw on considerable prior knowledge. More importantly, I had to draw on significant language and critical thinking skills. 

    Anyone who ever said that a philosophy degree like mine isn’t practical can eat my dust. Socrates was a prompt engineer. Most Western philosophers engage in some form of chain-of-thought prompting as a way of structuring their arguments. 

    Skills and knowledge aren’t dead. Writing and thinking skills most certainly aren’t. Far from it. If you doubt me, ask ChatGPT, “How might teaching students about Socrates’ philosophy and method help them learn to become better prompt engineers?” See what it has to say. 

    (For this question, I used the GPT-4 setting that’s available on ChatGPT Plus.)

    Assessments aren’t dead either

    Think about how either of the projects I described above could be scaffolded as a project-based learning assignment. Students could have access to the same tools I had: an LLM like ChatGPT and an LLM-enhanced search tool like Bing Chat. The catch is that they’d have to use the ones provided for them by the school. In other words, they’d have to show their work. If you add a discussion forum and a few relevant tutorials around it, you’d have a really interesting learning experience. 

    This could work for writing too. My next personal project with ChatGPT is to turn an analysis paper I wrote for a client into a white paper (with their blessing, of course). I’ve already done the hard work. The analysis is mine. The argument structure and language style are mine. But I’ve been struggling with writer’s block. I’m going to try using ChatGPT to help me restructure it into the format I want and add some context for an external audience.

    Remember my earlier point about generative AI being a commoditizing force? It will absolutely commoditize generic writing. I’m OK with that, just as I’m OK with students using calculators in math and physics once they understand the math that the calculator is performing for them. 

    Students need to learn how to write generic prose for a simple reason. If they want to express themselves in extraordinary ways, whether through clever prompt engineering or beautiful art, they need to understand mechanics. The basics of generic writing are building blocks. The more subtle mechanics are part of the value that human writers can add to avoid being commoditized by generative AI. The differences between a comma, a semicolon, and an em-dash in expression are the kinds of fine-grained choices that expressive writers make. As are long sentences versus short ones, decisions about when and how often to use adjectives, choices between similar but not identical words, breaking paragraphs at the right place for clarity and emphasis, and so on. 

    For example, while I would use an LLM to help me convert a piece I’ve already written into a white paper, I can’t see myself using it to write a new blog post. The value in e-Literate lies in my ability to communicate novel ideas with precision and clarity. While I have no doubt that an LLM could imitate my sentence structures, I can’t see a way that it could offer me a shortcut for the kind of expressive thought work at the core of my professional craft.

    If we can harness LLMs to help students learn how to write…um…prosaic prose, then they can start using their LLM “calculators” in their communications “physics” classes. They can focus on their clarity of thought and truly excellent communication. We rarely get to teach this level of expressive excellence. Now maybe we can do it on a broader basis. 

    In their current state of evolution, LLMs are like 3D printers for knowledge work. They shift the human labor from execution to design. From making to creating. From knowing more answers to asking better questions. 

    We read countless stories about the threat of destruction to the labor force partly because our economy has needed the white-collar equivalent of early 20th-Century assembly line workers. People working full-time jobs writing tweets. Or updates of the same report. Or HR manuals. Therefore our education system is designed to train people for that work. 

    We assume that masses of people will become useless, as will education, because we have trouble imagining an education system that teaches people—all people from all socio-economic strata—to become better thinkers rather than simply better knowers and doers. 

    But I believe we can do it. The hard part is the imagining. We haven’t been trained at it. Maybe our kids will learn to be better at it than we are. If we teach them differently from how we were taught. 

    Likely short-term evolution of the technology

    Those of us who are not immersed in AI—including me—have been astonished at the rapid pace of change. I won’t pretend that I can see around corners. But certain short-term trends are already discernable to non-experts like me who are paying closer attention than we were two months ago. 

    First, generative AI models are already proliferating and showing hints of coming commoditization around the edges. We’ve been given the impression that these programs will always be so big and so expensive to run that only giant cloud companies will come to the table with new models. That the battle will be OpenAI/Microsoft versus Google. GPT-4 is rumored to have over a trillion nodes. That large of a model takes a lot of horsepower to build, train and run. 

    But researchers are already coming up with clever techniques to get impressive performance out of much smaller models. For example, Vicuña, a model developed by researchers at a few universities, is about 90% as good as GPT-4 by at least one test and has only 12 billion parameters. To put that in perspective, Vicuña can run on a decent laptop. The whole thing. Tt cost $300 to train (as opposed to the billions of dollars that have gone into ChatGPT and Google Bard). Vicuña is an early (though imperfect) example of the coming wave. Another LLM seems to pop up practically every week with new claims about being faster, smaller, smarter, cheaper, and more accurate. 

    A similar phenomenon is happening with image generation. Apple has quickly moved to provide software support for optimizing the open-source Stable Diffusion model on its hardware. You can now run an image generator program on your Macbook with decent performance. I’ve read speculation that the company will follow up with hardware acceleration on the next generation of its Apple Silicon microchips.

    “Socrates typing on a laptop” as interpreted by Stable Diffusion

    These models will not be equally good at all things. The corporate giants will continue to innovate and likely surprise us with new capabilities. Meanwhile, the smaller, cheaper, and open-source alternatives will be more than adequate for many tasks. Google has coined a lovely phrase: “model garden.” In the near term, there will be no one model to rule them all or even a duopoly of models. Instead, we will have many models, each of which is best suited for different purposes. 

    The kinds of educational use cases I described earlier in this post are relatively simple. It’s possible that we’ll see improvements in the ability to generate those types of learning content over the next 12 to 24 months, after which we may hit a point of diminishing returns. We may be running our education LLMs locally on our laptops (or even our phones) without having to rely on a big cloud provider running an expensive (and carbon-intensive) model. 

    One of the biggest obstacles to this growing diversity is not technological. It’s the training data. Questions regarding the use of copyrighted content to train these models are unresolved. Infringement lawsuits are popping up. It may turn out that the major short-term challenge to getting better LLMs in education may be access to reliable, well-structured training content that is unencumbered by copyright issues. 

    So much to think about…

    I find myself babbling a bit in this post. This trend has many, many angles to think about. For example. I’ve skipped over the plagiarism issue because so many articles have been written about it already. I’ve only touched lightly on the hallucination problem. To me, these are temporary obsessions that arise out of our struggle to understand what this technology is good for and how we will work and play and think and create in the future. 

    One of the fun parts about this moment is watching so many minds at work on the possibilities, including ideas that are bubbling up from classroom educators and aren’t getting a lot of attention. For a fun sampling of that creativity, check out The ABCs of ChatGPT for Learning by Devan Walton. 

    Do yourself a favor. Explore. Immerse yourself in it. We’ve landed on a new planet. Yes, we face dangers, some of which are unknown. Still. A new planet. And we’re on it.

    Strap on your helmet and go.

  • ChatGPT Wrote This Article and then Totally Stole My Job!

    ChatGPT Wrote This Article and then Totally Stole My Job!

    As I outlined recently in my “e-Literate’s Changing Themes for Changing Times” post, I am shifting my coverage somewhat. I’ll be developing and calling out tags I use for these themes so that you can go to an archive page on each one. This one will be listed under the “AI/ML” “third-wave EdTech,” and “future of work” tags.

    I’ve been fascinated by the rapid progression of ChatGPT article fads:

    1. Look at this weird thing that writes stuff!
    2. I asked ChatGPT a question—and here’s what it answered!!
    3. I asked ChatGPT to write this article—and it totally did!!!
    4. Students could use ChatGPT to student essays write essays!!!! End of the world or totally awesome?????
    5. I asked ChatGPT for suggestions about preventing students from using ChatGPT to cheat—and it gave me five great suggestions (and five terrible ones)!!!!!!

    Waitaminute. Let’s back up.

    Students finding easy ways to cheat is not exactly a new thing. Remember, “to Chegg” is a verb now. Let’s back up to fad #3. Writers are asking ChatGPT to write their articles, publishing those articles, and then advertising that the articles published under their by-line were written in 30 seconds by a machine.

    Do they want to get replaced by an algorithm?

    It seems to me we’re thinking about the problem that these algorithms present in the wrong way.

    At the moment, ChatGPT is a toy

    Language-generating algorithms ChatGPT and their image-generating cousins are toys in both good and bad ways. In a good way, they invite people to play. Anyone, whether or not they know anything about programming, can explore the possibilities and limitations of a certain kind of cutting-edge AI. It’s fun. And, like all good games, you learn by playing. You can learn what they can and can’t do well. How to write a good prompt. And so on. They make AI more accessible and, in a certain way, more understandable.

    They’re toys in the bad sense because they weren’t designed for any particular purpose and tend to do weird things for any given task. For example, I tried to get DALL-E 2—ChatGPT’s image-generating sibling—to create a retro comic book cover depicting ChatGPT as a scary vampire. I tried half a dozen different prompts. The best result I got was with “An evil vampire on the cover of a retro comic book called ‘ChatGPT’”. Here’s the best image it came up with:

    “An evil vampire on the cover of a retro comic book called ‘ChatGPT’” as interpreted by DALL-E 2

    The algorithm does evil comic-book vampires pretty well. Spelling a word that was explicitly spelled for it in the prompt? Not so much.

    Remember, a large language model algorithm like the one behind DALL-E 2 and ChatGPT don’t understand anything. It’s taking a bag of words and putting together pieces it found on the internet to construct something that seems like an appropriate response.

    ChatGPT, for example, is notoriously bad at math. Because it doesn’t calculate. That’s not what it is designed to do. Its developers have been working on improving this limitation. So I asked it about how good at math it is:

    Me: How good are you at math?

    ChatGPT: I am able to perform basic arithmetic operations, such as addition, subtraction, multiplication and division. My training data includes mathematical concepts and I have been trained to recognize and respond to arithmetic questions. However, it is important to note that I may make mistakes in complex or unconventional mathematical computations.

    Me What is 2794 + 83948?

    ChatGPT: 87,742

    That answer is incorrect.

    So ChatGPT, the miracle of modern technology destined to destroy all education, is not as “smart” as a calculator. Note that you can ask Google or your Apple search prompt the same question and reliably get a correct answer. OpenAI (the makers of OpenAI) are likely trying to combine much simpler but mathematically reliable algorithms with the GPT-3 algorithm that underlies ChatGPT and DALL-E 2. The fact that they haven’t solved this problem yet suggests it is non-trivial. These different types of algorithms can be challenging to mesh. Many of the more impressive applications of artificial intelligence and machine learning are made possible by an ensemble of different algorithms working together, some of which fit together more naturally than others.

    Don’t get me wrong; the math problem will be solved. The OpenAI engineers will crack it just as they are now pretty good at preventing ChatGPT from making stock predictions or answering questions about current events when its data set hasn’t been updated since 2021. But it’s a game of whack-a-mole. Because you can ask ChatGPT anything, people do ask it anything. The creators are learning a lot about the questions people ask and what can go wrong with the answers. This new knowledge will help them design more specific solutions. But a general-purpose prompt tool like ChatGPT will be hard to make good at solving any one particular problem.

    I’m not convinced that ChatGPT, as it exists today, represents a big leap forward in essay cheating. It has length limitations, has to be fact-checked, can’t produce references, and spits out highly variable quality of reasoning and argumentation. Students would learn more by trying to fix the problems with a ChatGPT-generated draft than they would by going to a traditional essay mill.

    Short answer questions are a different matter. ChatGPT is already dangerous in this area. But again, students can already “Chegg” those.

    Yes, but…

    Could somebody write a better program specifically for writing school essays? Or magazine articles? Yes. That work is already underway.

    So what do we do about the essay cheating problem? Let’s start with the two most common answers. We can develop algorithms that detect prose that was written by other algorithms. That too is already underway. So we’ll have yet another flavor of the cheating/anti-cheating arms race that benefits nobody except the arms dealers. The anti-cheating tools may be necessary as one element of a holistic strategy, but they are not the ultimate answer.

    Second, we can develop essay-writing prompts and processes that are hard for the algorithms to respond to. This would be useful, partly because it would be good for educators to rethink their stale old assignments and teaching practices anyway. But it’s a lot of often uncompensated work for which the educators have not been trained. And it ends up being another arms race because the algorithms will keep changing.

    We miss the point if we respond to language-generating AI as a static threat that might become more sophisticated over time but won’t fundamentally change. ChatGPT is just a friendly way for us to develop intuitions about how one family of these algorithms works at the moment. You’re wrong if you think it is a one-time shock to the system. We’re just at the beginning. The pace of AI progress is accelerating. It is not just going to get incrementally better. It is going to radically change in capabilities at a rapid pace. It will continue to have limitations, but they will be different limitations.

    So what do we do?

    How about talking to the students?

    When adaptive learning hit peak hype, a glib response to teacher hysteria started making the rounds: “If you [teachers] can be replaced by a computer, then you probably should be.”

    Doesn’t that apply…um…generally?

    If all students learn is how to use ChatGPT to write their essays, why wouldn’t their hypothetical future employer use ChatGPT instead of hiring them? Why would students spend $30K, $40K, $50K, or more a year to practice demonstrating that a free-to-use piece of software does their best work for them? Students need to learn the work these tools can do so they can also understand the work the tools can’t do. Because that is the work the students could get paid for. Technology will make some jobs obsolete, leave others untouched, change some, and create new ones. These categories will continue to evolve for the foreseeable future.

    At a time when students are more conscious than ever about the price-to-value of a college education, they ought to be open to the argument that they will only make a decent living at jobs they can do better than the machine. So they should learn those skills. Why learn to write better? So you can learn to think more creatively and communicate that creativity precisely. Those are skills where the primates still have the advantage.

    Once we engage students openly and honestly on that point, we will start building a social contract that will discourage cheating and establish the foundational understanding we need for rethinking the curriculum—not just to keep from falling too far behind the tech but to help students get out in front of it. The current limitations of these AI toys demonstrate both the dangers and the potential. Suppose you want to apply the technology to any particular domain. In that case, whether it’s math, writing advertising copy, or something else, you need to understand how the software works and how the human expertise and social or business processes work. Whole echelons of new careers will be created to solve these problems. We will need thinkers who can communicate. Learning how to formulate one’s own thoughts in writing is an excellent way to learn both skills.

    Fighting the tech won’t solve the problem or even prevent it from getting worse. Neither will ignoring it. We have to engage with it. And by “we,” I include the students. After all, it’s their futures at risk here.

    (Disclaimer: This blog post was written by ChatGPT.)

    (I’m kidding, of course.)

    (I am able to perform basic humor operations, such as generating dirty limericks and “your momma is so ugly” jokes. My training data includes humorous concepts, and I have been trained to recognize and respond to knock-knock questions. However, it is important to note that I may make mistakes in complex or unconventional humor.)

  • The Catalysts for Competency-Based Learning and Prior Learning Assessments Have Arrived

    The Catalysts for Competency-Based Learning and Prior Learning Assessments Have Arrived

    As I outlined recently in my “e-Literate’s Changing Themes for Changing Times” post, I am shifting my coverage somewhat. I’ll be developing and calling out tags I use for these themes so that you can go to an archive page on each one. This one will be listed under the “changing enrollment” tag.

    Just before Christmas, The New York Times published an ostensibly feel-good story about a Syrian refugee who built a massively successful chocolate business in Canada. But the story buries the lede. The company’s CEO could have—and should have—been a doctor. He couldn’t get credit for his prior education in Syria. Canada is facing an acute healthcare crisis because of a shortage of skilled workers. Canada has figured out half the problem with the labor shortages that plague many industries there and here. They are welcoming immigrants willing to work hard and do the jobs. But they are missing the other half, enabling those immigrants, many of whom arrive with skills, to employ those skills where they are needed. This denial of economic opportunity causes a well-intentioned policy to fail to live up to its economic and humanitarian aspirations.

    Canada’s labor shortages are hardly unique, as anyone who has been paying attention to the US economy knows. We are suffering real economic pain from our inability to attract, train, and retain skilled workers in various industries. Worse, in this new era of economic, geopolitical, rapid technological advancement (e.g., ChatGPT) and climate volatility, we can expect the shifts in job markets to accelerate as supply chains get rewired, industries get disrupted, and reconfigured,

    Until recently, I have been a skeptic regarding Competency-Based Education (CBE) and Prior Learning Assessment (PLA) because I did not believe the economic drivers were present to force the massive reconfiguration of the higher education system. But situations change. Long-term economic forces will increasingly drive demand for more rapid reskilling than our current system can support. Meanwhile, a significant and growing percentage of U.S. colleges and universities face enrollment crises. While this problem is often framed by academia as a decrease in the supply of students—the so-called “enrollment cliff,” the hot job market, and so on—I think it is better understood as a failure to respond to changing demand and new opportunities. EdTech and venture investors have been arguing this for decades. I continue to believe that they were wrong. But, like I said, situations change.

    In this post, I will argue that now is the time for CBE and PLA at scale, using the Canadian healthcare labor market as the primary example. I will also make the case by focusing on skills mainly as an additive to the degree—the degree-plus-“skills” formula promoted by Coursera, boot camps and the like—institutions are missing opportunities. We will see that well-defined paths for pre-degree career credentials exist in critical industries. By neglecting these paths and leaving them to others, traditional higher education leaves itself vulnerable while failing to serve its mission to students and the public good.

    Willy Wonka could have invented the everlasting artificial heart

    Let’s return to our story about the Syrian refugee and his Canadian chocolate empire:

    Back in Syria, [Refugee Tareq] Hadhad’s father, Isam, had founded a confectioners in Damascus that eventually employed hundreds of people and shipped its chocolates throughout the Middle East. Bombing during the civil war leveled it.

    The Hadhads became privately sponsored refugees in Antigonish, Nova Scotia. While the town is the home of St. Francis Xavier University, it is generally known for having an aging population rather than being economically vibrant.

    Mr. Hadhad was midway through medical school when he fled Syria. But once in Canada, and with considerable help from the people of Antigonish, he vowed to re-establish his father’s business under the name Peace by Chocolate….

    This month, Mr. Hadhad opened a new, bigger shop and expanded the factory that produces the company’s chocolate. In all, Mr. Hadhad told me, Peace by Chocolate now employs about 75 people and could hire 30 to 40 more workers — if they were available in Antigonish. About 1,000 stores across Canada now sell its chocolates, thanks in part to a deal with the Empire Company, the Nova Scotia-based grocer that owns the Sobeys and Canada Safeway supermarket chains.

    The Syrian Family That Rebuilt a Chocolate Empire in Nova Scotia

    What an incredible story. Tucked in it is a comment about labor shortages, but given the aging population of Antigonish, that makes sense. Mr. Hadhad is doing precisely what the government of Canada and the people of Antigonish hoped he would do when they invited him and his family; create jobs that would attract more young workers. “Building a business in Canada, he said, is much easier in than in Syria,” he said.

    But.

    he was also keen to discuss what’s become something of a personal mission for him: eliminating barriers for newcomers and showing Canadians the economic value of immigrants.

    A former medical student, Mr. Hadhad is disturbed that many immigrants are unable to use their skills immediately when they come to Canada; instead, they often must undergo additional schooling, and face slow and costly certification processes.

    Mr. Hadhad was told that if he wanted to pursue his medical studies, he would have to return to high school, obtain a Canadian undergraduate degree and then take medical school admission exams.

    The Syrian Family That Rebuilt a Chocolate Empire in Nova Scotia

    Here’s a guy who was halfway through medical school when he arrived. He was told he would have to repeat all his education starting from high school to finish his degree and become a practicing physician.

    Let’s set aside the humanitarian aspect of this. Forget that most immigrants don’t arrive with the second set of skills Mr. Haddad had—running a chocolate business—or the confidence and support to reinvent themselves. Instead, let’s look at this purely from a policy perspective. Was Mr. Haddad’s career change a net gain or a net loss for Canada?

    About a month before The New York Times ran the chocolateer story, it ran another one with the headline, “Alleviating Canada’s Acute Shortage of Family Doctors“. According to that article, “Nova Scotia’s latest monthly tally, released in mid-October, showed that 110,640 people, or 11 percent of the population, were on the wait list for a family doctor.”

    Nova Scotia [is] not alone. The recently re-elected Coalition Avenir Québec government dropped its promise to ensure that everyone has a family doctor. More than 800,000 Quebecers are without one. In Ontario, the provincial advocacy group for family physicians estimates that 1.8 million residents do not have a family doctor and another 1.7 million people are under the care of physicians older than 65 who are nearing retirement.

    The desperation to secure a physician pushed Janet Mort in British Columbia to drastic measures. She took out an ad in a local newspaper in search of a physician to fill her 82-year-old husband’s prescriptions after his physician retired, as reported by Global News. Her strategy was successful.

    For others, the process to find a family doctor has meant working the phones to call individual clinics or to join growing provincial wait lists. Those who turn to the services of walk-in family doctors find longer wait room times and no continuity of care. And some people add to the congestion in overburdened hospital emergency departments.

    Alleviating Canada’s Acute Shortage of Family Doctors

    This is a many-faceted problem but one of the causes is that physicians are not trained in how to run a private practice as a business:

    [Katherine Stringer, the head of the Department of Family Medicine at Dalhousie University in Halifax,] acknowledged that while family doctors are in effect small business owners, the training they receive on how to run their business while in medical school is “very rudimentary.”

    As a result, Dr. Stringer said, for many new doctors “it’s a very stressful first year.” Emulating a strategy used for new technology companies, the medical school has brought in mentors to help new doctors find their way. Dalhousie is also working with the province on establishing teams to set up all of the patient record compiling needed for a new practice.

    Alleviating Canada’s Acute Shortage of Family Doctors

    Guess which could-have-been-a-doctor has proven he has prodigious business-building skills? While I believe in the healing power of chocolate, I suspect that the 110,640 people waiting to get a primary care physician would have preferred another option.

    Nor is primary care the only area where Nova Scotia’s health system suffers from a labor crisis. A quick search on the topic yields disturbing results. A Nova Scotia woman died on New Year’s eve while waiting for care in an emergency room. The same article notes that emergency room deaths are rising in the province while 43,000 people left Nova Scotia emergency rooms without being seen by a doctor last year.

    It gets worse. One hospital closed its emergency room for a month and is not sure if it will be able to run a full emergency room in the future.

    There are, of course, many reasons for Canada’s healthcare crisis. Cost-cutting measures and (often related) rise in time-consuming paperwork are significant drivers. But telling eager, trained physicians that they will have to repeat their education, starting in high school, does not help.

    Responding more nimbly to labor market changes

    The obvious solution, of course, is to test the immigrant healthcare workers on what they know and train them to fill the gaps. Canada is aware of this possibility. For example, The Globe and Mail reports Ontario is implementing a “Practice-Ready Assessment” program that “could add hundreds of foreign-trained doctors to the overstretched health care system within months.” The same article states, “Some estimates put the number of foreign-trained physicians living in Ontario but not working in their field as high as 13,000. They are blocked by licensing hurdles and other barriers because their medical training was done elsewhere.”

    Thirteen-thousand foreign-trained physicians are theoretically available to an overstretched system but have been blocked from practicing because they have not been given a chance to prove what they know. Ontario is one of just three Canadian provinces that are taking this approach so far. And they haven’t implemented it yet. A quick internet search will show similar mismatches with skilled nurses. Like a story in The New York Times from a few months ago entitled “‘Disaster Mode’: Emergency Rooms Across Canada Close Amid Crisis.” From the article: “Increasingly, I think many of us realize we are not going to, in the short term, train our way out of this…. We can’t produce nurses quickly, with the exception, possibly, of some foreign graduates.”

    Meanwhile, here in the United States, we find another labor mismatch crisis hidden in a feel-good story. New York City is addressing equity and climate change goals by training low-income workers to service modern electrical systems like heat pumps and electric vehicle charging stations. Here’s the story of one apprentice, Robert Clark:

    Before joining, he struggled to find work, in part because of a felony conviction for burglary. “It’s a no-brainer,” he said of joining the Civilian Climate Corps, which pays him $20 per hour to learn skills and receive the certifications that he needs to get work. He hopes to go back to school to become an engineer.

    Green energy has a problem: There aren’t enough electricians. Here’s one solution.

    Did you catch that last sentence? Mr. Clark wants to go back to school to become an engineer.

    There’s only one problem. New York City’s Civilian Climate Corps is a collaboration between the municipal government and employers. No colleges or universities are mentioned in the article or on the organization’s website. When Mr. Clark is ready to go back to college and become an engineer, will he get credit for the knowledge and skills he has learned through the program? As of today, I see no evidence of any pathway to do so. New York City has a robust collection of community colleges within the CUNY system. Why are they not involved? If Mr. Clark had completed his apprenticeship in a CUNY-affiliated program and received CUNY credit, it would be natural for him to return to CUNY someday to get his engineering degree. As it stands today, CUNY means nothing to him, and his knowledge means nothing to CUNY.

    We need to be able to meet students where they are and get them where they need to go. “Meeting students where they are” is often a euphemism for talking about skill deficits. There is so much more than deficits to “where students are,” including pre-existing skills. Today, we are seeing alternative skills networks being built around the edges of academia, particularly in credentialed trades like electricians and allied healthcare. Becoming certified in medical billing in the United States does not require a degree and can lead to earning a decent living. It’s an example of a relatively quick path to financial sustainability. I’m unsure whether similar pathways exist in Canada, where the medical paperwork processing explosion seems to be a newer phenomenon. Economies and job markets change.

    Meanwhile, people continue to aspire. Mr. Clark wants to go “back to school” to become an engineer. Nobody has told him that, as far as the system is concerned, he never went to school. Until academia better integrates itself into this network of ever-shifting needs and skill gaps, it will continue to face shrinking enrollments and dwindling relevance.

    To truly meet students where they are and get them to where they want to go, we need to assess what they already know on a granular level, give them credit for it, and help them fill in any gaps on an equally granular level. The knowledge Mr. Roberts gains working on heat pumps and electric car chargers will likely not line up neatly with traditional college course curricula. Mr. Hadhad may also have skill gaps that do not line up neatly with the courses he would have needed to take to become a Canadian physician. (For example, he would have no reason to know basic details about how the Canadian healthcare system works.) Both came from backgrounds where becoming an income earner quickly was a high priority. Both have highly valued skills that the workforce needs.

    Perhaps one day, Mr. Roberts could someday create a breakthrough in heat pump efficiency or electric car charging speeds. If so, I imagine two likely paths for him to learn the skills he will need to make that contribution to society. Either he will go to college, or he will learn what he needs on the job.

    In general, employers do not make good educators. If employer training becomes the dominant path, it will be a narrow and inefficient one. Some institution is needed to fill the educational role. Colleges and universities could provide this role while weaving career paths through the traditional liberal arts education that has served humanity so well for so long. But only if they build the pathways to accommodate these continuous and granular needs. If they don’t, then somebody else will.

    While we’re not going to be taking a deep dive into these issues, the webinar I’ll be facilitating this week will provide some basic groundwork for understanding CBE.

    It’s next Wednesday, January 18th, at 1 PM EST.

    Register here.

  • I’m Facilitating a Webinar on CBE on 1/18

    I’m Facilitating a Webinar on CBE on 1/18

    I’ll be facilitating a webinar on Competency-Based Education (CBE) next Thursday as the culmination of a three-part series about the changing higher education landscape hosted by Open LMS. The first webinar was about the connections between higher ed and workforce learning. The second was about international education. In this third webinar, we’ll be talking about how support for CBE ties in to the prior two webinar themes. CBE can be a very useful tool for colleges and universities to reach students that they often have trouble reaching. I’ll be publishing a longer post about this connection early next week.

    For now, I’d like to say a bit about how we’re going to approach CBE in the webinar and a bit about the host and guests.

    On CBE, I find there are generally two very different groups. The first camp is the believers who are thoroughly immersed in it and are thinking deeply about how to overcome the challenges of implementing it effectively at scale. The second group is more diverse but share the common trait that they have not bought into CBE yet. Some like the idea in principle but see the hurdles to getting started as daunting. Some worry that CBE could effectively dismantle aspects of education that they cherish by reducing it to parts. And some just haven’t thought about it much.

    This webinar is aimed at the latter group, although I encourage CBE enthusiasts to come, both because they help spread the word about an approach they believe in and because I do believe the webinar could still unearth some nuggets of new knowledge for them—especially those that are trying to push ahead but hitting obstacles early. CBE can be a slippery slope that feels like an all-or-nothing strategy. We’ll be breaking that down a bit and trying to make it more accessible to schools and programs that are interested but might not see an on-ramp.

    While this won’t be primarily a software demo webinar, we will be showing some software features to illustrate those on-ramps and some foundational points about CBE for the uninitiated.

    Our guests include the following:

    • Brad Koch, Vice President of Education & Strategy at Open LMS: Brad is another Open LMS guy that I’ve known and counted as a friend in EdTech for decades. I met him at Angel Learning, where he was involved in designing features that other LMSs are still catching up with today. He has worked on almost every major higher education LMS with a significant presence in the US.
    • Laurie Pulido, CEO of Ease Learning: Laurie has been thinking about CBE for a very long time, having been one of the earliest learning designers at SNHU back in 2005. That deep expertise shows in the product and service work Ease Learning has been doing.

    We’ll also be joined by Brad Schweitzer, Product Manager at Open LMS, who will be helping us with some of the product details.

    I want to say a word about our hosts Open LMS as well. As I have noted in past posts about this series, I am picky about who I do paid webinars for. I have been interested in Open LMS both because I know and respect the senior leadership, because they have incredible breadth of reach across sectors and geographies in education, and because they’ve been very flexible in working with me to turn this series into something other than a showcase for their products and services. While you will get more of a look at their product in this webinar than you did in the last two, they continue to support me in shaping the series to be focused on the broader needs of the broad range of customers and partners that they serve. The platform is interesting, competitive, and worth a look. But the series really shows the values of Open LMS, which is often harder to get a read on with a vendor until after you’ve already signed a contract.

    Again, the panel is Wednesday, January 18th at 1 PM EST. Register here.

  • EdTech’s Funding Problems Are Going to Get Worse

    EdTech’s Funding Problems Are Going to Get Worse

    As with much of the venture capital world, EdTech venture funding has slowed to a trickle overall and is almost completely frozen in some areas, such as early-stage investments. Conventional wisdom seems to be that this state of affairs won’t last forever. Since the proximate causes of the situation are falling stock prices and high inflation, the investing environment should improve significantly during 2023 as inflation falls and the stock market begins to recover.

    Unfortunately, this may be an oversimplification. Longer-term changes in the markets may cause lasting damage to the kinds of private equity (PE) companies that typically buy EdTech start-ups. And since venture-funded EdTech companies are much more likely to be sold to PE companies than they are to become publicly traded, the harm to EdTech venture funding caused by these long-term shifts could far outlast any slowdown in broader venture investing.

    If I am right, this is bad news not only for investors and start-ups but for EdTech companies of all sizes and, ultimately, for their academic customers.

    How venture funding works

    In order to understand the problem, we have to start with the basics. For example, where do venture capitalists (VCs) get the money that they invest? The short answer is “from other investors.” Suppose I manage Harvard’s endowment fund. I already own half of the stock market and am looking for other opportunities. I’m interested in venture capital, particularly in education, but I don’t have the time or expertise to vet individual start-ups. I might become a limited parter (LP) in a VC fund, which is a little bit like investing in an actively managed mutual fund. The VC firm takes my money and spreads it around to start-ups that the partners believe are good bets.

    The stock market and interest rates have strong impacts on my comfort level with high-risk investing. If my endowment fund is heavily invested in S&P 500 stocks and that investment has dropped by 20% over the past year (as the S&P 500 did in 2022), I may feel a little queasy about investing in more risky bets like startups. I’m more likely to preserve my capital.

    Suddenly VCs face a much more difficult environment for raising investment funds. If they are trying to raise a new fund, they’re going to have to wait a while. If they have funds to invest, they may want to slow down and make their money last.

    On top of their cash crunch, they also struggle to decide how much companies are worth. There’s an old investing adage that one should not try to catch a falling knife. Since nobody knows how much further the stock market will drop, and since risks in start-ups are both higher and harder to gauge than with publicly traded companies, many VCs are choosing to simply hold onto their cash and wait until the markets stabilize.

    Again, conventional wisdom is that these VCs can’t sit on their money forever. Their LPs expect them to invest and make returns. So at some point, the cash that is sitting on the sidelines will come flooding into the market.

    Unfortunately, that influx may be short-lived for EdTech.

    EdTech’s PE problem

    So far, we’ve looked at the VC markets and public stock markets. But there’s a third, less visible but huge slice of the capital markets that we haven’t examined yet: private equity (PE). And it turns out that PE is heavily involved with EdTech.

    Technically, venture capital is a type of private equity. Interpreted broadly, PE simply means that investors are putting their money into companies that are not for sale on publicly traded stock markets. Companies that are labeled as PE companies, much like their VC counterparts, raise their investment capital from individual and institutional LPs.

    But where VC funds can be thought of as growth investors, trading high risks for high returns, classic PE funds typically act more like value investors. They often look for companies they think are undervalued and could be turned around by cutting costs, changing management, and focusing the strategies of the companies they invest in.

    According to Dealroom, roughly 90% of EdTech is privately owned. This includes household names like Instructure, McGraw-Hill Education, Cengage Learning, Ellucian, and University of Phoenix, among many others.

    Proportion of EdTech that is privately owned; Source: Dealroom.co

    In many investment sectors, venture-backed companies will often aspire to become publicly traded companies. In EdTech, the vast majority are likely to be acquired by larger PE-backed companies. VC investors understand this and plan for it when choosing the companies they invest in. At this moment, PE firms and the companies they invest in are going bargain-hunting. Distressed start-ups are selling cheaply. expect to see a substantial uptick in announcements about EdTech mergers and acquisitions over the next six months.

    That party may not last long, however.

    Given the current financial climate, this state of affairs creates a medium-term and a long(er)-term problem for EdTech. Medium-term, private equity companies tend to be much slower in writing down their valuations. The stock market is in the midst of a brutal process of writing down the value of publicly traded companies. For example, here’s the one-year stock chart for 2U (TWOU):

    2U’s one-year stock price chart

    Could 2U’s stock price decline further? Sure. For example, if the stock market as a whole continues to drop, then 2U could easily drop further as well. The laws of gravity apply. That said, 2U’s stock price reflects real-time conditions, including minute-by-minute decisions by investors. It’s up-to-date.

    In contrast, PE-owned companies report their earnings quarterly and make their own calculations regarding the underlying value of their companies. They are financially motivated to drag their feet on reducing the estimated value of their investments. They are audited once a year. So there is much more lag in the revaluation of PE-owned companies in a bear market like this one. The PE-heavy EdTech sector is likely to get hit by another round of significant company valuation downgrades just around the time that the publicly traded markets are recovering.

    As the saying goes, sh*t rolls downhill. If PE-owned EdTech companies do not have the cash to make acquisitions, then VCs will have to rethink how hard it will be to sell the companies they invest in. They will adjust their investment strategies accordingly. The generic advice for venture-backed companies these days is to have enough cash on-hand to last two years until your next investment. I wonder if that time period needs to be longer in EdTech.

    The longer-term PE problem

    Unfortunately, the troubles don’t end there.

    Many EdTech investments by PE firms are made through what’s known as “leveraged buyouts.” Simply put, a leveraged buyout is when the investor borrows a lot of the money it uses to invest in or purchase the company. It’s a bit like a mortgage. You pay 20% of the price of the house in cash and finance 80% with the bank.

    What is a Leveraged Buyout? Credit: Statistica CC-BY-SA

    Of course, if you happen to have a variable-rate mortgage, need to refinance, or are looking to get a mortgage on a new house, you may suddenly find that your costs are much higher than you expected them to be.

    This happens with companies in leveraged buyouts too. When the loans—or “debt covenants”—used to buy these companies come to term, they usually have to be refinanced. Except that this time the interest rates are much higher. Suddenly, money that was earmarked for building new features or paying employees has to go toward paying interest on the loans.

    That’s only half the problem. Remember, PE firms have limited partners just like VCs do. PE companies employing leveraged buyouts are making a bet that the value of the companies they’re buying with debt will rise at a faster rate than their interest payments. As those payments go up, the investments become less attractive. PE firms are going to be more likely to make draconian budget cuts in their portfolio companies, try to unload the most heavily indebted ones in whatever way they can (including them off in pieces), and make fewer, more careful investments.

    One way to think about the choice LPs are faced with is the relationship to an alternative investment in Treasury bonds, which are so safe when held to term that they are referred to as “riskless investments.” Here’s a chart of the historic interest rates on the 10-year U. S. Treasury note:

    Interest rates on the 10-year US Treasury note, 1960-2022

    Interest rates have been historically low since about 2010. While interest rates will come down from their current levels, they are more likely to settle around their historical average of roughly four percent. This change has been sudden, impossible to time, and diverges from the norm that even 30-year-old PE employees have been in the business. PE companies will have to generate significantly better returns than the “riskless” alternatives even as they face the headwinds of higher interest rate payments. That means leveraged companies will need to increase their financial performance in order to provide the same benefits relative to their alternatives.

    I’m not sure the PE firms are ready for this.

    To be clear, the situation is not apocalyptic. PE investors were purchasing companies with debt when the interest rates were closer to the historical norm and will undoubtedly continue to do so in the future. The problem is more that many of their current investments may have been made based on assumptions about the interest rate environment that no longer hold. While the system will need a while—possibly a few years—to rebalance itself, it eventually will.

    Still, a debt hangover in PE would not be good news for anyone who cares about EdTech. Big companies that customers depend on will be squeezed, spending less, and laying off employees. VCs, knowing that their exit options have become more limited, will make fewer investments that are less risky and are more likely to be near-term profitable. We are likely to see even less investment in companies that can have significant and scaled impact on student outcomes because these companies often need to invest for a longer period of time before they make more money than they spend. Educators and educational institutions relying on companies of all sizes will be at increased risk.

    The handful of publicly traded EdTech companies will largely be subject to different dynamics and should recover with the stock market and the economy. While a few of them may be debt-heavy, many more of the PE-funded companies will have this problem. And unfortunately, a heavy debt load could take years rather than months or quarters to work its way through the system.

    We need to know more

    To get a better grip on the shape and severity of the problem, somebody who is more of a financial analyst than I am should compile some data on the scope of the problem (if they haven’t done so already). Ideally, I’d like to see a list of “red light” companies that are highly leveraged and whose covenants will be coming due soon and are already under financial stress. In other words, the EdTech firms for whom higher interest rates will put them in jeopardy. Next, it would be good to see a list of “yellow light” companies whose upcoming debt refinancing is likely to put them under stress, meaning they are not in mortal danger from higher interest rates but would be put under enough pressure that they would probably need to make significant additional cuts. And finally, I’d like to see an overall read on the debt situation for the sector, with a special emphasis on PE-funded companies that have historically been acquisitive.

    Ecologists have learned that the health of a given ecosystem is heavily determined by the health of the apex predators in that ecosystem. In much of EdTech, PE-owned companies are apex predators. They are at the top of the food chain. Clarity regarding the debt of PE-owned EdTech would give us one measure of the financial health of the EdTech ecosystem as a whole.

  • I Would Have Cheated in College Using ChatGPT

    I Would Have Cheated in College Using ChatGPT

    As I outlined recently in my “e-Literate’s Changing Themes for Changing Times” post, I am shifting my coverage somewhat. I’ll be developing and calling out tags I use for these themes so that you can go to an archive page on each one. This one will be listed under the “AI/ML” “third-wave EdTech,” and “future of work” tags.

    ChatGPT is creating all kinds of buzz about students cheating on essays. So it got me thinking. If I had had access to a tool like ChatGPT when I was in college, would I have used it to cheat?

    A robot writing an essay
    “A robot writing an essay” as interpreted by DALL-E 2

    Yes. Absolutely. One hundred percent. But I wouldn’t have thought of it as cheating.

    Cheating is a state of mind

    When I was in college, I had a massive chip on my shoulder. If I caught the slightest whiff that the professor didn’t care whether I was learning or the assignment was not well-designed to help me learn something, I would immediately flip into “grudge” mode.

    I never cheated. The whole point, in my immature mind, was to prove that I was smarter than the professor. So I would stay up late, party the night before the assignment was due, and give myself three or four hours to write it the next morning, hung over, before I had to turn it in. To give you a sense of just how far these grudges went, I didn’t actually enjoy partying very much. Sometimes I would go out of my way to do so because of the stupid assignment. It was all part of a game to challenge myself. No pain, no game.

    I wasn’t always so self-destructive about it. If I felt the professor was genuinely interested in my learning but had simply written an assignment I wouldn’t learn from, I’d take the writing prompt and try to be as creative—and subversive—as possible. But that only worked when the professor would understand and appreciate the joke. If I had a professor who had (in my judgment) given me a coloring book and was going to grade me on whether I colored inside the lines, that triggered my worst adolescent self. While I didn’t cheat in the conventional sense, my goal was to minimize effort required to get an adequate grade and show myself how smart I was in the process. My social contract with the teacher was no less broken than the student who copied somebody else’s work. I once defined cheating as “engaging in behaviors that are intended to facilitate passing without learning.” By that standard, I cheated my ass off.

    Something like ChatGPT would have been part of the game to me. If you’ve played around with it at all (or read some articles about it), you’ll know that writing prompts that lead the AI to generate good text is something of an art. I would have spent a lot of time crafting the best prompt possible. Then I would have edited the output. I still would have wanted to produce a good essay. If this process took more time to produce the end result than just writing it from scratch myself, it wouldn’t have mattered to me. I would have subverted the assignment into something that actually challenged me while flipping the bird to the instructor who had the temerity to underestimate or underappreciate their students in general and me in particular. That was the whole point of the game. I wanted to learn. And I wanted to care. Nothing pissed me off more than a professor who wasted my educational time.

    ChatGPT wouldn’t have violated my “no cheating” rule because I wouldn’t have been cheating according to my rules.

    Faculty tend to think that cheating is defined by their rules and the college honor code. The reality is far more complex. For me, it was heavily influenced by my social contract with each teacher, whether I felt they were holding up their end of the bargain, and how I could turn every assignment into a game that was a fun mental challenge. Other students may be influenced by which assignments they think are important for their career goals, how much work is reasonable to expect of them and, very often, or whether they think the instructor cares about their learning.

    I can’t emphasize that last point enough. I’ve conducted a fair few focus groups with students over the years. The results consistently supported the research evidence that students are heavily influenced by whether they believe their teacher cares about their learning. And this manifests in surprising ways. I remember one particular focus group vividly. We were talking about what factors cause students to engage in with a class more than they expected or planned to. They all agreed that having a teacher that cared about their learning was a major factor. I asked them how they can tell if a teacher cares. I got some of the usual, expected answers like “she knows my name” or “he talks to me after class to see how I’m doing.” One surprising answer that elicited a lot of nods from the group was, “She picked a random student during class and asks them a question about the reading. She didn’t grade us on the answer but it’s embarrassing if you don’t know it.”

    “You liked that?” I asked, somewhat incredulously.

    “No,” he replied. “I hated it. But I know she did it because she wanted us to learn. So I did the readings.”

    ChatGPT as a tutor

    The current fad with journalism about ChatGPT is to ask the bot some version of, “If you were writing this article, what would you write?” The value of that exercise depends on the incisiveness of the commentary regarding the output, the prompt, and the relationship to writing craft. Examples of this genre can range from pretty interesting to pretty terrible, depending on the quality of thinking of the human “writer” involved. Bad articles end either in “WHEEEEE!” or “NO HUMAN WILL EVER LEARN HOW TO WRITE A COMPLETE SENTENCE AGAIN!” Better articles give us insight into what the tool is good for as a writing aid and how that reflects on writing craft.

    Let’s see how well I fare at this genre. Can ChatGPT write this blog post for me? The base question I’m after is how to use a tool like ChatGPT to engage students in the craft of writing the way it would have engaged me. As you’ll learn by playing around with the tool or even reading some of the better articles on it, part of the craft in getting useful answers out of ChatGPT is writing a good prompt. My prompts in the experiments below are in bold. ChatGPT’s answers follow.

    I’m a college philosophy professor. I have created an assignment asking students to write an essay on a particular topic. They are allowed to use ChatGPT to generate their first draft but will be graded on the accuracy, originality of thought, and writing quality of their final draft. They will also be evaluated on the quality of the ChatGPT prompt that they write. What are the educational strengths and weaknesses of this assignment?

    There are both strengths and weaknesses to using a tool like ChatGPT for a college philosophy assignment. One potential strength is that it can help students generate ideas and get started on their essays. This can be particularly helpful for students who may be struggling to come up with a topic or approach to the assignment.

    Another potential strength is that using ChatGPT can help students practice using technology in their writing. This is an important skill in today’s world, and it can help students develop the ability to effectively use tools like ChatGPT in their future academic and professional endeavors.

    On the other hand, there are some potential weaknesses to this assignment. One is that ChatGPT is a tool that generates text based on prompts, but it does not have the ability to understand the meaning or significance of the words it generates. This means that the essays produced using ChatGPT may not be as original or thought-provoking as those produced by the students themselves.

    Another potential weakness is that using ChatGPT could discourage students from engaging with the material in a deep and meaningful way. Instead of reading and thinking critically about the assigned texts, they may simply use ChatGPT to generate a first draft and then submit it without further thought or analysis.

    Overall, the use of ChatGPT in a college philosophy assignment has both strengths and weaknesses. It can be a helpful tool for generating ideas and getting started on an essay, but it should not be used as a substitute for careful reading, thinking, and writing.

    My dialogue with ChatGPT

    Meh. ChatGPT’s answer has all the hallmarks of an adequate but unimaginative high school essay, right down to the form. It has not produced an essay that is “original or thought-provoking” or written content that engages with the prompt “in a deep and meaningful way.”

    We can marvel at the ability for the technology to do this well. It is genuinely miraculous and merits a brief pause for a moment of wonder. That said, the content itself is not enlightening or even particularly interesting by human writing standards.

    Let’s try again. I’ve altered the prompt using language from ChatGPT’s output in our first try. To make the changes more obvious, I’m showing the text I deleted from the original prompt and underlining the text I added:

    I’m a college philosophy professor. I have created an assignment asking students to write an essay on a particular topic. They are allowed to use ChatGPT to generate their first draft but will be graded on the accuracy, originality of thought, and writing quality of their final draft. They will also be evaluated on the quality of the ChatGPT prompt that they write. What are the educational strengths and weaknesses of this assignment? How can I improve this assignment to encourage students to produce more original, thought-provoking assignments in which the students engage with the material in a deep and meaningful way?

    One way to improve this assignment would be to provide more guidance on how to use ChatGPT effectively. For example, you could provide students with examples of good and bad ChatGPT prompts, and explain how to write a prompt that will generate thoughtful, original responses from the tool.

    Another way to improve the assignment would be to require students to engage more deeply with the material before using ChatGPT. For example, you could ask them to read and annotate the assigned texts, or to participate in a class discussion, before using ChatGPT to generate their first draft. This will help ensure that they have a strong understanding of the material and are able to generate original, thought-provoking ideas on their own.

    In addition, you could also consider modifying the grading criteria for the assignment. For example, you could place less emphasis on the accuracy of the ChatGPT-generated text, and instead focus on the student’s own writing and analysis in the final draft. This would encourage students to take the time to engage with the material and produce original, well-written essays.

    Overall, there are several ways you could improve this assignment to encourage students to produce more original, thought-provoking work. By providing more guidance on how to use ChatGPT effectively, requiring deeper engagement with the material, and modifying the grading criteria, you can help students develop the skills and habits needed to produce high-quality work in their philosophy studies.

    My dialogue with ChatGPT

    This answer is more interesting. The first suggestion—teach students how to write effective prompts—is a good start. I do think this is a skill that will be taught widely in schools within five years. (It should be sooner than that and it realistically might be later.) I don’t agree with the suggestion to de-emphasize the accuracy of ChatGPT’s text. Evaluating the quality of source material is an essential skill which becomes more important, complex, and interesting with the introduction of tools like this one. That said, a lot depends on context. I could imagine assignments for which I would take ChatGPT’s advice on this point. Notice, though, that the structure of the “essay” ChatGPT generated is still pretty stiff and formulaic. Leaning too heavily on a tool like this could have the effect of cultivating competent writing at the expense of stifling great writing.

    Still, I can see the shape of a pedagogical process—and preferably a supporting end-to-end tool—that teaches many of the skills involved with good writing, including some hard ones like checking sources and editing—while including some elements of creativity. If it is scaffolded properly—again, with the right tool and process but also with a good, solid rubric—it could enable educators to spend more of their time honing in on specific aspects of the writing process with less drudgery. Particularly if used judiciously as part of the writing curriculum rather than the whole thing, it could be quite useful.

    It’s also honest. It strengthens rather than weakens the social contract between student and educator by allowing the students to use a tool as long as they are open about it and are using it as part of a genuine learning process rather than a shortcut around thought work.

    Would I use ChatGPT to help me write blog posts?

    In principle, I have no problem with the idea of using machine-generated text in e-Literate posts as long as it is properly attributed. In practice, I haven’t been able to figure out a way to make it useful.

    Part of the value of e-Literate is that it can be surprising in both content and form. Novelty teaches while entertaining. Could a tool like ChatGPT develop the right sort of novelty to fit with this blog? On the surface, it probably could. We’re already starting to see examples of the tool being asked to write an article on X subject in the style of Y person. If trained on the thousands of posts I’ve written, I suspect that a tool like ChatGPT, or maybe the next generation of it, could learn to use more em dashes, write convoluted sentences, and be more snarky. It might even incorporate some themes that show up in my posts.

    But it doesn’t actually understand anything that it writes. ChatGPT distills and synthesizes answers that have already been given. It can only write about ideas that I have already thought of and written about. It can’t write about the idea I’m going to have tomorrow. e-Literate isn’t about what I know. It’s about what I’m learning. As such, I don’t yet see how I could get much value out of a tool like ChatGPT in the foreseeable future, at least for this blog. The only exception I can think of is for posts like this one that are about the tool. ChatGPT didn’t really write part of this post for me. It generated artifacts for me to analyze in my own writing.

    These lines of demarcation—the lines between when a tool can do all of a job, some of it, or none of it—are both constantly moving and critical to watch. Because they define knowledge work and point to the future of work. We need to be teaching people how to do the kinds of knowledge work that computers can’t do well and are not likely to be able to do well in the near future. Much has been written about the economic implications to the AI revolution, some of which are problematic for the employment market. But we can put too much emphasis on that part. Learning about artificial intelligence can be a means for exploring, appreciating, and refining natural intelligence. These tools are fun. I learn from using them. Those two statements are connected.

    Would I teach writing using ChatGPT?

    If I were teaching writing today, would I use an AI tool? In practice, probably not, simply because it would be too much work to cobble together the pieces. ((I have now guaranteed that I will get an avalanche of emails from EdTech startups claiming to solve this problem. Please direct your messages to my chatbot.)) This is the perennial challenge of EdTech, which, on balance, creates a vastly underestimated drag on the amount of time educators have to put into the thought work of delivering high-quality education. In principle, though, yes, I absolutely would. I have a clear picture in my head of what I would need in terms of the EdTech and what sorts of writing work I would (and wouldn’t) use it for.

    Do I think higher ed is ready for widespread adoption of these tools? That’s a harder question. Teaching this way requires a new skillset. Higher ed has abysmally under-resourced professional development support for teaching, on the whole. Also, teaching this way isn’t what our PhD system trains young academics to aspire to. Many will see it as a dumbing down of the work they have dedicated their lives to. And if implemented poorly, it can easily turn into that.

    So will AI text generation tools revolutionize or kill college writing? Both! Neither! For sure! Probably! Eventually! Somewhat! It’s…complicated.

    As usual.

  • Webinar on International Students on 12/7

    Webinar on International Students on 12/7

    When I talked to my friends at Open LMS about facilitating a webinar series, they gave me a lot of latitude to explore the aspects of their business that I find interesting, unusual, and valuable. The angle that caught my attention immediately is that, due to their Moodle heritage, they bridge a lot of silos. Which means that their user community bridges a lot of silos. Which, in turn, means that their user conferences, the first of which they are planning right now, present unusual opportunities to foster conversations among folks that normally aren’t in the same room together. What should those conversations be about? The first webinar was about bridging between universities and employers. The next webinar will be on international students.

    It turns out that only top-tier universities take advantage of the opportunity to recruit international students at a time when more of the world is looking for a US college degree. For example, the Institute for International Education (IEE) reports that community colleges are potentially missing an opportunity to boost their international enrollments by focusing on retention of those students.

    Community College international student sources, from IEE

    They can also partner with their four-year universities that have budgets to build recruitment engines. And there are potential opportunities for colleges and universities to partner with their overseas counterparts as well.

    In the upcoming webinar, I’ll be chatting with Dr. Lisa Avery, President of Linn-Benton Community College and Jon Boeckenstedt, Vice Provost of Enrollment Management at Oregon State University in Corvallis, who work together on just such efforts. We’ll be joined by my old friend Richard Garrett, Chief Research Officer at Eduventures and former Director of the Observatory on Borderless Higher Education. And I’ll be supported by my friend and fellow interlocutor Phill Miller, CEO of Open LMS.

    Up until now, international student recruitment has often been handled by a discrete office in certain types of universities. But as enrollment patterns change, it will increasingly become part of the whole campus—and inter-campus—tapestry that a wide range of US college and university stakeholders will become directly involved with in one way or another.

    It should be a fun conversation. Again, the session is Wednesday, December 7th, at 1 PM Eastern Time.

    Register here.