Long-time e-Literate readers know that I have been a fan of the concept of learning analytics for a number of years now. But it became apparent at this year’s Learning Impact conference that learning analytics are the new hotness. Everybody is talking about them, and increasing numbers of vendors (LMS vendors, ERP vendors, textbook vendors, etc.) are trying to figure out how to get in on the party.
Which means it’s probably time to start asking some critical questions about how well we really understand learning analytics and where the potential for failure and disappointment might be.
The area that has gotten the most attention for learning analytics lately is meta-cognition. Purdue’s Signals program showed us that at risk students often lack ability recognize when they need help. This is a skill can be taught by providing students with learning analytics. At-risk students who use Signals have significantly higher retention rates and show improved grades over students who didn’t have access to the program. And of course, mastery learning is an old concept that can easily be enhanced in a digital world to help students recognize which skills they have down and which ones need more work. The latest hot fads in the rhetoric of learning analytics are Amzon.com-like recommendations (“students who passed this competency used these learning objects…”) and game mechanics (“Level Up!!!!!”). Both of these were mentioned on an analytics panel at Learning Impact. But a funny thing happened when the two were juxtaposed. One of the panelests pointed out, as a way of touting the virtues of the Amazon.com-like approach, that you generally don’t get a message in a game that says, “Knights who slew this giant used the +3 flaming sword.”
But actually, you do see that kind of information in the gaming world. There’s even a name for it. It’s called “cheating.” In games, it’s precisely what you don’t know that makes them fun. Consider the words of master game designer Raph Koster, from his book A Theory of Fun for Game Design (which I reviewed quite a while back):
Games grow boring when they fail to unfold new niceties in the puzzles they present. But they have to navigate between the Scylla and Charybdis of deprivation and overload, of excessive order and excessive chaos, of silence and noise….
If your goal is to keep things fun (read as “keep the player learning”), boredom is always the signal to let you know you have failed.
The definition of a good game is therefore “one that teaches everything it has to offer before the player stops playing.”
That’s what games are, in the end. Teachers. Fun is just another word for learning.
One wonders, then, why learning is so damn boring to so many people. It’s almost certainly because the method of transmission is wrong. We praise good teachers by saying that they “make learning fun.” Games are very good teachers…of something. The question is, what do they teach?
Koster is really onto something. When we tell stories about great teachers, they tend to be similar to our stories of great artists. Success is characterized as the result of an ineffable talent rather than learnable skill. But I don’t think that’s right. Consider this bit of analysis from an excellent article on game design “seduction secrets” in The Guardian:
“An effective learning environment, and for that matter an effective creative environment, is one in which failure is OK – it’s even welcomed,” Koster says via phone from his hometown of San Diego. “In game theory, this is often spoken of as the ‘magic circle’: you enter into a realm where the rules of the real world don’t apply – and typically being judged on success and failure is part of the real world. People need to feel free to try things and to learn without being judged or penalised.”
Consistently, he says, the most successful games are the ones that provide us with interesting tools such as weapons or magic (or even angry birds) and allow us time to experiment with them. He provides as a defining example the 1985 platforming game Super Mario Bros, created by Nintendo’s renowned game designer Shigeru Miyamoto. On the first screen, players are given the ability to jump and can play with this for as long as they like, but to get to the next stage, they need to have mastered the skill so they can leap over an enemy and on to a platform. Afterwards, they learn about hidden bonuses and items, but only when each new addition has been perfected.
This “acquire, test, master” model is still intrinsic to game design. The recently released Portal 2, a brilliant, physics-based puzzler set in an abandoned science research facility, works in exactly the same way. Here, players wield a portal gun, a device that creates dimensional wormholes in walls, floors and ceilings – but they’re only introduced to one facet of the gun at a time, and when it has been mastered, new items such as super-bouncy gels are introduced. There is constant progress and a continually evolving challenge, but there is always room to experiment and to figure things out through intuition.
Acquire-test-master is not that different from the pedagogical pattern predict-observe-explain. (For more on this and a couple of other pedagogical patterns, see James Dalziel’s Practical e-Teaching Strategies site.)
Or consider this video:
Here is part of what the student featured in the video wrote as a comment on it:
NOTE: The physics in this might not be 100% correct, but this was more for fun than actual science/calculations. Also we had to give it a rating for how physically correct it was, I only used XP and RP which basically meant physics outside this universe.
Yes, it has happened. “Physical Impossibilities in My Little Pony: Friendship is Magic.” For our project, we had to find three scenes from any movie or TV show and use physics to find out if something was or wasn’t possible. I got 100% on it.
Think about his comment for a moment. “…[T]his was more for fun than for actual science….I got 100% on it.” That’s not a grade he’s talking about, at least not in his mind. It’s a game score. He was given some rules of the game, some autonomy to pick a goal, some puzzles to solve and tools with which to solve them. Let’s not get too carried away on the autonomy part of this, either. There’s a popular canard that students only gain satisfaction if they are learning stuff that is intrinsically meaningful to them. But what could possibly be intrinsically meaningful about learning how to get better at Tetris? Or Minesweeper? Or Angry Birds? The thing is, learning is inherently fun. That’s why humans invented games. Like reproduction, nature has wired us to take pleasure in an activity that promotes survival of the species. There’s something about the way we’re approaching schooling that tends to break that natural urge. We need to approach this problem scientifically and figure out what are the elements that make learning fun so that we can stop accidentally smothering them. Human factors expert Charles Mauro provides us with one good example of what this sort of analysis might look like in his cognitive teardown of Angry Birds. Here’s a sample:
It is a well-known fact of cognitive science that human short-term memory (SM), when compared to other attributes of our memory systems, is exceedingly limited. This fact has been the focus of thousands of studies over the last 50 years. Scientists have poked and prodded this aspect of human cognition to determine exactly how SM operates and what impacts SM effectiveness. As we go about our daily lives, short-term memory makes it possible for you to engage with all manner of technology and the environment in general. SM is a temporary memory that allows us to remember a very limited number of discrete items, behaviors, or patterns for a short period of time. SM makes it possible for you to operate without constant referral to long-term memory, a much more complex and time-consuming process. This is critical because SM is fast and easily configured, which allows one to adapt instantly to situations that might otherwise be fatal if one were required to access long-term memory. In computer-speak, human short-term memory is also highly volatile. This means it can be erased instantly, or more importantly, it can be overwritten by other information coming into the human perceptual system. Where things get interesting is the point where poor user interface design impacts the demand placed on SM. For example, a user interface design solution that requires the user to view information on one screen, store it in short-term memory, and then reenter that same information in a data field on another screen seems like a trivial task. Research shows that it is difficult to do accurately, especially if some other form of stimulus flows between the memorization of the data from the first screen and before the user enters the data in the second. This disruptive data flow can be in almost any form, but as a general rule, anything that is engaging, such as conversation, noise, motion, or worst of all, a combination of all three, is likely to totally erase SM. When you encounter this type of data flow before you complete transfer of data using short-term memory, chances are very good that when you go back to retrieve important information from short-term memory, it is gone!
One would logically assume that any aspect of user interface design that taxes short-term memory is a really bad idea. As was the case with response time, a more refined view leads to surprising insights into how one can use the degradation of short-term memory to actually improve game play engagement. Angry Birds is a surprisingly smart manager of the player’s short-term memory.
By simple manipulation of the user interface, Angry Birds designers created significant short-term memory loss, which in turn increases game play complexity but in a way that is not perceived by the player as negative and adds to the addictive nature of the game itself. The subtle, yet powerful concept employed in Angry Birds is to bend short-term memory but not to actually break it. If you do break SM, make sure you give the user a very simple, fast way to accurately reload. There are many examples in the Angry Birds game model of this principle in action. Probably one of the most compelling is the simple screen flow manipulation at the beginning of each new play sequence. When the screen first loads, the user is shown a very quick view of the structure that is protecting the pigs. Just as quickly, the structure is moved off screen to the right in a simple sliding motion.
Coming into view on the left is a bevy of bouncing, chatting and flipping birds sitting behind the slingshot. These little characters are engaging in a way that for the most part erases the player’s memory of the structure design, which is critical to determining a strategy for demolishing the pig’s house. Predictably, the user scrolls the interface back to the right to get another look at the structure. The game allows the user to reload short-term memory easily and quickly. Watch almost anyone play Angry Birds and you see this behavior repeated time and again. One of the main benefits of playing Angry Birds on the iPad is the ability to pinch down the window size so you can keep the entire game space (birds & pigs in houses) in full view all the time. Keeping all aspects of the game’s interface in full view prevents short-term memory loss and improves the rate at which you acquire skills necessary to move up to a higher game level.
Mauro reveals both that the difficulty due to lack of knowledge is an intrinsic part of the fun and that sometimes players like to dial down the level of difficulty. This gives us one clue as to how we might have to think about learning analytics. It might be important in certain circumstances to give students the option to not view the analytics. But the deeper point here is that we need to research the connection between specific cognitive processes and the motivation to learn. We need a theory of fun for learning design. Without this understanding, we can throw students some dashboards that may help them complete their schooling, but we won’t really be teaching them the intrinsic pleasure of learning. And without that, it’s going to be really hard for them to be successful in a world in which skills are constantly becoming obsolete and new knowledge is being generated at a blinding pace.
[…] What We Don’t Know About Learning Analytics […]