Promising Research Results On Specific Forms Of Adaptive Learning / ITS

Recently I described an unpublished study by Dragan Gasevic and team on the use of Knowillage / LeaP adaptive platform.1 The context of article was on D2L’s misuse of the results, but the study itself is interesting in terms of its findings that adaptive learning usage (specifically LeaP in addition to Moodle within an Intro to Chemistry course) can improve academic performance. I will share more when and if the results become public.

If we look to published research reports there are other studies that back up the potential of adaptive approaches, but the most promising results appear to be for a subset of adaptive systems that provide not just content selection but also tutoring. Last year a research team from Simon Fraser University and Washington State University published a meta-analysis on Intelligent Tutoring Systems (ITS) which they described as having origins from 1970 and the development of SCHOLAR.2 The study looked at 107 studies involving 14,321 participants and found:

The use of ITS was associated with greater achievement in comparison with teacher-led, large-group instruction (g .42), non-ITS computer-based instruction (g .57), and textbooks or workbooks (g .35). There was no significant difference between learning from ITS and learning from individualized human tutoring (g –.11) or small-group instruction (g .05). Significant, positive mean effect sizes were found regardless of whether the ITS was used as the principal means of instruction, a supplement to teacher-led instruction, an integral component of teacher-led instruction, or an aid to homework. Significant, positive effect sizes were found at all levels of education, in almost all subject domains evaluated, and whether or not the ITS provided feedback or modeled student misconceptions. The claim that ITS are relatively effective tools for learning is consistent with our analysis of potential publication bias.

Relationship of ITS and Adaptive Learning Software

Unlike most marketing and media descriptions of Adaptive Learning, the report is quite specific on defining what an Intelligent Tutoring System is and isn’t.

An ITS is a computer system that for each student:

1. Performs tutoring functions by (a) presenting information to be learned, (b) asking questions or assigning learning tasks, (c) providing feedback or hints, (d) answering questions posed by students, or (e) offering prompts to provoke cognitive, motivational or metacognitive change
2. By computing inferences from student responses con- structs either a persistent multidimensional model of the student’s psychological states (such as subject matter knowledge, learning strategies, motivations, or emotions) or locates the student’s current psychological state in a multidimensional domain model
3. Uses the student modeling functions identified in point 2 to adapt one or more of the tutoring functions identified in point 1

There are plenty of computer-based instruction (CBI) methods out there, but ITS relies on a multidimensional model of the student in addition to a model of the subject area (domain model). The report also calls out that CBI approaches that only model the student in one dimension of item response theory (IRT, more or less the model of a student’s ability to correctly answer specific questions) are not ITS in their definition. IRT can be one of the dimensions but not the only dimension.

A 2014 meta-analysis referred to by the above report further clarifies the conditions for a system to be an ITS as follows [emphasis added]:

VanLehn (2006) described ITS as tutoring systems that have both an outer loop and an inner loop. The outer loop selects learning tasks; it may do so in an adaptive manner (i.e., select different problem sequences for different students) based on the system’s assessment of each individual student’s strengths and weaknesses with respect to the targeted learning objectives. The inner loop elicits steps within each task (e.g., problem-solving steps) and provides guidance with respect to these steps, typically in the form of feedback, hints, or error messages.

For the sloppy field of Adaptive Learning, this means that the study looks at systems that model students, provide immediate feedback to students, and provide hints and support to students as they work through a specific task (inner loop). Adaptive Learning systems that only change the content or tasks presented to students adaptively (outer loop) do not qualify. Some examples of Adaptive Learning / ITS systems include McGraw-Hill’s ALEKS and AutoTUTOR. Knowillage / LeaP is an example of a system that is not an ITS.

Promising Findings

The results showed “the use of ITS produced moderate, statistically significant mean effect sizes” compared to large-group human instruction, individual CBI, and textbooks / workbooks. The results showed no statistically significant mean effect sizes compared to small-group human instruction and individual tutoring. In other words, the study shows improvements of ITS over large lecture classes, non-ITS software tools, and textbooks / workbooks but no real difference with small classes or individual tutors.

What is quite interesting is that the results hold across multiple intervention approaches. Using ITS as Principal instruction, Integrated class instruction, Separate in-class activities, Supplementary after-class instruction, or Homework give similar positive results.

Why Does ITS Give Positive Results?

The report hypothesizes that the primary reasons that ITS seems to provide positive results as follows [formatting added, excerpted]:

[ITS shared characteristics with other forms of CBI] Specifically, they have attributed the effectiveness of CBI to:

• greater immediacy of feedback (Azevedo & Bernard, 1995),
• feedback that is more response-specific (Sosa, Berger, Saw, & Mary, 2011),
• greater cognitive engagement (Cohen & Dacanay, 1992),
• more opportunity for practice and feedback (Martin, Klein, & Sullivan, 2007),
• increased learner control (Hughes et al., 2013), and
• individualized task selection (Corbalan, Kester, & Van Mer- riënboer, 2006).

[snip] The prior quantitative reviews also concluded that using ITS is associated with greater achievement than using non-ITS CBI. We hypothesize that multidimensional student modeling enables ITS to outperform non-ITS CBI on each of its advantages cited in the previous paragraph.

[snip] ITS may also be more effective than non-ITS CBI in the sense that ITS can extend the general advantages of CBI to wider set of learning activities. For example, the ability to score and provide individualized comments on a student’s essay would extend the advantage of immediate feedback well beyond what is possible in non-ITS CBI. This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

Student modeling also enables ITS to interact with students at a finer level of granularity than test-and-branch CBI systems.

These are very encouraging results for the field of ITS and a subset of Adaptive Learning. I view the results not as saying adaptive learning is the way to go but rather as there is evidence that adaptive learning working applied in a tutoring role can improve academic performance in the right situations.

We need more evidence-based evaluation of different teaching strategies and edtech applications.