e-Literate

Present is Prologue

Tag: research

  • Empowering Students in Open Research

    Phil and I will be writing a twice-monthly column for the Chronicle’s new Re:Learning section. In my inaugural column, “Muy Loco Parentis,” I write about how schools make data privacy decisions on behalf of the students that the students wouldn’t make for themselves, and that may even be net harmful for the students. In contrast to the ways in which other campus policies have evolved, there is still very much a default paternalistic position regarding data.

    But the one example that I didn’t cover in my piece happens to be the one that inspired it in the first place. A few months back at the OpenEd conference, I heard a presentation from CMU’s Norm Bier about that challenges of getting different schools to submit OLI student data to a common database for academic research. Basically, every school that wants to do this has to go through its own IRB process, and every IRB is different. Since the faculty using the OLI products usually aren’t engaged in the research themselves, it generally isn’t worth the hassle to go through this process, so the data doesn’t get submitted and the research doesn’t get done. Note that Pearson and McGraw Hill do not have this problem; if they want to look at student performance in a learning application across various schools, they can. Easily. Something is wrong with this picture. I proposed in Norm’s session that maybe students could be given an option to openly publish their data. Maybe that would get around the restrictions. David Wiley, who does a lot more academic research than I do, seemed to think this wasn’t a crazy idea, so I’ve been gnawing on the problem since then.

    I have talked to a bunch of researchers about the idea. The first reaction is often skepticism. IRB is not so easy to circumvent (for good reason). What generally changed their minds was the following thought experiment:

    • Suppose that, in some educational software program, there was a button labeled “Export.” Students could click the button and export their data in some suitably anonymized format. (Yes, yes, it is impossible to fully de-identify data, but let’s posit “reasonably anonymized” as assessed by a community of data scientists.) Would giving students the option to export their data to any server of their choosing trigger the requirement for IRB review? [Answer: No.]
    • Suppose the export button offered a choice to export to CMU’s research server. Would giving students that option trigger the requirement for IRB review? [Answer: Probably not.]

    There are two shades of gray here that are complications. First, researchers worry about the data bias that comes from opt in. And the further you lead students down the path toward encouraging them to share their data, such as making sharing the default, the more the uneasiness sets in. Second and relatedly, there is the issue of informed consent. There was a general feeling that, even if you get around IRB review, there is still a strong ethical obligation to do more than just pay lip service to informed consent. You need to really educate students on the potential consequences of sharing their data.

    That’s all fair. I don’t claim that there is a silver bullet. But the thought experiment is revealing. Our intuitions, and therefore our policies, about student data privacy are strongly paternalistic in an academic context but shift pretty quickly once the institutional role fades and the student’s individual choice is foregrounded. I think this is an idea worth exploring further.

  • Promising Research Results On Specific Forms Of Adaptive Learning / ITS

    Recently I described an unpublished study by Dragan Gasevic and team on the use of Knowillage / LeaP adaptive platform. ((When the study started Knowillage was an independent company; mid-way through study D2L bought Knowillage and renamed product as LeaP.)) The context of article was on D2L’s misuse of the results, but the study itself is interesting in terms of its findings that adaptive learning usage (specifically LeaP in addition to Moodle within an Intro to Chemistry course) can improve academic performance. I will share more when and if the results become public.

    If we look to published research reports there are other studies that back up the potential of adaptive approaches, but the most promising results appear to be for a subset of adaptive systems that provide not just content selection but also tutoring. Last year a research team from Simon Fraser University and Washington State University published a meta-analysis on Intelligent Tutoring Systems (ITS) which they described as having origins from 1970 and the development of SCHOLAR. ((I would link to G+ post by George Station here if it were not for the ironic impossibility of searching within that platform.)) The study looked at 107 studies involving 14,321 participants and found:

    The use of ITS was associated with greater achievement in comparison with teacher-led, large-group instruction (g .42), non-ITS computer-based instruction (g .57), and textbooks or workbooks (g .35). There was no significant difference between learning from ITS and learning from individualized human tutoring (g –.11) or small-group instruction (g .05). Significant, positive mean effect sizes were found regardless of whether the ITS was used as the principal means of instruction, a supplement to teacher-led instruction, an integral component of teacher-led instruction, or an aid to homework. Significant, positive effect sizes were found at all levels of education, in almost all subject domains evaluated, and whether or not the ITS provided feedback or modeled student misconceptions. The claim that ITS are relatively effective tools for learning is consistent with our analysis of potential publication bias.

    (more…)

  • Comment from member of research team on USA Today flipped classroom article

    Update 10/26: We now have Rachel Levy and Nancy Lape (who was the researcher interviewed by USA Today) both agreeing with Darryl’s comments. That’s three of the four members of the research team. While I do not claim to understand how the reporter developed her story line (I have asked for comment), it is quite clear that the article does not represent the work of the Harvey Mudd research team and has a misleading headline and lede.

    As a follow up to my response to the USA Today article on flipped classroom research, there was a very informative comment one Google+ from Darryl Yong (one of the members of the Harvey Mudd research team). I thought this comment deserved to be seen by a wider audience, so I’m reproducing in full.

    Thanks, Phil, for this post. The USA Today article paints an inaccurate picture of our work and I wanted to try to clarify some things and continue this conversation with you and your readers. (I am only writing on my own behalf and not for my collaborators on this study.)

    My biggest regret is that the article greatly oversimplifies things by portraying our study as an attempt to answer whether flipped classrooms work or not. That kind of research question is too blunt to be useful. Our goal is to better understand the conditions under which flipped classrooms lead to better student outcomes. As Phil and others here point out, there are many different manifestations of what we mean by the term “flipped classroom” and that we should be wary about talking about it as if there is one canonical implementation. How are the benefits of flipped classrooms affected by school and student contexts, and are there other benefits that we haven’t yet characterized? We don’t have any preconceived answers to these research questions. While folks might disagree with us whether these research questions are interesting or not, they should at least know that we’re aware of the good work that others have already done and we’re trying to build on it rather than debunk it.

    (more…)

  • A response to USA Today article on Flipped Classroom research

    Update 10/25: Bumped comment from Darryl Yong, a member of the research team, into its own post here.

    Update 10/26: We now have Rachel Levy and Nancy Lape (who was the researcher interviewed by USA Today) both agreeing with Darryl’s comments. That’s three of the four members of the research team. While I do not claim to understand how the reporter developed her story line (I have asked for comment), it is quite clear that the article does not represent the work of the Harvey Mudd research team and has a misleading headline and lede.

    USA Today published an article today titled “‘Flipped classrooms’ may not have any impact on learning“, based on research from four Harvey Mudd professors. This research is backed by a “$199,544 grant from the National Science Foundation to study the effects of the flipped classroom on students’ learning”. This is newsworthy, right? A real research report with NSF funding finding no statistical difference in learning outcomes from flipped classroom seems to contradict much of the recent promise of ed tech.

    Upon closer reading, however, there are some major problems with the story. Exaggerated claims by ed tech enthusiasts are not helpful, but neither are exaggerated claims by ed tech skeptics. We at e-Literate have been critical of both flavors (witness our analysis of San Jose State claims, Desire2Learn claims, and edX claims for examples of the former).

    Let’s review today’s story as an example of the latter.

    In a flipped classroom, students watch their professors’ lectures online before class, while spending class time working on hands-on, “real world” problems.

    The potential of the model has many educators thrilled — it could be the end of vast lecture halls, students falling asleep and boring, monotone professors.

    This is a decent summary, although I would argue there should not be a one-size-fits-all mentality. Flipped classrooms, even where successful, should not replace all lecture-based classes. But for national media, this is not a bad description.

    But four professors at Harvey Mudd College in Claremont, Calif. who are studying the effectiveness of a flipped classroom have bad news for advocates of the trend: it might not make any difference.

    This is the lede, and the claim that we should examine. What is the basis of their study?

    (more…)