I sometimes get frustrated with the strong allergic reaction that even many good educators have to the request for hard data as part of an educational decision-making process. This allergy shows up in all different ways in all different places. In elementary school, for many years (and this may still be true today, for all I know), you could predict a person’s position on reading education based on their political disposition. Liberals liked whole language and conservatives liked phonics. James Moffett wrote a riveting if ultimately one-sided account of the conservative attachment to phonics (among many other things), with the more extreme proponents making claims like, “Phonics cured my daughter’s asthma!” (No, I’m not joking.) I have seen no corresponding documentation of the liberal version of this bias, but I have known many teachers, invariably liberals, who have vehemently asserted the value of whole language, where the only evidence they had was that most of the (middle-class) kids (from educated families) in their classrooms learn to read just fine. Nowhere in these debates was a discussion about whether it might be a good idea to construct some empirically rigorous tests to indicate which approach might provide what kinds of benefits under which circumstances. There are some things in education that do not lend themselves to rigorous empirical study, but this is not one of them—especially these days with the improvements we have in brain imaging equipment.
Another example: When I was a PhD student in English, I called a meeting of my fellow graduate students to discuss what we could do to improve the quality of our pedagogy. Mind you, this was at a department with a strong composition program that prided itself on its commitment to teaching. One suggestion I raised was that we could run a norming session for the English 101 final essays so that we could share best practices and ensure we were grading progress consistently across the program. This wasn’t a crazy suggestion from out of the blue; it was a successful practice that was already being employed in other universities. Yet you would have thought that I was asking everyone to get bar code tattoos on their foreheads. Somehow, taking such an step would violate their fundamental rights to individuality as professors, and besides, you can’t “norm” a thing like that. Teaching is an art, I was told.
The same sort of tension frequently creeps into educational technology conversations when it comes to anything remotely smacking of assessment, grading, or analytics. Test engines and grade books are derided as mere “management” tools, while retention early warning systems are apparently the first wave in the Rise of the Machines. As with the cases of phonics and final exam norming, these conversations are immensely frustrating to me in large part because I often find myself arguing with some of the talented, creative teachers who I respect the most. And I don’t always handle these situations well. (See for example, my somewhat snippy response to my friend Joe Ugoretz regarding the value of grade books and test engines or my regrettably snarky swipe at Jim Groom regarding the death of the LMS.) I think it’s mostly because I’m trying to convince myself that I’m not crazy, that I’m not some kind of Cylon or Terminator sleeper bot. Why am I the only teacher who sees it this way? What’s wrong with me?
It is therefore with great relief that I read Education Secretary Arne Duncan’s comment on the value of analytics:
Using technology to improve student achievement makes teachers feel almost as if “they’re cracking a code,” he explained. With adequate student data, teachers come to realize that effective instruction is not based on “just a guess or an assumption or a hunch, and all that is being driven by technology.”
Yes. That is how I feel.
I think there is a fear—a legitimate fear, bourne out by history and experience—that the bureaucracy will take the raw data as a substitute for judgment of teachers and students. I get that. It’s something to be fought vehemently. And maybe the tools we have today have more bureaucratic influence on their design than they should. But this is no reason to reject the value of data, or the scrutiny of peer review, or the use of tools that provide visible and measurable data regarding student activities. Like doctors and engineers, teachers are professionals. Nobody seriously imagines that the existence of an fMRI machine makes a doctor’s judgment less important. To the contrary, the more data we have, the more we benefit from the judgment of a trained and experienced expert.
Joe says
Michael, you’re not a bot or a borg. You’re trying to think things through–that doesn’t put you on the dark side! (or not quite).
I don’t think it’s a good idea to automatically reject all assessment or data. And I do think that knee-jerk rejection does happen. But I also think that there are things about teaching and learning that can’t be measured that way, where even the attempt is harmful.
And (as I’m sure you know) the tendency among administrators and agencies is to rely only on those quantifiable metrics.
But it’s also possible to romanticize the intangible, and to use it as an excuse for an avoidance of facing the hard questions that have to be faced to improve teaching and learning.
And yes, I’m contradicting myself–it’s an issue where (even though I sometimes make strong claims in one direction or another) I can see both sides with some ambivalence.
I think that if you look at Scholarship of Teaching and Learning, at projects like the Visible Knowledge Project, you see that there are opportunities for “thick” description and analysis–for treating teaching and learning as a subject worthy of intense scholarly research. That way you can have assessment and measurement, and still value complexity and innovation and intangibles.
jack says
“Nobody seriously imagines that the existence of an fMRI machine makes a doctor’s judgment less important.”
On the other hand, many people seriously imagine that the results of standardized tests should absolutely trump the judgement of classroom teachers. In fact there have been major political campaigns and legal machinations centered on removing autonomy from classroom teachers (I’m talking about public elementary and secondary education in the U.S., specifically).
Michael Feldstein says
That’s true, Jack, but it’s not a reason to close our eyes to tools and data that can help us improve our teaching. It’s up to the educational community to make the fMRI analogy and put these tools in their proper place. Certainly, our current U.S. Secretary of Education understands the proper role.
Stephen Downes says
Point taken, but on the other hand, we need to be rigorous in our questioning of what counts of data and of what counts as a legitimate inference from that data.
A lot of the data presented from various sources seems to me no more relevant than data that would be produced through phrenology or tea-leaf reading.
This includes not only the misuse of standardized tests, but also research involving studies less than a single class in size, Pew reports that sample only Americans, brain scans and other arcane measurement systems, opinion surveys, race-based studies, and more.
As for inferences, it’s open season on logic. I see conclusions in research papers wildly unsupported by the data, the widespread use of equivocation and ambiguity, and more. The mere fact that you have data does not license any opinion you happen to have.
Though your own work is far more rigorous than most people’s, you nonetheless far prey to some of these illegitimate tactics. For example:
– you write, “construct some empirically rigorous tests to indicate which approach might provide what kinds of benefits under which circumstances.” This is spoken as though some such test has been devised. What might count as an “empirically rigorous test” under such circumstances? What, even, would count as “better”? The one side talks a lot about reading being a form of “decoding”. The other, about reading as comprehension. For myself, I would want to look at transferability of underlying skills, such as abstraction and formalism, into other disciplines.
– you write “One suggestion I raised was that we could run a norming session for the English 101 final essays so that we could share best practises and ensure we were grading progress consistently across the program.” This is based on a (a priori?) presumption that practises are linearly ordered such that one (or some?) could constitute “best” practises.
Now I happen to agree that “With adequate student data, teachers come to realize that effective instruction is not based on ‘just a guess or an assumption or a hunch, and all that is being driven by technology.'” I think that basing one’s work and beliefs on the evidence is important.
But I also reserve the right to be choosey about my evidence. My view of education is that it is a complex discipline, that it won’t be understood through simple cause-effect relationships, through “best practise” and other superficial generalizations. I believe that because that is where the evidence has taken me. The data tell me not to simply rely on the data.
Michael Feldstein says
As you say, Stephen, point taken, but I think you take it a bit too far. When I write, “It might be a good idea to construct” and you reply that this is “spoken as though some such test has been devised” and that I have “fallen prey” to an illigegitimate tactic”, I would say that you are reading the verb tense incorrectly and the intent uncharitably. Yes, there are different goals that could be identified for reading. I have myself often commented on the absurdity of the implied position that student need to be taught to either sound out words or decode their meaning but not both. Regardless, the range of cognitive tasks involved in early reading acquisition should generally be testable. I don’t think there’s anything illegitimate about asking for each side to justify claims of effectiveness by proving that their pedagogical methods achieve whatever goals that they purport to achieve.
Likewise, the norming suggestion was not based on an “(a priori?) presumption that practices are linearly ordered such that one (or some?) could constitute ‘best’ practices.” Once again, I called for questions to be empirically investigated and you have interpreted that call as a presumption that answers have already been found. This reflects another set of rhetorical tactics that we must be careful to not fall prey to, i.e., the tactic of equating any call for empirical evidence with bureaucratic reductivism.
John Rodgers says
After working with teachers for a short time with the belief that collaboration, discussion and data could improve our overall practice, I too wondered why there was such significant resistance to any such suggestions. I eventually came to realize that teachers seldom followed the forms they have been told over and over are essential components of good teaching, and they were reluctant to be exposed. (If keeping a day planner reduced student learning, how would an administrator ever find out).
Data I collected suggested note taking and unit tests weren’t as effective as other more active learning strategies, and in the end were an ineffective use of time (based on the results of, yes, standardized testing). Yet I think that positive data would play a secondary role to the absence of the traditional expectations of proper “teaching form” when evaluating the practice (Imagine sitting with a group of math teachers and blurting that out when your turn to share comes along).
Jim says
You are both a cylon and a terminator sleeper bot 🙂