Information Sheet #157

February 8, 2011

The Myth of Improvement


[The following passages are from the “The Emperor (Still) Has No Clothes: Revisiting the Myth of Improvement” by Cy Knoblauch and Lil Brannon--the introductory essay in the anthology Key Works on Teacher Response, edited by Richard Straub and published by Boynton/Cook in 2006. The anthology assembles the results of two generations of researchers seeking to demonstrate the benefits of faculty responses to student writing in post-secondary education.  As Knoblauch and Brannon note, we currently lack empirical evidence that faculty responses to student writing have any demonstrable impact on students’ writing skills.   ~Bob Marrs]

How odd to think that a practice as intuitively sensible as commenting on student texts in order to facilitate more and better writing should have occasioned such a labor of scholarly justification. And how disconcerting to find, after all the theorizing, the case studies, the controlled experiments, not to mention all the dutiful and well-intentioned scribbling of millions of teachers in the margins of billions of student essays over the same half century, that the matter remains unresolved. Yet that appears, at least to us, to be the state of things. We find nothing in our own experience as teachers, and nothing in the accumulated research, to alter the fundamental impressions we formed twenty-five years ago in our own contributions to the literature. First, there is scant evidence that students routinely use comments on one draft to make rhetorically important, and qualitatively superior, changes in a subsequent draft, although students will make limited, usually superficial corrections in order to comply with overt or tacit instructions. Second, there is still less evidence to show that they change their practices from one assignment to the next in ways that measurably represent or affect their development as writers. And third, the very possibility of acquiring such evidence is compromised by the imperfect assessment instruments available for the task. . . .

The response literature is reminiscent of the old folk tale "The Emperor's New Clothes," where, despite ample cause for skepticism, there is powerful incentive for everyone to participate in sustaining an illusion. In this version of the tale, a reassuring narrative about the "improvement” of writing ability as a result of teacher commentary belies a persistently unconvincing demonstration that it occurs. The more speculative and theoretical scholarship gives lip service to the concept of improvement, tending to evade direct questions about changed performance while subtly invoking "development" as the dividend from some recommended instructional change. The empirical research, meanwhile, bravely puts the issue of changed performance at center stage but then repeatedly finds itself obliged to settle for local, mostly cosmetic or mechanical revision of student writing as its best case for the value of commentary. What passes for "development" in student writers, in this research, is little more than their willingness to follow a teacher's directions for correcting errors.

The root of the problem, we believe, is precisely the narrow focus on performance that has dominated the literature. . . . There is a perfectly sensible retort to the morose conclusions with which we opened: Of course, learners respond superficially to teacher comments, draft to draft, and of course they change little as writers, if at all, from one assignment to the next. That's why they're called learners. If people were willing to adopt a more patient attitude toward learning, the result could be a new infusion of common sense in the discussion of what students and teachers can reasonably achieve in the course of three or four months of instruction. Only writers who are already mature and accomplished (including, of course, some students) will self-assuredly sort through reader reactions, choosing some, rejecting others, allowing the best to influence substantive revision. The less mature, including most students, need time to grow. In any case, commentary doesn't create better writers; at best, it may create better writing—and then only when the writer is already skilled enough to take advantage.

Unfortunately, however, lots of people aren't comfortable with such a sensible (they might say, "defeatist") point of view. Some teachers aren't comfortable because, having put a lot of sweat and tears into the burdensome task of grading essays, they would prefer to think that their efforts will be reliably compensated. They point to anecdotes of dramatic change (successes do certainly occur) and extrapolate to the conviction that "most" students routinely learn to write in their classrooms. Some scholars aren't comfortable either, because, if prompt and steady improvement can't be observed as an index of instructional success, then what defines "effective" pedagogy? Why should one pedagogical regimen, or writing assignment, or style of commentary, be preferred over another? Taxpayers and public officials, meanwhile, aren't the least bit comfortable, because, if short-term improvement isn't measurable, then how can anyone know that something useful is going on, how can there be accountability in schools? Surely, a lack of improvement in students must mean poor teaching, poor curricula, and/or poor use of resources, and surely there are ways to engineer learning in visible increments to assure everyone that schools "work." Not surprisingly, then, no matter what the research has been able or unable to confirm, there is substantial pressure to stifle any frank concession that students seldom improve quickly as writers. As a result, educational public policy, literacy instruction, and even, to some extent, the research literature itself all continue to insist that the emperor is clothed.

Janet Emig explored this durable public self-deception more than twenty years ago, calling it "magical thinking," essentially a belief that what teachers teach reliably determines what students learn. We have referred to it elsewhere as the "myth of improvement," by which we mean a belief that particular teaching activities cause identifiable advances in learning in a smoothly upward trajectory over specific increments of time. Educational research has recognized the myth for a long time, but the public generally accepts it as true, so that arguments about educational effectiveness continue to be framed in its terms. There's a well-established, two-part response to the question, why is the myth of improvement a myth? The first part, thoroughly documented, is that human development is unpredictable, irregular, and time-consuming; the second, similarly documented, is that, while no one doubts that it occurs, and that teachers can help, it is difficult to measure in small increments.

We recall a pointed anecdote from Jimmy Britton, the renowned British scholar-educator, who once likened the American preoccupation with educational measurement to an empirically curious gardener who pIants a new seedling. The gardener digs a large bed, fills it with rich earth, loads it with nutrients, inserts the tree, waters it daily, protects it from the elements—and then digs it up every six months or so to see how its roots are doing! The romantic metaphors of organism and growth may not suit the taste of the contemporary analytical palate, but Britton's story draws attention to something obsessive in our zeal to identify small increments of development over short periods of time in human behaviors as subtle as the practices of language.

On the one hand, just as the gardener uses recognized techniques to nurture a plant, so there are no profound mysteries regarding the appropriate strategies for supporting the development of reading and writing abilities: students need frequent opportunities to read and write; they need supportive responses to their efforts; they need chances to analyze their misreadings or to revise their writing; and they need time, commonly years of time, for these experiences (not to mention many others outside the classroom) to have cumulative impact.

On the other hand, there is nothing but mystery regarding the subtle developmental pathways and timetables involved: readers and writers don't all progress in the same ways, or at the same rates, or at the same times, in response to the same specific instructional stimuli. The curious gardener, of course, is luckier than teachers are because she can at least discover a lengthening of the roots. Her problem is a lack of faith in her work and a neurotic inquisitiveness that, once satisfied, does more harm than good to the growth of the plant. One could say as much about the mania for writing assessment, but the problem is even worse in education: we know what we need to do in order to develop literate practices, but we don't have fully reliable ways to measure the results. And the shorter the time interval, the less reliable the measurement.

Identifying improvement has always been, and remains, a dubious enterprise in the language arts. Thirty years ago, Paul Diederich of the Educational Testing Service explained what can and cannot realistically be accomplished in writing assessment. Nothing in the considerable research effort since then has shown Diederich to be significantly incorrect in his conclusions. To be sure, he observed, raters can be trained to evaluate written texts. With appropriate preparation, and assuming a limited range of evaluative options (good, average, poor), they will make roughly similar judgments. But there are two important caveats. First, evaluation instruments cannot be freed from some degree of subjectivity. Raters may believe they are looking through a textual window at the writer, but in fact they are looking in a textual mirror at themselves: what they see derives from the values and predispositions they bring to the reading. The degree of subjectivity is proportionate to the subtlety of the feature being evaluated. Raters can usually agree about a sentence fragment; they are less likely to agree about an effective use of irony.

Second, supposing that evaluation can provide a crude snapshot of a writer's competence at a given moment in time (a reasonable presumption if the protocol is sound), nothing in that success implies an ability to distinguish "progress" between two such moments if the interval between them is as short as the duration of a course or academic program. In other words, even if it seems reasonable to assume that a writer has "matured" (or at least changed) between the first and last assignment of the semester, we don't have instruments sufficiently refined to detect, let alone identify, the maturation. Given that reality, why should anyone be surprised that research hasn't been able to identify the effectiveness of teacher commentary?

Despite the efforts of a generation of scholars—Emig, Britton, and Diederich among many—to "expose" the truth about the emperor's clothes, the myth of improvement remains alive and well in educational culture, impressively resistant to challenge. If sustaining the myth had no more serious consequence than to motivate teachers and students to persevere in the rich variety of reading and writing activities that, over the long haul, do indeed enhance competence, there would be little harm in it. Teachers regularly and wisely cook their books, grading more sternly at the start than at the end of a term, giving extra points for multiple revisions or bonus assignments, rewarding effort because they appreciate that learners need to believe they're making progress. No harm, no foul. More seriously, however, and less consciously, teachers, mistake obedience for development, sometimes trick themselves into rewarding students who are merely canny enough to write in the ways that School requires. That's acceptable for the students who guess correctly, but less so for those who don't or won't.

Most seriously, the myth has broad consequences for educational public policy. For one thing, it stimulates a voracious public appetite for educational "results," where the instant gratification that American culture has come to expect from credit cards, cell phones, and fast-food restaurants carries into the classroom as well, now as early as prekindergarten. It encourages impatient politicians, their reelections riding on promises to fix things, who eagerly mandate "value-added" testing as though making it a legal requirement were sufficient to make it meaningful. The myth guarantees additional power—and profit—for a rapidly expanding assessment industry, which offers, through the quantifying of teaching and learning, a positivist mirage about the feasibility and benefits of an engineered society. Driven by assessment mandates, curriculum designers have discovered the advantages of concocting programs of study that are best able to deliver measurable results—by emphasizing what Ann Berthoff used to call "muffin tin!” forms (like the five-paragraph essay) and mechanical correctness (like avoiding the comma splice), because multiple observers will be better able to agree on whether students have incorporated such recognizable features into their texts and assess their performance accordingly. The myth of improvement is not benign, and it needs to be challenged.

This website created and maintained by the Coe Writing Center. Copyright 2001.
Email Dr. Bob Marrs with any questions, comments or suggestions.