May 19, 2004
Indiana Essays Being Graded by Computers
NDIANAPOLIS - In the computer lab at Warren Central High School in mid-May, Craig Butler, a junior, squinted at the question on his screen, paused to ponder his answer and began typing.
Craig was one of 48,500 Indiana juniors gathering in high schools across the state to take the end-of-year online English essay test. Unlike most essay tests, however, this one is being graded not by a teacher but by a computer.
Craig has already decided he prefers computer grading. "Teachers, you know, they're human, so they have to stumble around telling you what you need to do," he said at a practice session. "A computer can put it in fine print what you did wrong and how to fix it."
But his English teacher, Richard P. Dayment, wonders whether the computer is up to the task. "For the computer to do the subjective grading that's necessary on an essay, I'll want to see it before I have faith in it," he said.
Indiana is the first state to use a computer-scored English essay test in a statewide assessment, and its experience could influence testing decisions in other states. Eighteen states now require students to pass a writing test for high school graduation, and, starting next year, both the SAT and the ACT will include writing in their college admission exams.
"In five years at least 10 more states will be at or beyond the pilot stage" of automated essay scoring, predicts Richard Swartz, executive director of technology products and services at the Educational Testing Service, designers of Indiana's online essay-grading software.
While Indiana's essay test is not a pass-fail "high stakes" test, it is part of an assessment of student achievement in the 40-credit state curriculum, known as Core 40, recommended by Indiana educators and business leaders as preparation for success in college and the work force. Scores on the Core 40 tests, offered for the first time this year in English and algebra, will help determine college readiness and course placement for students and the performance ratings of high schools.
With the increasing number of mandates to test student writing, "there's a certain inevitability to computerized essay grading," said Stan Jones, Indiana's commissioner of higher education. Indiana's computerized essay scoring, he said, will reduce by half the cost of administering a pencil-and-paper test and will free teachers from distributing, collecting and, above all, grading thousands of test booklets.
Moreover, automated grading will yield almost instant results, allowing teachers to provide immediate feedback to their students. It would take weeks or months to receive grades on a statewide pencil-and-paper test.
To dispel skepticism over computer scoring, student essays were simultaneously graded by a computer and trained readers during a two-year pilot program. Using artificial intelligence to mimic the grading process of human readers, the computer's automated scoring engine, known as e-rater, generated grades on a six-point scale that were virtually identical to those of the readers.
Still, skepticism abounds. Although English teachers at Warren Central applaud the computer's ability to evaluate spelling, punctuation, grammar and organization, Richard C. Reed, the department chairman, made it clear that "we are not 100 percent sold on the computer's ability to grade content."
Kathryn L. Allison, the English department chairwoman at North Central High School nearby, doubts that the computer can accurately assess the quality of grammatically correct and well-structured student essays that lack substance or are wrong on the facts. "Are kids going to be rewarded for having pedestrian-type answers?" she asked.
Students, too, worry about the computer's accuracy.
"I always wonder if, like, the computer is going to grade everything right," said Jared Rampersaud, a senior at North Central, who took the test during the 2003 trial run, adding that "the teacher knows me and the computer doesn't." Jared's classmate Mollie Mott agreed. "We're always told that even the computer makes mistakes," she said. "I just think it helps if a person can actually look at" the essay.
How soon other states will emulate Indiana will depend, in part, on how well the machine's performance compares to that of human graders. So far, pilot tests in Oregon, Pennsylvania, Massachusetts and South Dakota have failed to persuade those states to abandon human grading.
Access to computers and the Internet is, of course, critical. Indiana ranks among the top 10 states in student access to computer technology, with one instructional computer for every three students, according to a report published in May by Education Week magazine, and all of Indiana's public high schools are wired for high-speed Internet.
Even so, scheduling is tricky. "Let's say you've got a computer lab that's set up for all English classes," said Wesley D. Bruce, Indiana's director of school assessment. "You need a two-week window to test every 11th grader. So you're throwing 9th, 10th and 12th graders out of lab time."
Technical glitches are another hurdle. During a trial run, an Internet configuration error prevented students at one school from submitting their completed essays for grading.
While pleased to be in the vanguard of a technology that could transform essay grading nationwide, state education officials are mindful of the risks. "With paper and pencil we've spent decades figuring out what's going to go wrong and how to deal with it," Mr. Bruce said. "With online we just don't know where all the problems are."
"We hope we're on the leading edge and not the bleeding edge," he said warily.