Tests aren't neutral: the public should have a voice
By by Amy Stuart on Mar 13, 2003 12:00 AM
Standardized tests are designed to hold educators accountable to the public, and test scores are given more and more weight in policy decisions as demands for accountability grow. But in most cases, the public doesn't really know what the tests measure or how they measure it.
Do they assess the skills and knowledge that our children need most? Do they do so in a fair, accurate, and useful way?
How do we hold the test makers accountable?
Standardized tests are prepared, conducted, and scored in a highly secretive manner. This saves money by allowing testing companies to use the same tests year after year, but at the cost of public access to the questions and answers.
Because teachers do not find out how students do on particular questions, it is difficult to use test scores to improve instruction. A numeric score in a vague subject area like "geometry" tells nothing about a child's specific needs. Parents can learn little about how to support their children's learning at home.
One exception to this rule is Arizona, where tests are now made public after several incidences of scoring mistakes by the publisher, CTB/McGraw Hill. New York, too, has agreed to release testing booklets because of a series of scoring problems there that led to public pressure for access to the tests.
A common source of scoring errors is a practice called "score equating," in which publishers adjust students' scores according to the supposed difficulty of the test questions. For example, two students with different numbers of correct answers would receive the same score, because one version of the test was considered more difficult than the other.
This complicates the scoring process and, along with other factors, has resulted in many problems. In July 2002, hundreds of Nevada high school students were told that they had failed the math portion of the high-stakes state test, when in fact they had passed it. Test results have been thrown out or delayed because of scoring errors in North Carolina, Georgia, Texas, New York and New Mexico, among others.
But even when tests are scored correctly, there is no guarantee that they measure what we want to know about our children's learning. Most states have so many content standards at each grade level that exams can test only an arbitrary sample. Furthermore, the number of questions on any one topic is too small to gauge accurately a student's understanding of that topic.
With only one or two questions on a particular topic, the margin of error is so great that a child's score for that particular concept is virtually meaningless. A bad day, a minor distraction, or a careless mistake could lead to a false assessment of a student's understanding.
For this reason, the American Psychological Association opposes the use of high-stakes tests for students. Their professional standards state, "In educational settings, a decision or characterization that will have major impact on a student should not be made on the basis of a single test score."
Even at the school level, researchers Thomas Kane of UCLA and Douglas Staiger of Dartmouth College found that school rankings based on test scores "generally resemble a lottery." This is because factors such as high student mobility rates -- which mean constantly-changing student bodies at many schools -- make year-by-year comparisons of school scores virtually meaningless.
All of these concerns underscore the need for multiple and diverse assessments of student progress. Tests are not infallible, and even when they work properly, they are not intended to provide definitive information about individual students or even individual schools.
These issues also underscore the importance of making test questions and the test-development process public so that parents and teachers can see for themselves what the results really mean. If individual schools and students are to be punished for low test scores, it is imperative that -- at the very least -- the evidence be made public.
Before handing over to testing companies the power to tell us who is succeeding and who is failing, parents, teachers and others concerned about educational quality must ask tough questions about what these tests measure and how they measure it.
Those who would hold educators accountable for students' learning must in turn be held accountable by the public. We must ensure that what they measure is what we really want to know -- and that they measure it fairly and accurately.