Lies, damned lies, and statistics

When politicians, policymakers, and researchers are looking to make a point with regard to education reform, they almost uniformly point to statistics. I’m just as guilty of that, counseling that one needs to demonstrate measurable student improvement, while putting a real student and community face on those gains. We need to see who’s benefiting, not just that there are scientifically significant gains in scores.

This past week, we’ve witnessed two examples of the dangers of statistics as a rhetorical tool. The first comes in the NYT, in Schemo’s article that somehow claims Madison, WI students are better off by being denied research-based instruction. In it, the reporter throws around statistics claiming that student reading scores have dramatically improved after stripping the schools of feared phonics and scientifically based reading.

But a flurry of activity over the weekend indicates that the NYT’s numbers simply don’t add up. This can best be read on “D-Ed Reckoning”, where a further review of the data shows that Madison may indeed be suffering for failing to provide its students with research-proven reading instruction.

On the flip side, critics are attacking President Bush for his selective use of NAEP scores to demonstrate that Reading First is working. After years of critics calling for proof that RF works, they now say the NAEP data is attributed to other factors, because it is too soon to see the effects of RF on today’s elementary school students.

Through my informal educational research education, I know that you need to allow for up to five years to start seeing demonstrable, measurable improvement as a result of an instructional change. Improvement doesn’t happen overnight. But you can see signs of it almost instantaneously, the result of changes in behavior and changes in classroom attitude.

More importantly, though, I know that statistics are best serves as a “proof” of your message, not as the message itself. For instance: Reading First works. We know it works because we are seeing the improvement in classrooms across the nation. In schools large and small. Rich and poor. Black, Hispanic and white. And we know that a stronger adherance to research-proven methods is the only way to close the gap for those kids struggling to read at grade level. That’s the message.

How do we prove it? We have decades of scientific research to demonstrate effectiveness. We have classrooms, schools, and districts that have effectively used it. And we have NAEP and statewide data that demonstrate the improvements. Again, NAEP is a proof point, not the message itself. That’s where the President failed. By using NAEP as the message, we trigger a debate of the efficacy of data itself. Instead, he should be connecting the message with the people — the kids, parents, and teachers — he needs to carry RF forward.

As for the NYT, I’ll leave it to others to ascribe motives to their mis-reading of the Madison data. We all know data can often be shaped and pulled and prodded to meet our own political or rhetorical needs. What a shame that Madison students are being used as pawns in the reading wars. They deserve proven reading instruction. It’s a shame many of them can’t read the NYT piece about their own schools.

(Originally posted March 12, 2007)

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s