The Neverending Quest for Good Data

Why is it so hard to find good, meaningful scientific data to prove the efficacy of an education reform?  Do we know what good data is?  Is it too expensive to capture?  Is it deemed unnecessary in the current environment?  Is it out-of-whack with the thinking of the status quoers?

EdWeek’s Kathleen Manzo has been raising some of these issues over on her blog — Curriculum Matters.  (  And no, Eduflack has no qualms whatsoever with her taking me to task on whether the proof points I use to demonstrate Reading First is working are truly scientifically based proof points.  To the contrary, I appreciate the demand to “show me” and have greatly enjoyed the offline conversations with Manzo on what research is out there and whether that research — the good, the bad, and the ugly — meets the hard standards we expect.

For the record, I am not a methodologist, a neuropsychologist, nor an academic to the nth degree.  I learned about research methodology and standards and expected outcomes from NRPers like Tim Shanahan and Sally Shaywitz and from NICHDers such as Reid Lyon and Peggy McCardle.  My knowledge was gained on the streets, so take it for what it is worth.

When NCLB and RF were passed into law, the education community took a collective gasp of concern over the new definition of education research.  The era of squishy research was over.  The time for passing action research or customer satisfaction surveys as scientific proofs of effectiveness had met its end.  Folks starting scratching their heads, wondering how they would implement (and fund) the longitudinal, double-blind, control-grouped studies defined as scientifically based education research.

The common line in 2002 and 2003 was that only two reading programs, for instance, met the research standards in SBRR.  Those two?  Direct Instruction and Success for All.  Not Open Court.  Not Reading Recovery.  Not Voyager.  Only DI and SFA.

So what has happened over the years?  In 2002, the fear was that every educational publisher would have to adopt a medical model-style research network a la NICHD.  Millions upon millions of dollars would need to be spent by the basals to prove efficacy.  It was to be a new world order in educational research.

Where are we today?  As Manzo correctly points out, five years later there is little (if any) research out there that is now really meeting the standard.  Even the large IES interim study of RF effectiveness — that $31 million study of our RF districts — fails to meet our standards for high-quality, scientific research (if you listen to the researchers who know best).  Why?  Why is it so difficult for us to gather research that is so important?

First, we have interpreted the law the way we want to interpret the law … and not the way it was written or intended.  Those being asked to implement the research models simply didn’t want to believe that Reid Lyon and Bob Sweet really wanted them to pursue such zealous and comprehensive research.  So it was interpreted differently.  Neither consumers (school districts, teachers, and parents) nor suppliers (basals, SES providers, etc.) saw the necessity of longitudinal, control-grouped, double-blind, peer-reviewed research.  We settled for what we could get.  We knew that documents such as the NRP report of the previous National Research Council study met the requirements.  So instead of doing our own research, in the early years of RF we simply attached the NRP study as our “research base” to demonstrate efficacy.  Forget that the ink on the instructional program wasn’t dry, it was “scientifically based.”  And there were no checks or review process to prove otherwise.

Second, we are an impatient people, particularly in the education reform community.  Take a look at the NICHD reading research network, and you’ll see it takes a minimum of five years to see meaningful, long-term impact of a particular intervention.  RF grants were first awarded in 2002, with most early funders using the money for the 2003-04 school year to start.  That means just now — for the 2008-09 school year — would we truly be able to see the impact of RF interventions.  But have we waited?  Of course not.  We declared victory (or defeat) within a year or two of funding.  If test scores didn’t increase after the first full academic year, the nattering nabobs of the status quo immediately declared RF a failure, simultaneously condemning the need for “good” research.

We need to see results.  If our second grader isn’t reading, we want her reading by third grade, tops.  We don’t have the patience or the attention span to wait five to seven years to see the true efficacy of the instruction.  We need a research model that provides short-term rewards, instead of measuring the long-term effects we need.  A shame, yes, but a reality nonetheless.

The final side to our research problem triangle is the notion of control groups.  In good science, we need control groups to properly measure the effects of intervention.  How else do we know if the intervention, and not just a change in environment or a better pool of students, should be credited or student gains?  That is one of the great problems with the IES interim study.  We are measuring the impact of RF funding, but were unable to establish control groups that did not benefit from RF materials, instruction, and PD (even if they didn’t receive any hard RF dollars).

But in our real-life classroom environment, who wants their kid to be in that control group?  We all want the best for our children; we don’t want them to get the sugar pill while all the other students are getting scientifically based reading and a real leg up on life.  How do you say to teachers — in our age of collective bargaining — that these teachers on my right will get scientifically based professional development, but these two on my left will get nothing?  How do we say these students on this side of the district will get research-based instruction and materials, but this cluster here will get instruction we know to be ineffective.  Politically, our schools and their leaders can’t let real scientifically based research happen in their schools.  Too much grief.  Too many problems.  Too little perceived impact.

So where does this all leave us?  At the end of the day, we all seem to be making do with the research we can get, hoping it can be held to some standard when it comes to both methodology and outcomes.  We expect it to have enough students in the study so we can disaggregate the data and make some assumptions.  We expect to do the best we can with the info we can get.

Today, we see that most “scientifically based” research is cut from the same cloth.  No, we aren’t following the medical model established by NICHD’s reading network, nor are we following the letter of the law as called for under NCLB and RF.  Some come close, and I would again refer folks to the recent RF impact studies conducted in states such as Idaho and Ohio.  The methodology is strong, the data is meaningful.  And it shows RF is working.

What we are mostly seeing, though, is outcomes-based data.  School X scores XX% on the state reading assessment last year.  This year they introduced Y intervention, and scores increased XX%.  Is it ideal?  No.  But it is a definite start.  We are a better education community when we are collecting, analyzing, understanding, and applying data.  Looking at year-on-year improvement helps us start that learning process and helps us improve our classrooms.  It isn’t the solution, but it is an important step to getting there (particularly if we are holding all schools and students to a strong, singular learning standard).

Yes, Kathleen, we do need better research.  We know what we need, we know how to get there.  But until we demonstrate a need and a sense of urgency for the type of research NCLB and IES are hoping for, we need to take the incremental steps to get us there.  Let’s leave the squishy research of days of old dead and buried.  We’ve made progress on education research over the past five years. We need to build on it, not destroy it. 


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s