The Neverending Quest for Good Data

Why is it so hard to find good, meaningful scientific data to prove the efficacy of an education reform?  Do we know what good data is?  Is it too expensive to capture?  Is it deemed unnecessary in the current environment?  Is it out-of-whack with the thinking of the status quoers?

EdWeek’s Kathleen Manzo has been raising some of these issues over on her blog — Curriculum Matters.  (  And no, Eduflack has no qualms whatsoever with her taking me to task on whether the proof points I use to demonstrate Reading First is working are truly scientifically based proof points.  To the contrary, I appreciate the demand to “show me” and have greatly enjoyed the offline conversations with Manzo on what research is out there and whether that research — the good, the bad, and the ugly — meets the hard standards we expect.

For the record, I am not a methodologist, a neuropsychologist, nor an academic to the nth degree.  I learned about research methodology and standards and expected outcomes from NRPers like Tim Shanahan and Sally Shaywitz and from NICHDers such as Reid Lyon and Peggy McCardle.  My knowledge was gained on the streets, so take it for what it is worth.

When NCLB and RF were passed into law, the education community took a collective gasp of concern over the new definition of education research.  The era of squishy research was over.  The time for passing action research or customer satisfaction surveys as scientific proofs of effectiveness had met its end.  Folks starting scratching their heads, wondering how they would implement (and fund) the longitudinal, double-blind, control-grouped studies defined as scientifically based education research.

The common line in 2002 and 2003 was that only two reading programs, for instance, met the research standards in SBRR.  Those two?  Direct Instruction and Success for All.  Not Open Court.  Not Reading Recovery.  Not Voyager.  Only DI and SFA.

So what has happened over the years?  In 2002, the fear was that every educational publisher would have to adopt a medical model-style research network a la NICHD.  Millions upon millions of dollars would need to be spent by the basals to prove efficacy.  It was to be a new world order in educational research.

Where are we today?  As Manzo correctly points out, five years later there is little (if any) research out there that is now really meeting the standard.  Even the large IES interim study of RF effectiveness — that $31 million study of our RF districts — fails to meet our standards for high-quality, scientific research (if you listen to the researchers who know best).  Why?  Why is it so difficult for us to gather research that is so important?

First, we have interpreted the law the way we want to interpret the law … and not the way it was written or intended.  Those being asked to implement the research models simply didn’t want to believe that Reid Lyon and Bob Sweet really wanted them to pursue such zealous and comprehensive research.  So it was interpreted differently.  Neither consumers (school districts, teachers, and parents) nor suppliers (basals, SES providers, etc.) saw the necessity of longitudinal, control-grouped, double-blind, peer-reviewed research.  We settled for what we could get.  We knew that documents such as the NRP report of the previous National Research Council study met the requirements.  So instead of doing our own research, in the early years of RF we simply attached the NRP study as our “research base” to demonstrate efficacy.  Forget that the ink on the instructional program wasn’t dry, it was “scientifically based.”  And there were no checks or review process to prove otherwise.

Second, we are an impatient people, particularly in the education reform community.  Take a look at the NICHD reading research network, and you’ll see it takes a minimum of five years to see meaningful, long-term impact of a particular intervention.  RF grants were first awarded in 2002, with most early funders using the money for the 2003-04 school year to start.  That means just now — for the 2008-09 school year — would we truly be able to see the impact of RF interventions.  But have we waited?  Of course not.  We declared victory (or defeat) within a year or two of funding.  If test scores didn’t increase after the first full academic year, the nattering nabobs of the status quo immediately declared RF a failure, simultaneously condemning the need for “good” research.

We need to see results.  If our second grader isn’t reading, we want her reading by third grade, tops.  We don’t have the patience or the attention span to wait five to seven years to see the true efficacy of the instruction.  We need a research model that provides short-term rewards, instead of measuring the long-term effects we need.  A shame, yes, but a reality nonetheless.

The final side to our research problem triangle is the notion of control groups.  In good science, we need control groups to properly measure the effects of intervention.  How else do we know if the intervention, and not just a change in environment or a better pool of students, should be credited or student gains?  That is one of the great problems with the IES interim study.  We are measuring the impact of RF funding, but were unable to establish control groups that did not benefit from RF materials, instruction, and PD (even if they didn’t receive any hard RF dollars).

But in our real-life classroom environment, who wants their kid to be in that control group?  We all want the best for our children; we don’t want them to get the sugar pill while all the other students are getting scientifically based reading and a real leg up on life.  How do you say to teachers — in our age of collective bargaining — that these teachers on my right will get scientifically based professional development, but these two on my left will get nothing?  How do we say these students on this side of the district will get research-based instruction and materials, but this cluster here will get instruction we know to be ineffective.  Politically, our schools and their leaders can’t let real scientifically based research happen in their schools.  Too much grief.  Too many problems.  Too little perceived impact.

So where does this all leave us?  At the end of the day, we all seem to be making do with the research we can get, hoping it can be held to some standard when it comes to both methodology and outcomes.  We expect it to have enough students in the study so we can disaggregate the data and make some assumptions.  We expect to do the best we can with the info we can get.

Today, we see that most “scientifically based” research is cut from the same cloth.  No, we aren’t following the medical model established by NICHD’s reading network, nor are we following the letter of the law as called for under NCLB and RF.  Some come close, and I would again refer folks to the recent RF impact studies conducted in states such as Idaho and Ohio.  The methodology is strong, the data is meaningful.  And it shows RF is working.

What we are mostly seeing, though, is outcomes-based data.  School X scores XX% on the state reading assessment last year.  This year they introduced Y intervention, and scores increased XX%.  Is it ideal?  No.  But it is a definite start.  We are a better education community when we are collecting, analyzing, understanding, and applying data.  Looking at year-on-year improvement helps us start that learning process and helps us improve our classrooms.  It isn’t the solution, but it is an important step to getting there (particularly if we are holding all schools and students to a strong, singular learning standard).

Yes, Kathleen, we do need better research.  We know what we need, we know how to get there.  But until we demonstrate a need and a sense of urgency for the type of research NCLB and IES are hoping for, we need to take the incremental steps to get us there.  Let’s leave the squishy research of days of old dead and buried.  We’ve made progress on education research over the past five years. We need to build on it, not destroy it. 


“Good For You!”

Why is it so hard for some people to see the benefit of national education standards?  It seems like a such a common sense issue — identifying a common standard for all U.S. students and then doing what it takes to get all students to meet it.  Maybe it is just too simple a concept, particularly since it seems to face such disdain or disenchantment from a significant number of people who should know better.

A perfect example of this was on display this past weekend on NBC’s Meet the Press.  For those who missed it, Obama surrogate U.S. Sen. Claire McCaskill was asked a throw-away education question following 20 minutes of questions on Iraq, gas, and the economy.  Host Tom Brokaw asked Missouri’s junior senator about No Child Left Behind and the seemingly bipartisan opposition to the law.

McCaskill set her sights on the issue of national standards.  NCLB is too inflexible, expecting too much from too many.  She believes — as many Democrats do — that having one bar for all students to clear is somehow unfair.  We should recognize all schools who demonstrate improvement.  When they improve, she said, “good for you.”  If they don’t improve, then we have a problem.

Where do we start with all of this?  Should we really applaud those schools that increase from 10% to 14% proficiency in 4th grade reading or math?  Is that really worth celebrating?  All that does is tee up the states to reduce their individual “standards” each year to allow for a couple of percentage points increase year on year.  Then all schools can have that “good for you.”  Don’t believe me, take a look at how some states have “readjusted” their standards in recent years.

At the end of the day, standards should be inflexible.  Standards are rigid.  Just today, I was talking with a former urban educator who, without doubt, falls into the “liberal” side of the puzzle.  She was saying that the best thing about NCLB was it held all students to one expectation.  Didn’t matter if they came from a low-income family.  Didn’t matter if their parents were educated.  Didn’t matter if they were an only child or the youngest of eight.  Didn’t matter the language spoken at home or in the neighborhood.  We should expect all students to be proficient.  And we shouldn’t be celebrating anything until they reach that level.

That is the ultimate root of the concept of abolishing the soft bigotry of low expectations that governed the establishment of NCLB.  That bigotry consists of the excuses we make for failure.  The socioeconomic reasons.  The family structure.  The neighborhood strife.  The lack of resources.  We’ve replaced the Horatio Alger story with a “But, if” story.  We celebrate process, without worrying about the end results.  And that’s just a cryin’ shame.

We’d all like to believe that all of the schools in McCaskill’s Show Me State are reading and computing at proficient levels, and all can get that pat on the back and the “good for you” that she wants to hand out along with increased federal appropriation.  But we all know that’s not the case.  We need to set a national standard because we need something for the nation to aspire to.  We need a bar that means something, a common bar that every single student in the United States must clear to demonstrate effective learning.  Is that really asking for too much?


No Literacy Crisis?!?

It now seems we are in the full funeral procession for Reading First.  In today’s USA Today’s dueling editorials, the nation’s newspaper of record calls for Reading First to be brought back from the dead.  USA Today notes ( that the program works, evidenced by the growing number of statements from educators and from recent studies — such as those released last month by CEP — that demonstrate improvement.  And RF offices in states from Idaho to Ohio to Alabama have added their voice to save the necessary program. 

The latest defense of the program can be found in yesterday’s Boston Herald (, in a passionate piece from the head of the Bay State Reading Institute.

Eduflack has said it once, and I’ll say it again.  RF works.  It has boosted student reading achievement, as demonstrated by student assessments in states across the nation.  It has improved reading teacher development, empowering teachers to use data to target specific interventions at the students who need them.  And it has focused SEA and LEA spending on programs that work, demanding solid, research-based proof.  After flatlining for decades, achievement is on the rise, and improvements are due, in large part, to the direct and indirect impact of Reading First and the embrace of scientifically based reading.

Unfortunately, such results don’t seem to be enough for some people.  Case in point — Stephen Krashen’s opposition-editorial ( to USA Today’s cogent stance.  Krashen reheats many of the same misguided stances of years past.  He embraces the urban legends and conspiracy theories of NRP attacker Elaine Garan.  And he reiterates the fallacy that RF is all about phonics, and nothing about reading comprehension (an untruth we all should know to be false).

Krashen raises two points, though, that merit continued discussion.  The first is the notion that 99 percent of Americans (adults and kids) can read and write at basic levels.  He uses this to say there is no literacy crisis, and thus no need to change the way we teach or increase our worry about student reading achievement.  99 percent?  Really? 

First, we must look at his term “basic.”  In our 50 states, we define reading ability by students who are proficient or better.  Basic does not mean proficient.  It means a fourth grader reading at a first grade level, or an adult with third grade literacy skills.  In today’s society, basic doesn’t cut it.  We need proficient readers.  And like it or not, 40 percent of today’s fourth graders are not proficient readers.  That should signal crisis to every teacher, parent, business leader, or elected official in this country. 

More disturbing, though, are Krashen’s closing comments.  Of course he applauds the death blow to RF.  In doing so, he advocates that the money should be spent on libraries in urban areas. A noble goal, yes, but it demonstrates a complete naiveté when it comes to federal appropriations.  The death of RF means the elimination of federal reading funding.  It does not mean we get to propose new programs to fund.  Nor does not mean we now have a bucket of $300 million or $1 billion of education funding that can now be spent on other programs.  The checkbook is closed.  The well is dry.  This is a bad thing.

For more than a decade now, reading instruction has received hundreds of millions of dollars from the federal government.  Funding started under the Clinton Administration and America Reads in 1997.  It increased in 2002 with the passage of RF.  Since then, billions of dollars has been directed to the LEAs to improve reading instruction in the communities that need it most.

Over the past two years, that money has been cut.  This year, Congress is zeroing out the funds.  For the first time in more than a decade, the federal government will not be supporting reading instruction in this country.  And that is a cryin’ shame.  For folks like Dr. Krashen — who have dedicated their professional lives to literacy — they should be embarrassed that RF has been defunded.  They should be offended that reading instruction is no longer a federal priority.  Instead of celebrating their defeat of RF, they should be outraged that a program supported by everyone from the IRA to the U.S. Department of Education has prematurely come to end.  They should be fighting to save reading funding for teachers and schools, not throwing parties because a program supported by their opponents in the faculty senate has been dealt a defeat.

But why should we expect that?  That would put kids and teachers above research dollars, professional reputations, and “ideological” camps. 

Is Opinion Research?

For nearly a decade now, “research” has been the buzz word in education reform.  It comes in many flavors, and it usually comes with a number of adjectives — scientifically based, high quality, effective, squishy, and such.  And by now we all know that “scientifically based research” is in the NCLB law more than 100 times.

With all of the talk about research, we know there is good research and there is not so good research.  We have action research passed off as longitudinal.  We have customer satisfaction studies passed off as randomized trials. We have people mis-using, mis-appropriating, and downright abusing the word “research.”

Through it all (at least for the past seven years or so), the U.S. Department of Education was supposed to be the arbiter between good and bad research.  IES was founded to serve as the final, most official word on what constitutes good education research.  Dollars have been realigned.  Programs have been thoroughly examined.  Priorities have been shaken up.

So where does it all leave us?  In this morning’s Washington Post, EdSec Margaret Spellings launches a passionate defense of the DC voucher program.  (Personally, I’m still waiting for such a defense of Reading First, a program helping millions upon millions of more students in schools beyond our nation’s capital, but what can you do?)

It should come as no surprise that Spellings sought to use research to demonstrate the effectiveness and the need for the DC voucher program.  Without doubt, vouchers have had a real impact on the District of Columbia.  It has reinforced the importance of education with many families.  It has opened doors of schools previously closed off to DC residents.  It has forced DC public schools and charters to do a better job, as they seek to keep DC students (and the dollars associated with their enrollment) in the DCPS coffers.  And, of course, we are starting to see the impact vouchers are having on student achievement among students who previously attended the most struggling of struggling schools.

Spellings points out all of this in her detailing of the research validating the voucher program.  But there is one “research” point Spellings uses that just has Eduflack scratching his head.  From the EdSec’s piece — “The Institute of Education Sciences (IES) found that parents of scholarship children express confidence that they will be better educated and even safer in their new schools.”
Such a statement is downright funny, and quite a bit concerning.  In all of the discussions about scientifically based research, high-quality research, the medical model, double-blind studies, control groups, and the like, I don’t remember public opinion surveys meeting the IES standard for high-quality research.  Parents feel better about their children because of vouchers?  That’s a reason to direct millions in federal funding to the program? 

Don’t get me wrong.  I’m all for public opinion polling and the value of such surveys (along with the focus groups and other qualitative research that helps educate them).  But it is one of the last things that should be used to validate a program or drive government spending on educational priorities.

If DC is to keep vouchers, it should keep them because it is driving improvement in student performance and is giving a real chance to kids previously in hopeless situations.  It should be saved with real data that bears a resemblance to the scientifically based research we demand of the our programs and that we expect our SEAs and LEAs to use in decisionmaking.  It should be actionable research, with a clear methodology that can be replicated.
Otherwise, we’re just wrapping up opinion in a research wrapper.  That may be good enough for some for-profit education companies and others trying to turn a quick buck on available federal resources, but it shouldn’t make the cut for the government — particularly the branch of ED that is in charge of high-quality research.  Ed reform should be more than a finger-in-the-wind experiment.  And Spellings and IES should know that by now.