What Works for the WWC

Last week, This Week in Education revealed that Russ Whitehurst was leaving the Institute of Education Sciences.  That should come as no surprise, as Whitehurst’s congressionally appointed term expires in November 2008, and he has made clear he was not seeking reappointment.  TWIE’s announcement was followed by Fordham Flypaper’s news that Whitehurst was moving over to Brookings’ Brown Center for Education Policy, presumably to fill the very capable shoes of the departing Tom Loveless.

Interestingly, no one in DC education policy circles seems to be talking about what comes next for IES.  Who will serve as the next director?  What will the priorities be?  Heck, we don’t even know where an Obama or McCain administration stands on IES, its mission, and its programs.  One thing seems certain, though, the future of the Institute won’t be determined until well after the next president is sworn into office. 
The defining experiment of IES has been the What Works Clearinghouse.  Released six years ago with much fanfare, the WWC was intended to be a Consumer’s Reports of sorts for education practitioners.  It was to sort through all of the education research data, determine what works, and provide guidance to school district officials, building leaders, and teachers on what was most effective and how education dollars should be spent.
A noble goal, and a much needed role in today’s education universe. Unfortunately, a funny thing happened on the way to implementation.  WWC became a methodological monster.  In its zeal to distinguish itself from the past work of the U.S. Department of Education’s former Office of Education Research and Improvement, WWC laid out strict and complicated criteria for every piece of research it would examine.  The result?  Most research was kicked out during the initial stages, found to lack the methodological rigor WWC called for.  For these studies, we never got to the issue of effectiveness or impact because they lacked effective control groups or didn’t form the proper study structures, as called for by WWC and its advisors.
Over the years, we have accepted the WWC process as fact.  Some organizations have pleaded mea culpa, asking WWC for advice on what to do.  Others have carefully constructed a single study to meet WWC criteria so they could claim approval (while claiming their competitors were rejected).  And still others refused to acknowledge the value and authority of WWC, and continued to do things their way, stating that their research was true, clear, and effective.
In recent months, WWC has drawn some real attention, mostly for its negative findings on the instructional impact of reading programs Open Court and Reading Mastery. For those watching WWC all these years, we just chalked it up to WWC being WWC.  Eduflack was of similar mind, reminding individuals that, at the end of the day, decisionmakers at the district or school level are simply not making decisions based on WWC findings.  While it may have been the intention, WWC is now merely a forum for researchers to try and outresearch each other, for methodological masters to do their jousting and determine who the true jedi was.  Despite the best of intentions and the most noblest of needs, WWC had become irrelevant.
Then an interesting document crossed my desk.  Under the headline “Machinations of the What Works Clearinghouse,” Zig Englemann provides a terrific analysis on the WWC’s decisions regarding Reading Mastery.  The headline?  “What Works Clearinghouse is so irreparably biased that it would have to be thoroughly reoriented and reorganized under different management rules to perform the function of providing reliable, accurate information about what works.”
With regard to Reading Mastery, Englemann points to the fact that there are more than 90 research studies on the program (and its predecessor DISTAR Reading), with most of these studies appearing in “refereed journals.”  Yet WWC found that “no studies of Reading Mastery that fell within the scope of the Beginning Reading review meet WWC evidence standards.”
Honestly, Eduflack is just plain tired of hearing that no studies fell within the scope.  For years, I defended WWC and IES, seeing both as necessary components to strengthening the research base of the education field.  When Mathematica announced that it was reopening and re-examining the Beginning Reading field — a prior review that had caused much heartburn across the reading community — I was heartened that change may be in the works.  But my atypical optimism has quickly been replaced by my real and necessary skepticism.  Englemann is right, we need to build a new, better, and more effective IES before it dies from within.
Englemann does a terrific job in looking at the WWC methodology, identifying the problems in techniques, approaches, and analyses undertaken by WWC (who knew that research conducted prior to 1985 didn’t count, even if it met every methodological standard laid out by WWC and its methodological advisors).  I would highly recommend the study for all those seeking to better understand how good research has been deemed outside the federal scope these past few years.
But yelling into the wind only gets us so far.  For the record, I still believe in the mission and goals of WWC, and I particularly believe in the mission, goals, and objectives of IES.  We need a strong, independent voice in the federal government that can tell us what works and can point us to valuable research on improving practice and instruction.  We need a voice that ensures what is proven effective is what is being practiced.  We need the evidence, plain and simple.
Now is a unique time for IES and the Board that oversees it.  Currently, a number of board member nominations are pending before Congress.  Clearly, a search for a new IES director must be underway.  So what’s the message IES and its Board should be spending in the near future?
* The voice of the practitioner is just as important as the voice of the researcher.  Methodology is important, but it is no more important that ensuring an intervention works in a school like mine, in a class like mine, with kids like mine.
* We cannot exclusively value process over results.  Again, the methodology is important, but so are outcomes.  Which is more valuable to our schools — methodologically strong programs with middling results or middling methodology with extremely strong results?  Yes, we want strong methodology and results, but if I have to choose, I want to invest in what works and what’s working on a large scale.
* We need to broaden the audience.  IES should be playing to a wide range of stakeholders, and not merely the research community and the education publishers it happens to review at a given time.  It should be a resource for teachers, school leaders, and policymakers.  To do that, we need to simplify the message and broaden the appeal.
* IES can become a Consumer Reports for the education sector.  But to do so, it needs to change its thinking and approach.  It needs to be user friendly.  It needs to be collaborative.  And it needs to think more broadly about the impact it is having, not just on research but on education and the community at large.
* IES needs to inspire and lead.  This isn’t just about being the stick to ED’s funding carrot.  It isn’t just about good, methodologically sound research.  IES needs to invest the time and effort into educating stakeholders as to why scientifically-sound education is important.  Why do we care about methodology?  What are the benefits to scientifically based instruction?  And what can I do to get the science into my kids’ schools?  It needs to demonstrate how education sciences improves student achievement and boost student options for both school and
Recent attempts by IES and WWC to quickly turn around focused research reports are a good first step. But if the Institute is going to build its legacy, it needs to focus on building both public awareness for its mission and objectives and establishing public support and enthusiasm for achieving those goals.  It’s not a simple process, but it is doable.  And it is necessary for IES to survive and thrive.
Regardless of who is EdSec come January, IES should be a priority for either administration.  It possesses enormous potential and opportunity.  Hopefully, someone will remember that and we can build on its best qualities, learn from its worst, and do what is right by states, districts, schools, and students.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s