Under the Hood of the IES Reading Study

I know, I know, Eduflack is like a dog with an unbelievably potent bone on this whole IES interim study on Reading First.  I can’t help it.  Maybe its because I’m a contrarian.  Maybe I hate to see folks pile on to something that deserves a good defense.  And maybe I’m just practicing insanity, believing that if I keep focusing on the benefits again and again, someone may hear it and change their thinking and their practice.

I come here today not to prosthelatize on RF.  Instead, I want to serve as a conduit for needed information.  If we’ve learned anything from the back-and-forth on the IES study, it is that there are some real questions with regard to the methodology and the project design.  Rather than just trust the salesman that the engine under the study hood is legit, I’ve brought in an expert mechanic of my own.

Today, we hear from the University of Illinois-Chicago’s Tim Shanahan.  If you’ve heard of the IES study, you know Tim.  A leader on the National Reading Panel, Dr. Shanahan has served on a number of similarly influential groups on reading instruction.  He is also the former reading czar of Chicago Public Schools and recently completed his tenure as president of the International Reading Association.

I met Tim a decade ago, when I began my service to the NRP.  Immediately, I found that he was one of those rare breeds who knew the research cold, but could explain it to anyone’s grandma so she understood it … thoroughly and completely.  Even more, he had the patience and the perseverance to teach this old dog about research methodology and scientific approaches, giving me the foundational understandings I have put to use virtually every day since. 

Put simply, there are few researchers I trust more than Dr. Tim Shanahan.  He is as straight a shooter as they come.  And for our purposes today, Tim was an advisor to the IES study, so he knows of what he speaks.  So we asked some questions, he provided far better answers.



EDUFLACK: What does the IES study really say?  How strong are the findings?

SHANAHAN: THE IMPLEMENTATION STUDIES INDICATE THAT THE DIFFERENCES BETWEEN RF AND NON-RF SCHOOLS WERE PRETTY MODEST (ABOUT 50 MINUTES OF INSTRUCTIONAL DIFFERENCE PER YEAR IN AMOUNT OF INSTRUCTION), MEANING THAT RF KIDS PROBABLY RECEIVED FEWER THAN 30 HOURS OF ADDITIONAL READING INSTRUCTION EACH YEAR DUE TO THE INTERVENTION. CLEARLY A MODEST INTERVENTION, ESPECIALLY GIVEN THE SIMILARITIES IN CURRICULUM, INSTRUCTIONAL MATERIALS, PROFESSIONAL DEVELOPMENT, AND ASSESSMENTS.

Q: How valid are the findings, knowing there may be contamination across groups (that both the RF and non-RF groups may have been doing the same things in the classroom)?

A: MOST SCHOOLS EMPLOY SOME KIND OF COMMERCIAL CORE PROGRAM. WHEN READING FIRST EMPHASIZED THE ADOPTION OF PROGRAMS WITH CERTAIN DESIGNS ALL MAJOR PUBLISHERS CHANGED THEIR DESIGNS TO MATCH THE REQUIREMENTS.

READING FIRST SCHOOLS ALL BOUGHT NEW PROGRAMS IN YEAR 1; ALMOST ALL OTHER TITLE I SCHOOLS ADOPT NEW CORE PROGRAMS EVERY FOUR OR FIVE YEARS. THAT MEANS IN YEAR 1, 100% OF THE RF SCHOOLS GOT A NEW PROGRAM, AND 25% OF THE OTHER SCHOOLS DID. IN YEAR 2, THAT NUMBER WENT TO 50%, IN YEAR THREE 75%. ALL RF SCHOOLS HIRED COACHES IN YEAR 1, SO DID MORE THAN 80% OF THE OTHER SCHOOLS. ETC.

THIS ISN’T A CASE OF SPOT CONTAMINATION, IT WAS INTENTIONAL AND PERVASIVE (IN FACT, IT WAS PART OF THE RF LAW ITSELF—20% OF THE STATE MONEY, THAT MEANS $1 BILLION TOTAL WAS DEVOTED TO GETTING NON-READING FIRST SCHOOLS TO ADOPT THESE REFORMS).


Q: Given that contamination, are there contamination rates that can be tolerated in the design?  For example, let’s say 15 percent of the RF and comparison groups received identical programs/PD.  Is this level of contamination tolerable?  What if there is a 30 percent overlap – is this level tolerable?  Are there ways to estimate the degree to which percent contamination will indicate a need to increase sample size? 

A: THE PERCENTAGES OF OVERLAP WERE 75-100% DEPENDING ON THE VARIABLE. THE ONLY ONE WHERE WE HAVE ANY KIND OF IDEA ABOUT WHAT IS TOLERABLE IS WITH TIME.

FROM PAST RESEARCH, ONE SUSPECTS THAT 100 HOURS OF ADDITIONAL INSTRUCTION WOULD HAVE A HIGH LIKELIHOOD OF GENERATING A LEARNING DIFFERENCE, A 50-60 HOUR DIFFERENCE WOULD STILL HAVE A REASONABLE CHANCE OF RESULTING IN A DIFFERENCE. AT 25-30 HOURS A SMALL DIFFERENCE IN LEARNING MIGHT BE OBTAINED, BUT IT IS MUCH LESS LIKELY (ESPECIALLY IF THE CURRICULA WERE THE SAME).



Q: Did the evaluation design include procedures/strategies to  avoid contamination between RF and the comparison group?

A: IT [THE IES STUDY] NOT ONLY DID NOT TRY TO AVOID CONTAMINATION, IT COULDN’T POSSIBLY DO IT SINCE THE SOURCES OF THE CONTAMINATION WERE SO PERVASIVE. FIRST, THE FEDERAL POLICY EXPLICITLY CALLED FOR SUCH CONTAMINATION TO BE PUSHED. SECOND, STATES AND LOCAL DISTRICTS MADE THEIR OWN CHOICES (AND THEY FELT ENTICED OR PRESSURED TO MATCH RF).

FOR EXAMPLE, SYRACUSE, NY RECEIVED READING FIRST MONEY FOR SOME SCHOOLS, BUT MANDATED THAT ALL OF ITS SCHOOLS ADOPT THE SAME POLICIES AND PROGRAMS. THERE SHOULD HAVE BEEN NO DIFFERENCES BETWEEN RF AND NON-RF SCHOOLS IN SYRACUSE, THE ONLY DIFFERENCE WOULD BE IN FUNDING STREAM—HOW THE CHANGES WERE PAID FOR, AS THE NON-RF SCHOOLS ATTENDED THE SAME MEETINGS AND TRAININGS, ADOPTED THE SAME BOOKS AND ASSESSMENTS, RECEIVED THE SAME COACHING, PUT IN PLACE THE SAME POLICIES, ETC.


Q: Did the evaluation design describe practices in the comparison groups?

A: YES, THE IMPLEMENTATION STUDIES SHOW THE SIMILARITIES IN PRACTICES AND HOW, OVER TIME, THE PRACTICES THAT WERE SIMILAR AT THE BEGINNING BECAME INCREASINGLY SIMILAR EACH YEAR. THAT WILL BE CLEARER IN THE NEXT STUDY OUT

Q: Did the evaluation design account in any way for contamination, crossover, compensatory rivalry, etc.?

A: NO. THE FEDERAL LAW CALLED FOR THE EVALUATION OF READING FIRST IN TERMS OF THE EFFECTIVENESS OF THE INSTRUCTIONAL MODEL, BUT DID NOT CALL FOR A STUDY OF THE IMPACT OF READING FIRST UPON THE ENTIRE EDUCATIONAL SYSTEM.

EVEN THOUGH I HAD PERSONALLY MADE A BIG DEAL OUT OF THE PROBLEM FROM THE VERY FIRST STUDY DESIGN MEETING, THE METHODOLOGISTS THOUGHT THEY COULD HANDLE MY PROBLEM SIMPLY BY ACCOUNTING FOR THE RF ROLLOUT EACH YEAR. THEIR ASSUMPTION WAS THAT RF WOULD IMPLEMENT SOME CHANGES IN YEAR 1, OTHERS IN YEAR 2, AND STILL OTHERS IN YEAR 3 AND THAT THIS PATTERN OF IMPLEMENTATION WOULD ALLOW THEM TO EXAMINE A CONTINUING LAG BETWEEN THE RF AND NON-RF SCHOOLS.

I DIDN’T UNDERSTAND THAT THEY WERE THINKING THAT AND THEY NEVER ASKED DIRECTLY ABOUT THAT. LAST YEAR, I FIGURED OUT WHAT THEY WERE THINKING AND I HAD TO EXPLAIN SEVERAL TIMES THAT RF PUT ALL OF ITS REFORMS IN PLACE DURING YEAR 1, WITH NOTHING NEW IN YEARS 2 AND 3, SO IT WOULD BE IMPOSSIBLE TO TEST THE EFFECTS OF DIFFERENT PARTS OF THE IMPLEMENTATION, ETC. USING THEIR APPROACH. I MIGHT HAVE BEEN ABLE TO GET THIS FIXED IF I HAD UNDERSTOOD THAT THEY WERE ASSUMING THAT KIND OF DESIGN (OR IF THEY HAD ASKED ME ABOUT THAT SPECIFICALLY).


Q: Can we assume that the RF group is just like the comparison group except for exposure to RF funding?

A: READ THE IMPLEMENTATION PART OF THE REPORT (AND THERE IS ANOTHER STUDY COMING LATER THAT WILL MAKE THIS CLEARER) AND YOU’LL SEE THE DEGREE OF SIMILARITY IN THE KEY FACTORS BETWEEN THE TWO SETS OF SCHOOLS. I RAISED THIS AS A THEORETICAL PROBLEM ORIGINALLY, BUT THE IMPLEMENTATION STUDY CLEARLY SHOWS THAT CONTAMINATION WAS A BIG PROBLEM (IT CANNOT TELL US WHETHER THE CONTAMINATION CAME FROM THE $1 BILLION FEDERAL EXPENDITURE ON THIS, BECAUSE THE STATES AND LOCAL DISTRICTS OFTEN SIMPLY ADOPTED THE SAME IDEAS.

AS ONE ILLINOIS DISTRICT TOLD ME, “IF THIS IS THE RIGHT STUFF TO DO, THEN WE ARE GOING TO DO IT WITH EVERYONE.” 


That’s a lot to chew on, but it is a worthwhile meal.  Even for the most simple-minded of laypeople (like Eduflack), it is clear that the IES study had no real control group.  We had RF schools and non-RF schools, both pools of which were doing similar things with similar materials.  How can we compare the two groups as haves and have nots when the only measure of separation is the bucket of money that was paying for the approach?

Dr. Whitehurst, I’ll yield the pulpit to you if you’d like to respond.
 

 

One thought on “Under the Hood of the IES Reading Study

  1. As Reid Lyon has pointed out on Ednews.com, there was sufficient funding for IES to conduct and experimental design rather than a regression discontinuity design, but IES elected not to run a true experiment. I will disagree with Tim that all RF schools implemented mandated changes in the same year. There were states and school districts within states that did not their RF applications approved in the first year of funding. In actuality states were approved for RF funding over a three year period so the discontinuity design could have worked.Similarly. there are many districts where RF schools did one thing and all other schools did something else — New York City and Boston schools, for instance. But the larger problem with this interpretation is that after four decades of evaluation of federal reading programs we now have another evaluation with the same results — federal model did not improve reading achievement. My reading of these data indicate that there is no hope that our federal policy makers have any clue as to how to improve reading instruction. Nonetheless, those same policy makers continue to seek out the advice of different authors of commercial reading programs as advisors and they give the same answer — new materials will solve the problem. This scheme has not worked since the original first grade studies and yet federal policy makers continue to buy this argument and enrich publishers and program authors.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s