For more than a week now, Eduflack has been a bit of a one-trick pony. Through the ole reform goggles, I’ve been unable to turn away from the issue of Reading First and IES’ interim study of this important law. It may have been a bit much for some, but it was something that just had to be done. Today, nearly 40 percent of fourth graders are still unable to read at grade level or better. We spend billions each year on textbooks and classroom libraries and SES programs. We are expecting nearly half of today’s teachers to retire in the next decade. So if not now, when?
With all of these factors, it only stands to reason that we should do anything and everything we can to ensure our schools — and our kids — are getting reading instruction that works. They need effective learning. How can anyone say that a student with no or poor reading skills has a real chance to succeed in society? They can’t. Reading is the building block for success in K-12, higher ed, career, and life.
For that reason, Eduflack has put the IES interim study under the microscope. We’ve heard from experts such as Reid Lyon, Tim Shanahan, and Richard Allington. We’ve scrutinized the methodology. We’ve pined for what could have been. Now we eagerly await for the next study that Dr. Shanahan has promised is on the way.
We close this chapter of the debate with questions, not with statements of fact. If the last week has taught me anything, it is that we know far less than we should. If these questions are keeping me up at night, they must be keeping others up as well. So I offer these so that the media, policymakers, educators, and influencers can ask them as well, knowing that together we may get some real answers.
* The Reading First law set aside $150 million for research and assessment over the last six years. By most reports, IES spent approximately $30 million. Where is the remainder of this money? What is it going toward? Are we measuring the effectiveness of this reallocation?
* What is the real intent of the IES study? Personally, I think we should be studying ROI for Reading First spending. Six years and billions of dollars later, where is student reading achievement? This study seems to be more process over outcomes.
* How can we measure RF versus non-RF schools or classrooms? Are we suggesting that non-RF schools are not using scientifically based reading in their classes? Of course not. Both buckets are using the same textbooks and have access to the same professional development and the same supplemental materials. The only real difference between RF and non-RF is from whose account the check is being cut.
* Forget how IES has interpreted it, what does the federal law say should be part of this assessment? RF has gotten into some trouble when it comes to the law’s intent (and letter) and its implementation. The law seems pretty clear and comprehensive to me. (Just check out section 1205)
* Why has IES taken a different path? And is there time to get us on the right path?
We need to follow the money here. Had IES spent the full $150 million and gotten a study like this back, advocates and nay-sayers would be screaming from the mountaintops about mismanagement and poor decisions. Yes, we have a bad study. But the nation was given the money to do a great study. Some would even say a $150 million national assessment study would be a researcher’s dream. So why wasn’t that dream fulfilled, particularly after Congress wrote the check to make it a reality? We’ve created a problem that never should have arisen.
A big check. Clear congressional intent. Opportunity to make a lasting, meaningful impact on both education and education research. It all was there. Today, we’re left holding a flawed study, and we still have no clear idea that RF — or more importantly, SBRR — works.
Yes, there is a value to doing an impact study like IES’. Such studies are valuable for the internal agency and for the structure of its future funding opportunities. But we also have a clear need for a study that tells us whether the program is working or not.
We need to get our kids reading. We need them reading at grade level. And we need to identify what works and get it in every classroom across the nation. Whatever it takes. Until we have answers to these questions, though, we may never have a national study that gives us the data — and the guidance — we need to make every child a reader.
I yield the floor and will hold my tongue until more data (and opinion, of course) is presented.
IES
Under the Hood of the IES Reading Study
I know, I know, Eduflack is like a dog with an unbelievably potent bone on this whole IES interim study on Reading First. I can’t help it. Maybe its because I’m a contrarian. Maybe I hate to see folks pile on to something that deserves a good defense. And maybe I’m just practicing insanity, believing that if I keep focusing on the benefits again and again, someone may hear it and change their thinking and their practice.
I come here today not to prosthelatize on RF. Instead, I want to serve as a conduit for needed information. If we’ve learned anything from the back-and-forth on the IES study, it is that there are some real questions with regard to the methodology and the project design. Rather than just trust the salesman that the engine under the study hood is legit, I’ve brought in an expert mechanic of my own.
Today, we hear from the University of Illinois-Chicago’s Tim Shanahan. If you’ve heard of the IES study, you know Tim. A leader on the National Reading Panel, Dr. Shanahan has served on a number of similarly influential groups on reading instruction. He is also the former reading czar of Chicago Public Schools and recently completed his tenure as president of the International Reading Association.
I met Tim a decade ago, when I began my service to the NRP. Immediately, I found that he was one of those rare breeds who knew the research cold, but could explain it to anyone’s grandma so she understood it … thoroughly and completely. Even more, he had the patience and the perseverance to teach this old dog about research methodology and scientific approaches, giving me the foundational understandings I have put to use virtually every day since.
Put simply, there are few researchers I trust more than Dr. Tim Shanahan. He is as straight a shooter as they come. And for our purposes today, Tim was an advisor to the IES study, so he knows of what he speaks. So we asked some questions, he provided far better answers.
EDUFLACK: What does the IES study really say? How strong are the findings?
SHANAHAN: THE IMPLEMENTATION STUDIES INDICATE THAT THE DIFFERENCES BETWEEN RF AND NON-RF SCHOOLS WERE PRETTY MODEST (ABOUT 50 MINUTES OF INSTRUCTIONAL DIFFERENCE PER YEAR IN AMOUNT OF INSTRUCTION), MEANING THAT RF KIDS PROBABLY RECEIVED FEWER THAN 30 HOURS OF ADDITIONAL READING INSTRUCTION EACH YEAR DUE TO THE INTERVENTION. CLEARLY A MODEST INTERVENTION, ESPECIALLY GIVEN THE SIMILARITIES IN CURRICULUM, INSTRUCTIONAL MATERIALS, PROFESSIONAL DEVELOPMENT, AND ASSESSMENTS.Q: How valid are the findings, knowing there may be contamination across groups (that both the RF and non-RF groups may have been doing the same things in the classroom)?
A: MOST SCHOOLS EMPLOY SOME KIND OF COMMERCIAL CORE PROGRAM. WHEN READING FIRST EMPHASIZED THE ADOPTION OF PROGRAMS WITH CERTAIN DESIGNS ALL MAJOR PUBLISHERS CHANGED THEIR DESIGNS TO MATCH THE REQUIREMENTS.
READING FIRST SCHOOLS ALL BOUGHT NEW PROGRAMS IN YEAR 1; ALMOST ALL OTHER TITLE I SCHOOLS ADOPT NEW CORE PROGRAMS EVERY FOUR OR FIVE YEARS. THAT MEANS IN YEAR 1, 100% OF THE RF SCHOOLS GOT A NEW PROGRAM, AND 25% OF THE OTHER SCHOOLS DID. IN YEAR 2, THAT NUMBER WENT TO 50%, IN YEAR THREE 75%. ALL RF SCHOOLS HIRED COACHES IN YEAR 1, SO DID MORE THAN 80% OF THE OTHER SCHOOLS. ETC.
THIS ISN’T A CASE OF SPOT CONTAMINATION, IT WAS INTENTIONAL AND PERVASIVE (IN FACT, IT WAS PART OF THE RF LAW ITSELF—20% OF THE STATE MONEY, THAT MEANS $1 BILLION TOTAL WAS DEVOTED TO GETTING NON-READING FIRST SCHOOLS TO ADOPT THESE REFORMS).
Q: Given that contamination, are there contamination rates that can be tolerated in the design? For example, let’s say 15 percent of the RF and comparison groups received identical programs/PD. Is this level of contamination tolerable? What if there is a 30 percent overlap – is this level tolerable? Are there ways to estimate the degree to which percent contamination will indicate a need to increase sample size?
A: THE PERCENTAGES OF OVERLAP WERE 75-100% DEPENDING ON THE VARIABLE. THE ONLY ONE WHERE WE HAVE ANY KIND OF IDEA ABOUT WHAT IS TOLERABLE IS WITH TIME.FROM PAST RESEARCH, ONE SUSPECTS THAT 100 HOURS OF ADDITIONAL INSTRUCTION WOULD HAVE A HIGH LIKELIHOOD OF GENERATING A LEARNING DIFFERENCE, A 50-60 HOUR DIFFERENCE WOULD STILL HAVE A REASONABLE CHANCE OF RESULTING IN A DIFFERENCE. AT 25-30 HOURS A SMALL DIFFERENCE IN LEARNING MIGHT BE OBTAINED, BUT IT IS MUCH LESS LIKELY (ESPECIALLY IF THE CURRICULA WERE THE SAME).
A: IT [THE IES STUDY] NOT ONLY DID NOT TRY TO AVOID CONTAMINATION, IT COULDN’T POSSIBLY DO IT SINCE THE SOURCES OF THE CONTAMINATION WERE SO PERVASIVE. FIRST, THE FEDERAL POLICY EXPLICITLY CALLED FOR SUCH CONTAMINATION TO BE PUSHED. SECOND, STATES AND LOCAL DISTRICTS MADE THEIR OWN CHOICES (AND THEY FELT ENTICED OR PRESSURED TO MATCH RF).
FOR EXAMPLE, SYRACUSE, NY RECEIVED READING FIRST MONEY FOR SOME SCHOOLS, BUT MANDATED THAT ALL OF ITS SCHOOLS ADOPT THE SAME POLICIES AND PROGRAMS. THERE SHOULD HAVE BEEN NO DIFFERENCES BETWEEN RF AND NON-RF SCHOOLS IN SYRACUSE, THE ONLY DIFFERENCE WOULD BE IN FUNDING STREAM—HOW THE CHANGES WERE PAID FOR, AS THE NON-RF SCHOOLS ATTENDED THE SAME MEETINGS AND TRAININGS, ADOPTED THE SAME BOOKS AND ASSESSMENTS, RECEIVED THE SAME COACHING, PUT IN PLACE THE SAME POLICIES, ETC.
Q: Did the evaluation design describe practices in the comparison groups?
A: YES, THE IMPLEMENTATION STUDIES SHOW THE SIMILARITIES IN PRACTICES AND HOW, OVER TIME, THE PRACTICES THAT WERE SIMILAR AT THE BEGINNING BECAME INCREASINGLY SIMILAR EACH YEAR. THAT WILL BE CLEARER IN THE NEXT STUDY OUTQ: Did the evaluation design account in any way for contamination, crossover, compensatory rivalry, etc.?
A: NO. THE FEDERAL LAW CALLED FOR THE EVALUATION OF READING FIRST IN TERMS OF THE EFFECTIVENESS OF THE INSTRUCTIONAL MODEL, BUT DID NOT CALL FOR A STUDY OF THE IMPACT OF READING FIRST UPON THE ENTIRE EDUCATIONAL SYSTEM.EVEN THOUGH I HAD PERSONALLY MADE A BIG DEAL OUT OF THE PROBLEM FROM THE VERY FIRST STUDY DESIGN MEETING, THE METHODOLOGISTS THOUGHT THEY COULD HANDLE MY PROBLEM SIMPLY BY ACCOUNTING FOR THE RF ROLLOUT EACH YEAR. THEIR ASSUMPTION WAS THAT RF WOULD IMPLEMENT SOME CHANGES IN YEAR 1, OTHERS IN YEAR 2, AND STILL OTHERS IN YEAR 3 AND THAT THIS PATTERN OF IMPLEMENTATION WOULD ALLOW THEM TO EXAMINE A CONTINUING LAG BETWEEN THE RF AND NON-RF SCHOOLS.
I DIDN’T UNDERSTAND THAT THEY WERE THINKING THAT AND THEY NEVER ASKED DIRECTLY ABOUT THAT. LAST YEAR, I FIGURED OUT WHAT THEY WERE THINKING AND I HAD TO EXPLAIN SEVERAL TIMES THAT RF PUT ALL OF ITS REFORMS IN PLACE DURING YEAR 1, WITH NOTHING NEW IN YEARS 2 AND 3, SO IT WOULD BE IMPOSSIBLE TO TEST THE EFFECTS OF DIFFERENT PARTS OF THE IMPLEMENTATION, ETC. USING THEIR APPROACH. I MIGHT HAVE BEEN ABLE TO GET THIS FIXED IF I HAD UNDERSTOOD THAT THEY WERE ASSUMING THAT KIND OF DESIGN (OR IF THEY HAD ASKED ME ABOUT THAT SPECIFICALLY).
Q: Can we assume that the RF group is just like the comparison group except for exposure to RF funding?
A: READ THE IMPLEMENTATION PART OF THE REPORT (AND THERE IS ANOTHER STUDY COMING LATER THAT WILL MAKE THIS CLEARER) AND YOU’LL SEE THE DEGREE OF SIMILARITY IN THE KEY FACTORS BETWEEN THE TWO SETS OF SCHOOLS. I RAISED THIS AS A THEORETICAL PROBLEM ORIGINALLY, BUT THE IMPLEMENTATION STUDY CLEARLY SHOWS THAT CONTAMINATION WAS A BIG PROBLEM (IT CANNOT TELL US WHETHER THE CONTAMINATION CAME FROM THE $1 BILLION FEDERAL EXPENDITURE ON THIS, BECAUSE THE STATES AND LOCAL DISTRICTS OFTEN SIMPLY ADOPTED THE SAME IDEAS.AS ONE ILLINOIS DISTRICT TOLD ME, “IF THIS IS THE RIGHT STUFF TO DO, THEN WE ARE GOING TO DO IT WITH EVERYONE.”
That’s a lot to chew on, but it is a worthwhile meal. Even for the most simple-minded of laypeople (like Eduflack), it is clear that the IES study had no real control group. We had RF schools and non-RF schools, both pools of which were doing similar things with similar materials. How can we compare the two groups as haves and have nots when the only measure of separation is the bucket of money that was paying for the approach?
Dr. Whitehurst, I’ll yield the pulpit to you if you’d like to respond.
RF Works, Just Ask Idaho
If we believe the initial buzz from this month (along with the interim study from IES), the Reading First program just doesn’t seem to do the job it was intended to take on. By now, those who care have heard all about the IES study, as well as the growing criticism about its shortcomings, most notably its methodology.
Throughout this debate, we’ve heard little from the practitioners who have put RF to work in their states or communities. From those who have seen the positive effects of scientifically based reading research. From those who have determined what works for their schools and their kids. Until now.
Over at www.ednews.org, we’re seeing continued comment on this RF debate. Of particular note is a comment recently posted by Steven Underwood, the Reading First School Improvement Coordinator for Boise State University’s Center for School Improvement & Policy Studies. The headline — Reading First is working in Idaho. Not just working, but really working. Almost as if RF was designed to help struggling schools boost student reading proficiency.
Rather than summarize Underwood’s contribution to the debate, let’s here directly from the horse’s mouth, with a thanks to Underwood for letting Eduflack use the words originally posted at www.ednews.org.
“I applaud the efforts to help the nation’s most at-risk children by consulting a large body of research and theory, sifting out opinion from facts, and making policies and practices that benefit children. It is unfortunate, but many of the critics of Reading First both here and elsewhere seem to speak foremost of theory and secondarily of students. I am saddened by the number of critics who neither have worked in Reading First schools nor fully understand their practices. To continue the analogy of the car from previous posts, many critics, who undoubtedly mean well in their criticisms, seem to misunderstand the repair work that is being done and seem to be completely unaware of the data that demonstrate that Reading First is having a positive impact on student outcomes. In the criticisms, it seems like people are criticizing the mechanic who is working on the complex engine (of literacy among disadvantaged students) without themselves having ever been truly successful at fixing engines which demonstrate the same types of problems. Literacy among our nation’s needy children has been a nationwide concern for years, and Reading First is the first systemic approach to find success in addressing that concern. Had the [IES] study been conducted more in line with the mandate given to IES, we would be able to better understand the impact of Reading First at the national level. However, since the study was not well designed and did not meet its mandate, being people of reason, we are obliged to evaluate all of the other data that has been provided through systems such as the annual performance reports over the course of the years. As one studies these data, Reading First is arguably the most powerful federal education program to date. As part of No Child Left Behind, Reading First has demonstrated powerful results among those children in our nation who have traditionally been “left behind” in literacy skills.
In support of this, allow me to briefly summarize results from the state of Idaho. To qualify to become a Reading First school in Idaho, a district has to have the highest level of needs (e.g. the largest percentages of free and reduced lunch in the state) and the lowest available financial resources to meet those needs. The reason for this qualification is that student performance has so often been correlated with socio-economic status. Even though Idaho Reading First schools have such high needs, they have not only grown in their data more quickly on state reading measures, but have closed or nearly closed the gap in all grade levels. Idaho has a universal K-3 reading screener, the IRI, which measures fluency and basic comprehension. From 2003 to 2007, Reading First schools in Idaho improved on this measure at a rate that exceeded the state’s growth during the same timeframe and currently have an overall average that is within 4 percentage points of the state average.
More importantly, Idaho’s economically disadvantaged students grew at a rate in Reading First schools that far surpassed their economically disadvantaged peers in state averages. Among this subpopulation, which is a focus in the NCLB legislation, Reading First schools performed at a rate of improvement between 2003 and 2007 that was 12% better than the state average in Grade 1, 10% better in Grade 2, and 7% better in Grade 3. These results are also mirrored in the comprehensive outcome measure for Idaho Reading First schools. Idaho Reading First schools have consistently performed more than 10 percentile points above the national cut-score on the Normal Curve Equivalence for ITBS Reading Comprehension. This average far surpasses the last year in Idaho in which the ITBS was given to all students (2001), which again demonstrates that Reading First is closing the gap among the neediest children in our state. Furthermore, among economically disadvantaged students, Reading First schools have improved ITBS scores at rates between 20% and 24% in Grades 1-3 from 2004 to 2007, which again demonstrates alignment of reading comprehension results with one of the primary missions of Reading First. Lastly, and very importantly, Idaho Reading First schools are demonstrating greater overall gains and closing the achievement gap on the Grade 3 AYP measure for reading, the ISAT.
Whereas in 2003, the participating schools were significantly behind the state average, Idaho Reading First Schools are now within 2 percentage points of the state average. While the IES interim report may show no statistical significance in its study sample, the reality of Reading First in Idaho shows a vastly different picture. As mentioned before, it is unfortunate that some well-meaning educators criticize Reading First based upon political preference, theory alone, opinion, or incomplete and misleading information. The interim study published by IES did not do an adequate job in meeting its mandate, nor was it representative of the nationwide set of Reading First schools, nor did it triangulate multiple sets of reading data, nor did it identify all of the pertinent variables, nor did it operate on the basis of a true pre-Reading First baseline. With these and other criticisms of the impact study in mind, I respectfully ask our critical colleagues who believe Reading First to be ineffective to review the broader set of data that exist. Reading First has set a high standard for our nation’s public elementary schools who serve its neediest children. According to multiple sets of data in multiple states, this high standard is paying off for thousands upon thousands of children.”
There you go. Reading First is working in Idaho. In a state where the motto is “Let it be perpetual,” they are making reading instruction improvements that will empower a generation of new readers. And I’m betting there are a lot more states like it that are showing similar gains and similar benefits from RF and the implementation of SBRR in the classroom. We should be out there cultivating these positive stories, spotlighting those schools, LEAs, and SEAs that are making a difference and boosting student achievement. I know that is harder than promoting our failures and explaining why AYP can never be achieved, but we can learn a lot more examining what works rather than volleying around excuses for what doesn’t.
Lookin’ for Edu-R&D Sugardaddies
For years now, we have heard IES Director Russ Whitehurst lament the dirth of funding for education research and development. Compare the U.S. Department of Education’s research budget with that of the U.S. Department of Health and Human Services, it is embarrassing (even if you do it as a percentage of the total agency budget).
The good folks over at Knowledge Alliance (formerly NEKIA) have waved a similar banner. If we expect a scientifically based educational experience, we need to invest in scientifically based research. If we are going to do what works, we need to investigate it. And if we are going to drive the squishy research from the K-12 kingdom, we need to make meaningful investments in the strong, scientific, longitudinal research we are seeking.
Yet education R&D still seems to be feeding from the scraps of practice. We have few industry leaders that are funding R&D the way we see it in the health industry. And that view becomes even more acute today, when the Howard Hughes Medical Institute announces a $600 million grant to fund the research of 56 top medical researchers. The Washington Post has the full story here — http://www.washingtonpost.com/wp-dyn/content/article/2008/05/27/AR2008052701014.html?hpid=topnews.
It has all got Eduflack thinking of the impact such an investment could have on education. Just imagine if a philanthropy offered up $200 or $100 or even $50 million to education’s top researchers to develop major findings in how to improve public education. Science and math instruction. ELL. Teacher training. Effects of technology. Charters. The list of possible topics is limitless. In reading alone, you can take a look at the list of potential research subjects offered by the National Reading Panel in 2000. Today, most of those still haven’t been pursued.
But we all recognize that such sugardaddies are few and far between in education reform. We put our money on educational practice. We fund practitioners. R&D is an add-on, often used just to test the ROI for funders, be they philanthropic or corporate.
Yes, we have significant education investment from groups like the Bill & Melinda Gates Foundation. They have made a significant contribution to funding education reforms, particularly in our urban areas. But the focus is not on R&D, it is on classroom practice. Valuable indeed, but it doesn’t mean we don’t need a similar investment on the research side. In fact, such R&D investment can ensure Gates’ money is being wisely spent.
Without question, the money available in the education industry is at levels never imagined in generations past. Somewhere among those growing pots, there must be a potential sugardaddy (or a collection of sugarbabies) who can do for education what the Hughes Institute is doing for medicine.
As we struggle with the definitions of SBRR and the findings of the WWC, just imagine the impact we can have with a nine-figure investment in education R&D, particularly if it is led through a public-private partnership.
Today, education reform is kinda like filling a lake with teaspoon. We’re adding some drops here or there, but we can’t necessarily see the impact. With stronger R&D, we have the option of at least adding water by the barrel full, if not more. And that’s the only way to raise the opportunity boats of the kids who need it most.
The Saga of RF Profiteers Continues
Last week, Eduflack opined on where all of the Reading First profiteers have gone. (http://blog.eduflack.com/2008/05/21/calling-all-rf-profiteers.aspx) As the program is under siege and the funding has dried up, those who personally profited the most are nowhere to be found. A word of thanks to the Core Knowledge blog for throwing some additional spotlight on the important issue.
Over the weekend, we received an interesting comment from Richard Allington, the former president of the International Reading Association. Sure, Allington has long been tagged as a RF opponent, but no one can question that he understands the concept of scientifically based reading research.
His posting no doubt got me thinking. But more importantly, it got Reid Lyon thinking. As a godfather of RF, Reid definitely knows what he is talking about, and the volume of his RF conversation has increased dramatically in recent weeks. And it is important that we listen.
So without further ado, Reid Lyon’s response to Allington’s thoughts on RF profiteers …
“I believe that these interchanges among individuals with different perspectives on Reading First are helpful, as improvements are impossible with productive debate. In my mind, the debates are more productive when sufficient details are presented to support a particular point of view. Riccards brings up the detail that publishers and vendors were selling to districts and schools before the Technical Assistance Centers were ever established. He is correct,. Many did not need a “list” to garner a substantial amount of reading First funding. Bob Sweet and I predicted that when the legislative language for Reading First was softened to its use of the “based on” criterion, that a feeding frenzy would ensue with everybody and their brother hawking a program based on SBRR.
Like Allington, we felt in drafting the initial language requiring program-specific language that publishers and vendors would be highly motivated to test their products. That still has not happened. I need more details on which programs were “banned.” I know that Chris Doherty was compelled by the law to not fund programs with no basis in SBRR and he followed that law. The Wright program was not funded because it was not comprehensive and did meet additional criteria in the law. The Wright program, to its credit, attended to the reviews of its product and made substantial changes so that it now meets all criteria.
Allington may be talking about Reading Recovery as a “banned” program but Reading Recovery was funded by some states using Reading First funds. The allegations made by Success for All are baseless as indicated by no findings by the OIG of that product being placed at a disadvantage in either its first major auditing report or its audit of New York State. There has been absolutely no evidence of any state or district being pressured by the Reading First office to either drop SFA or not implement SFA. In fact, emails between different state’s Reading First officials, SFA, and a Technical Assistance Center reveal substantial positive interactions in trying to ensure that SFA could participate fully in Reading First.
In addition, a review by Elbaum et al. (2000), it was found that gains for the poorest readers were often minimal, which Elbaum et al. suggested may be related to the need for more explicit instruction in decoding. A recent meta-analysis also found that RR was effective for many grade 1 students (D’Agostino & Murphy, 2004). This study disaggregated RR outcomes by whether the outcomes involved standardized achievement tests or the Observation Survey, which parallels the RR curriculum. It also separated results for students who successfully completed RR (i.e., met program criteria and were discontinued) versus those who were unsuccessful or left the program before receiving 20 lessons (i.e., were not discontinued) and according to the methodological rigor of the studies. When the comparison group was low-achieving students, average effect sizes on standardized achievement tests for all discontinued and not discontinued students were in the small range (.32), and higher for discontinued (.48) than not discontinued (-.34) students. This finding was consistent with Elbaum, Vaughn, Hughes, and Moody (2000), who reported that RR was less effective for students with more severe reading problems. D’Agostino and Murphy (2004) found that analyses based on just the more rigorous studies included in their meta-analysis in which evaluation groups were more comparable on pretests showed smaller, but significant effect sizes on standardized measures. Disaggregation according to whether the student was discontinued or not was not possible. Effect sizes were much larger for the Observation Survey measures, but these assessments are tailored to the curriculum and also have severely skewed distributions at the beginning and end of grade 1 that suggest the Observation Survey should not be analyzed as a continuous variable in program evaluation studies (Denton, Ciancio, & Fletcher, 2006).
By assessing in greater detail the degree to which well defined groups of students respond positively to well defined interventions, we increase the likelihood that particular programs will be implemented in a more thoughtful manner rather than as a magic bullet – and this is the case for all programs.
Allington also concluded that the IES Interim Report on the Reading First Impact Study should be the final word on the effectiveness of the program. Details are critical in drawing this conclusion and they are missing in both Allington’s statement and in the media coverage on the report. Two details are noteworthy – the sample is not representative of the universe of all Reading First schools nationally, and the ability to draw meaningful conclusions about the null results is very limited due to the contamination between Reading First and Non-Reading First schools with respect to shared professional development and common instructional programs.Allington has jumped to faulty conclusions in the past before. Recently he asked the field to read two invited papers in an issue of the Elementary School Journal that he edited that ostensibly overturned the results obtained by the Phonics Subgroup of the NRP. However, a formal replication of both these two studies published in a top ranked peer reviewed archival journal (Journal of Educational Psychology) did not support the conclusions of either paper regarding the impact of systematic phonics instruction on reading outcomes. This is science at its best when replication adjudicates claims arising from publication of data particularly when the process is characterized by mature scientific dialogue.
I predict that the jury is not yet out on the effectiveness of Reading First. Who knows, if the evaluation carried out By IES actually aligned with the evaluation required in the law, more detail would have helped us interpret the results with greater confidence. But I bet that even if these flawed comparisons showed Reading First Schools to be superior to non-Reading First schools, many would have argued that Reading had not been in place long enough to make these claims.”
The saga continues. Dr. Allington, I’ll offer you a chance to respond, if you are so inclined.
SBRR Fights Back
It’s no secret that Reading First has been education’s biggest punching back these past few years. Earlier this month, IES released its interim study on the report, causing great glee with the whole languagers and the defenders of the status quo. Some used the study to write RF’s obituary. A few voices, including Eduflack, used the opportunity to highlight the flaws in the study. (http://blog.eduflack.com/2008/05/02/rf-read-all-about-it.aspx)
For years now, Eduflack has been unabashedly supportive of RF. I still believe, when all is said and done, it could have a greater POSITIVE impact on education policy than any other piece of federal legislation. For that to happen, the law needs to be properly funded AND it needs to be implemented with true and complete fidelity.
Having worked with the National Reading Panel, I am a true believer in the principles embedded in RF. We know students need a comprehensive, integrated reading instruction platform that focuses on phonics, phonemic awareness, fluency, vocabulary, and reading comprehension. We know that scientifically based reading research should rule the roost, with schools implementing only that which has been proven effective and proven to work in schools. We know that teachers must know the science. We know that students must be regularly assessed, with targeted interventions used to get all students reading at grade level.
Yet, we still debate on the value of RF and SBRR. And its been far easier to scream into the wind questioning RF. Few have been out there defending the law, calling for the need for proven research and proven instruction in our classrooms … particularly those classes who need it the most.
In RF’s darkest hours, though, we are now starting to see SBRR’s strongest proponents rising to its defense. It would have been easy to just awkwardly swallow the IES study, accept Congress’ funding slash, and forget the RF era. But we will not go quietly into the night.
When the IES study came out, the Fordham Foundation released a study — penned by Sol Stern — looking at the real failures of RF. The piece was strong, citing the operational weaknesses we’ve all heard. But it maintained that the law itself was still strong, worthy of our support, and needed by our students.
This week, the latest issue of the Manhattan Institute’s City Journal came out, and Stern was at it again. Under the headline “Reading First Still Works,” Stern presents a strong and cogent analysis of the IES study and the flaws in its methodology. We can only hope that IES will take his critique seriously, and will correct the flaws before its 2009 report is complete.
The Stern piece is well worth the read time — http://www.city-journal.org/2008/eon0520ss.html. Its helps even us amateur researchers see the difference between strong and weak methodology. More importantly, though, it reminds us that programs like RF are well worth fighting for.
Here’s hoping that Stern’s continued work can serve as a rallying cry for RF and SBRR supporters and advocates throughout the country. Teaching our children to read is of paramount importance. Using proven effective methods is the only way to go. We need to remember that. Results should trump politics, particularly on an issue like student reading achievement.
Some Ed Reccs for Senator McCain
Thanks to the Fordham Foundation’s Flypaper blog (http://www.edexcellence.net/flypaper/) we now have a good sense for the great minds advising expected Republican Presidential Nominee John McCain on education policy. As to be expected, it is an impressive bunch. Their challenge, though, will be to get education issues to stand out on the Arizona Senator’s proposed domestic policy agenda.
No, McCain is not known in DC circles as one of the Senate’s leaders on education. But that doesn’t mean he can’t rise to the occasion. The presidential bully pulpit is a strong one, and education remains a top five domestic policy issue for most. If you can’t figure out how to fix the economy in the short term, you certainly can focus on education for the long-term economic benefit.
More than a year ago, Eduflack offered a top-five list of education ideas for the Republican nominee to think about when constructing an education platform. A lot has changed since then. The latest State of the Union seemed to de-emphasize the future of vouchers. Research still isn’t sure the long-term impact of charter schools. And the expected Democratic presidential nominee has been known to talk about merit pay for teachers.
That said, let’s take a look at those March 2007 recommendations:
“1. National standards benefit the nation. Such standards don’t mean we are denying local control. They empower our local districts to remain competitive in their state, across the nation, and throughout the world. National standards, both for students and teachers, are the only way today’s students can succeed in tomorrow’s global economy.
2. Invest in education R&D. We all understand the value of investing in medical or technology R&D. Now is the time to invest in research focused on improving our schools and educational quality in our classrooms. Such investment is key to triggering true innovation at the state or national level, leading to improved economies, better jobs, and better lives.
3. Respect the practitioners. It is easy for some to say our schools have failed because our teachers have failed. If any Republican wants to engender change in our schools, they need to respect the teachers delivering the curriculum. They are on the front lines. Without their support, reform will fall flat, destined for a garbage heap of good but failed ideas.
4. Don’t fear additional spending. NCLB scared off many a Republican, particularly with increased federal education spending. The feds are still only responsible for about 8 cents of every dollar spent on public K-12 education. Additional funding is good for the system, as long as we are spending it on research-proven instruction and improvements we know will boost student achievement.
5. Focus on what works. For decades, our schools have been bombarded with the latest in snakeoils and silver bullets. Today’s educators want to see what works in schools like theirs, with kids lke theirs. NCLB is all about replicable school reforms. Now is the time to spotlight what is going right in your hometown or your home state, and use it as the model for why we need to continue federal education reforms. Many of today’s improvements are directly tied to NCLB efforts. Take credit for it.”
Interestingly, these reccs ring as true today as they did 15 months ago. But I’d offer a few caveats to Arizona’s senior senator:
* Don’t hitch your wagon to NCLB, attach yourself to the intent. It isn’t about “NCLB” the proper, it is about doing what works and funding what is proven effective. Forget the title of the law. Focus on the outcomes. The federal government has a role in public education. Claim that role, focusing on the future and expected goals.
* Don’t forget, you were a teacher too. As a leader in the Navy, you instructed and taught. You molded and trained young men. It may not have been the ABCs or the quadratic equation, but you understand the importance of good teaching. Remind us of it.
* Shine your education agenda through the filter of economic opportunity. Too often, we view education in a vacuum. We can’t afford to do that in today’s economy. Education policies should be positioned as opportunities to better prepare today’s kids for the opportunities of tomorrow. That doesn’t mean turning our K-12 schools into trade schools, but it does mean an education that is relevant to both the student and the world.
* Borrow (and steal) from the Arizona experience. As you are looking at what is relevant, take a close look at what your Governor has been doing. Her focus on innovation and STEM education shows what we need to be thinking about in education reform. Speaking from the Arizona experience, you can let the home state serve as a model for others in need. You come from a state that gets it.
Eduflack isn’t naive. I recognize that education is not going to be a primary discussion topic for you between now and November. I don’t expect it will be an issue for a keynote speech during the Minneapolis convention. But I know it is a basic bread-and-butter issue that can play well in the blue states and with independent voters.
The days when a GOP president wanted to eliminate the U.S. Department of Education are over. Now, you have the opportunity to strengthen the Department, making it more efficient and better focused on the end result. You have a team of advisors who understand data, how to use it, and the importance of measurement and assessment. Take advantage of it. Improve the system. Reject the status quo.
And please, Senator, don’t lose sight of recc #1. It may not be popular with some, but national standards are worth a good, long look. Someone, some day is going to adopt national standards. And it will result in a legacy many seek, but almost none achieve.
Just my three cents (inflation, after all). Feel free to crib from, improve, or adopt wholesale.
What Happens in 2014?
Yesterday, a who’s who of the education blob gathered to discuss the future of education research. Hosted by Education Sector, AED, AIR, and the Knowledge Alliance, folks gathered for “Towards 2014: Education Research on the Leading Edge of School Improvement?”
It was an opportunity to soak in all that Checker Finn, Russ Whitehurst, Rick Hess, Mike Smith, and the like have to say about the state of education research. The forum was a follow-up to a similar event hosted by similar organizations back in 2002, when we were all just learning to let scientifically based research roll of our tongues (and before IES was even part of our vocabulary).
For those who missed it, you can get the main thrust from Knowledge Alliance President Jim Kohlmoos’ guest blog on edbizbuzz — http://blogs.edweek.org/edweek/edbizbuzz/2008/05/friday_guest_column_is_educati.html#more.
What is particularly interesting is how little seems to have changed over the past six years. Yes, we are all now aware of what SBR is, and why it is important. But we seem to still struggle in two key areas, agreeing on what SBR is and applying it to practice.
For many, SBR is like the popular definition of pornography — we know it when we see it. Ask us to define scientifically based research (as it applies to education) and we grasp for words. Show us a recently completed research study or a journal article, and we can tell you whether it makes the cut or not. Isn’t diagnosing SBR after the fact what has gotten us in the trouble we’re in? Shouldn’t we know if a study meets the scientific standards BEFORE we have spent millions of dollars on its execution? Without a firm understanding of methodologies and research models, we risk a system where we simply slap an SBR label on the outcomes we happen to like.
We don’t seem to have this problem in medicine. We know what are scientifically based studies and what are surgically enhanced fluff. So why is education so different?
Some will use the statistic Russ Whitehurst uses — that the research portion of the U.S. Department of Education’s budget is less than one half of one percent of the total budget, where Health and Human Services is spending nearly 42 percent of its budget on research. Doesn’t that mean it is even easier for ED to ensure that its research dollars are wisely spent?
I’ll be the first to advocate for additional spending on educational research. In many urban school districts — those with schools branded as dropout factories — we are spending $10,000, $12,000, even $15,000 per student on education. As taxpayers, we have a right to know our money is well-spent. As parents, we have a right to know that our kids are getting effective instruction. As members of our social community, we have the right to know our schools work. Research is the cornerstone to all of that.
Which gets us back to the previous issue — we’re still struggling to put SBR to use in the classroom. We understand the power of the buzzword, and are quick to describe our ideas or solutions as research-based or proven-effective. But have we really studied what is happening the classroom? Are we really measuring the effectiveness of specific interventions over the long term? Are we really looking at the comprehensive research base available before deciding on a textbook or supplemental material? Are we making sure what works is what we are using?
Unfortunately, “no” seems to be used an awful lot to answer those questions. And it doesn’t have to be that way. At the forum, Checker Finn called for one-stop shops on educational research, where we all have more access to statistical information. Add to that the means to train teachers, administrators, and decisionmakers to both understand and apply SBR, and we may have a real winner here.
Data is important, but it is also dangerous. Put it in the hands of someone who doesn’t understand what they are looking at (or worse, thinks they do when they don’t), and you can do far more damage than just maintaining the status quo. As part of our ed R&D investment, we should be training a cadre of educator scientists who help practitioners distill the facts, identify what works, and move that research into practice. That was the goal, six years ago, with NCLB and SBR. And that should still be our goal today.
Yes, Eduflack knows he is a cynic. But after this forum, he is cautiously optimistic. SBR is no longer a punchline to a status quoer’s ed reform joke. We all seem to understand the importance of sound, replicable research. Now, we are starting to break it down and see what makes the cut and what hits the trash. With luck (and real commitment), we should see some wholesale understanding and implementation by 2014. Let’s just hope we’re all there to see it (and still give a damn about it).
And Now a Word from a RF Godfather …
Last week, the education reform community was abuzz with discussion on IES’ interim report on Reading First’s effectiveness. There’s been a great deal of good talk on the topic, particularly from Mike Petrilli and the folks over at Fordham Foundation.
Today, we’ve got a great interview on Educationnews.org with Reid Lyon. If you’re bothering to read anything on RF, then you have to know who Reid is. I won’t try to summarize the interview, for I couldn’t do it justice. Instead, I’ll just forward the link.
Be warned, it is long. But it is chock full of good information and good opinion. Definitely worth the read. And it begs the question — what is the other $140 million set aside for RF assessment/evaluation being used for?
http://ednews.org/articles/25335/1/Interview-with-Reid-Lyon-Reading-First-is-the-largest-concerted-reading-intervention-program-in-the-history-of-the-civilized-world/Page1.html
Happy reading!
RF: Read All About It
Today’s big education news story seems to be the IES study on the effectiveness of Reading First. For those who have missed the IES announcement of the study, or the USA Today or NY Times piece, or the countless blog entries, the good researchers over at IES determined that Reading First has been ineffective, to date. Looking at elementary schools implementing RF programs, the researchers found that teacher behavior has changed, but student performance still has not improved.
Some are already questioning the methodology, asking if the type of poor-performing school studied by IES impacted the outcome. And more criticisms are sure to come.
Ask Eduflack, and he thinks it is still too soon to know the true effectiveness. If you ask a good educational researcher, they’ll tell you it typically takes at least five years to see the effectiveness of a reform. RF was signed into law in 2002, with state grant applications soon following. That means the earliest checks were likely cut for the 2003-04 school year. So if we’re lucky, IES has looking at year three, maybe year four of implementation. So let’s give it another year or two before we eulogize Reading First.
The bigger issue, though, is the implications of the study. Many will use this to reinforce the IG findings and to validate the attacks that RF has faced from the beginning. Think about it — if the implementation was bad, the awards were skewed, and the impact non-existent, the law must be no good. Right?
But we’re truly missing the bigger picture. The IG investigation and the recent Sol Stern/Fordham Foundation report have reached similar conclusions. RF is a well-intentioned and well-conceived program. The flaw was in the implementation. The feds, the SEAs, and the LEAs have not followed the true letter or spirit of the law. Some cut corners. Some skipped sections of the law. And some simply didn’t understand it.
Read Stern’s report closely. Talk to the brains behind the law — the Reid Lyons and the Bob Sweets of the world — and they will tell you the same thing. The law is strong. We need to better enforce it. We need to better follow it. We need to better live it.
Instead, we let the status quoers use RF funding to support non-SBRR programs. We let schools continue so-called balanced literacy programs. And we failed to ensure that “what works” was really getting into the classroom.
I’d still like to believe that RF can be saved. We have the technology to rebuild her. If the IES study tells us anything, it is that we need to enforce RF with greater fidelity. We need to follow both the letter and the intent of the law. If we don’t, we may hasten the death of RF and the implementation of SBRR. And that’s no good for the teachers who have already changed their practice and for the kids who need to be reading at grade level.
Eduflack By Patrick R. Riccards 