A map to student success. It is an intriguing concept. Today at the Reading First Conference, Deputy Education Secretary Ray Simon proudly placed the label on the Reading First program, boldly proclaiming we needed more RF. “We need more Reading First, not less. That makes sense!” he said to a cheering room of 5,000 educators.
In doing so, Simon offers an intriguing idea. Let’s set aside the politics of RF. Let’s ignore the whole language zealots for a while and try to mute out those critics who have sought to sabotage one of the most significant public investments in student achievement in recent years. Instead, let’s just think about the notion of a map to student success.
How do we become such cartographers? First, we set goals. Check, all children reading at grade level by fourth grade. Then we identify the research base. Check, see National Research Council and National Reading Panel, among others. Then we correlate the research base to an instructional approach. Check, thanks to Reading First. Then we train teachers to deliver the instructional approach with fidelity. Still a work in progress. Finally, we equip our educators and instructional leaders to effectively capture data to determine if the delivery of our approach is achieving our goals. And on that point, we all know the verdict is still out.
I share Simon’s and First Lady Laura Bush’s desire to see RF funding continue. The program hasn’t been in place long enough to truly measure its effectiveness. Zeroing out funding now just adds RF to the pile of good ideas lacking the complete follow-through to know if they really worked. But even if RF doesn’t see another dime of federal money, Simon is absolutely right. RF has provided us a clear map to student success, a map that can be replicated to help build effective student learning opportunities, be they in reading, math, and science, or even social studies, foreign languages, or the arts.
RF has also provided us a map of how — and how not — to move a good idea from concept paper to legislation to law to the classroom. It has shown us the need for a big tent of advocates, supporters, and champions. It has demonstrated the need to secure buy-in from both the ivory towers and Main Street, USA. And it has firmly declared that good data and better student performance numbers will always rule the roost.
The challenge, and opportunity, today is getting that map into the hands of every teacher, parent, policymaker, funder, and concerned citizen. We know how to get to a nation where virtually all children are proficient or better in reading. That path is available to all those who care. Now we just need more people to use the map for its intended purposes, whether the map’s developer is defunct or not.
SBRR
Forget the Drumbeat, Its a RF Orchestra
Need proof that Reading First is working in classrooms throughout the United States? Need to see for yourself that teachers have embraced the law and the impact it is having on students? All you have to do is check out the throngs of teachers and school leaders assembled here in Nashville for the National Reading First Conference.
In Tennessee this week, representatives from RF districts across the nation are here to continue to learn and improve their practice. They are here to ensure that research-proven practice will continue to permeate the schools. They are here to continue building on their understanding of the research base. And they are here to celebrate the law.
Eduflack will be posting observations from the conference throughout the week. We have First Lady Laura Bush addressing the assembly today. Noted reading researcher Tim Shanahan is slated to discuss ELL. But the important discussions will be in the breakouts and the hallways, as these passionate educators share their personal experiences with one another, turning the entire week into an extended teaching moment.
Hopefully, this isn’t the swan song for RF. If Congress continues its ways, and fails to pass the FY2009 budget, Reading First will receive an additional year of funding (albeit at the lower FY2008 funding level). And maybe, just maybe, a new President and a new EdSec will find a way to save the law. We owe that glimmer of hope to each and every educator gathered here this week, who refuse to give up and remain committed to ensuring every student gains the reading skills they need to succeed.
“Legacy”
For years now — well before the lawsuits, the IG investigations, and the delays in reauthorization — Eduflack trumpeted that No Child Left Behind could and should be this Administration’s domestic policy legacy. Like it or not, NCLB had the opportunity to transform and improve public education in the United States for decades to come.
I can hear the belly laughs already, but think about it. The largest federal investment in K-12 education in history. A commitment to improving student achievement. Unmatched accountability. Proven-effective, research-based instruction. Content-based professional development. Supplemental education and school choice for those in struggling schools. Every child reading at grade level by fourth grade. Education that was results based, not process based. A sea change from the status quo.
The opportunities to cement that legacy have been there. When Margaret Spellings took over in 2005. When high school improvement gained attention from ED in 2005 and 2006. The release of the NCLB Commission report in early 2007, following by genuine congressional interest to reauthorize and strengthen the law. There was even the moment when Spellings declared the law 99.99% pure. All were opportunities, and virtually all were squandered. Opportunities lost, legacies missed.
In today’s USA Today, Greg Toppo quotes our educator in chief — First Lady Laura Bush — as stating that NCLB will indeed be a legacy of her husband’s Administration. The question today, though, is what type of legacy will it become? In 2005 or 2006, the opportunity was there to demonstrate the enormous benefit the law — or at least the intent of the law — could have on K-12 education throughout the nation. Today, that legacy has the strong possibility of being cloaked in negativity, leaving a lasting mark for unfunded mandates, high-stakes testing, and teacher-proof instruction.
It doesn’t have to be that way. Spellings and her team have six months remaining to leave the legacy the law should have, the legacy deserved by the good folks who created NCLB nearly seven years ago. Even without reauthorization (which none of us expect to see before a new edsec takes the helm at Maryland Ave.), there is one last chance to do it right.
Continued flexibility for the states is a good start. Marketing recent reading and math gains for the students who needed NCLB the most helps too. Spotlighting the teachers and schools who have improved under the law reminds us all of what is possible. Reminding us that NCLB is about more than just elementary school, as evidenced by ED’s American competitiveness work goes a long way, as does promotion of the law’s investment in teachers and their continued training and development. And who can argue with the value of better data and better understanding of data, allowing our schools to use such information to make better spending, leadership, and instructional decisions.
Of course, Eduflack would personally like to see a metaphorical charge up San Juan Hill to save Reading First, reminding the world that literacy skills are needed to succeed in school, career, and life, and the only way to gain those skills is to ensure that our classrooms are using only the very best and the very proven instructional approaches.
So what comes next? Spellings and company have six months. They lose two of them for the election, and lose a few more weeks in January for transition. That leaves three months for a legacy campaign. Hard, yes. Impossible, not quite. But the clock is ticking. The question remains … is anyone at ED watching the clock?
The Neverending Quest for Good Data
Why is it so hard to find good, meaningful scientific data to prove the efficacy of an education reform? Do we know what good data is? Is it too expensive to capture? Is it deemed unnecessary in the current environment? Is it out-of-whack with the thinking of the status quoers?
EdWeek’s Kathleen Manzo has been raising some of these issues over on her blog — Curriculum Matters. (http://blogs.edweek.org/edweek/curriculum/) And no, Eduflack has no qualms whatsoever with her taking me to task on whether the proof points I use to demonstrate Reading First is working are truly scientifically based proof points. To the contrary, I appreciate the demand to “show me” and have greatly enjoyed the offline conversations with Manzo on what research is out there and whether that research — the good, the bad, and the ugly — meets the hard standards we expect.
For the record, I am not a methodologist, a neuropsychologist, nor an academic to the nth degree. I learned about research methodology and standards and expected outcomes from NRPers like Tim Shanahan and Sally Shaywitz and from NICHDers such as Reid Lyon and Peggy McCardle. My knowledge was gained on the streets, so take it for what it is worth.
When NCLB and RF were passed into law, the education community took a collective gasp of concern over the new definition of education research. The era of squishy research was over. The time for passing action research or customer satisfaction surveys as scientific proofs of effectiveness had met its end. Folks starting scratching their heads, wondering how they would implement (and fund) the longitudinal, double-blind, control-grouped studies defined as scientifically based education research.
The common line in 2002 and 2003 was that only two reading programs, for instance, met the research standards in SBRR. Those two? Direct Instruction and Success for All. Not Open Court. Not Reading Recovery. Not Voyager. Only DI and SFA.
So what has happened over the years? In 2002, the fear was that every educational publisher would have to adopt a medical model-style research network a la NICHD. Millions upon millions of dollars would need to be spent by the basals to prove efficacy. It was to be a new world order in educational research.
Where are we today? As Manzo correctly points out, five years later there is little (if any) research out there that is now really meeting the standard. Even the large IES interim study of RF effectiveness — that $31 million study of our RF districts — fails to meet our standards for high-quality, scientific research (if you listen to the researchers who know best). Why? Why is it so difficult for us to gather research that is so important?
First, we have interpreted the law the way we want to interpret the law … and not the way it was written or intended. Those being asked to implement the research models simply didn’t want to believe that Reid Lyon and Bob Sweet really wanted them to pursue such zealous and comprehensive research. So it was interpreted differently. Neither consumers (school districts, teachers, and parents) nor suppliers (basals, SES providers, etc.) saw the necessity of longitudinal, control-grouped, double-blind, peer-reviewed research. We settled for what we could get. We knew that documents such as the NRP report of the previous National Research Council study met the requirements. So instead of doing our own research, in the early years of RF we simply attached the NRP study as our “research base” to demonstrate efficacy. Forget that the ink on the instructional program wasn’t dry, it was “scientifically based.” And there were no checks or review process to prove otherwise.
Second, we are an impatient people, particularly in the education reform community. Take a look at the NICHD reading research network, and you’ll see it takes a minimum of five years to see meaningful, long-term impact of a particular intervention. RF grants were first awarded in 2002, with most early funders using the money for the 2003-04 school year to start. That means just now — for the 2008-09 school year — would we truly be able to see the impact of RF interventions. But have we waited? Of course not. We declared victory (or defeat) within a year or two of funding. If test scores didn’t increase after the first full academic year, the nattering nabobs of the status quo immediately declared RF a failure, simultaneously condemning the need for “good” research.
We need to see results. If our second grader isn’t reading, we want her reading by third grade, tops. We don’t have the patience or the attention span to wait five to seven years to see the true efficacy of the instruction. We need a research model that provides short-term rewards, instead of measuring the long-term effects we need. A shame, yes, but a reality nonetheless.
The final side to our research problem triangle is the notion of control groups. In good science, we need control groups to properly measure the effects of intervention. How else do we know if the intervention, and not just a change in environment or a better pool of students, should be credited or student gains? That is one of the great problems with the IES interim study. We are measuring the impact of RF funding, but were unable to establish control groups that did not benefit from RF materials, instruction, and PD (even if they didn’t receive any hard RF dollars).
But in our real-life classroom environment, who wants their kid to be in that control group? We all want the best for our children; we don’t want them to get the sugar pill while all the other students are getting scientifically based reading and a real leg up on life. How do you say to teachers — in our age of collective bargaining — that these teachers on my right will get scientifically based professional development, but these two on my left will get nothing? How do we say these students on this side of the district will get research-based instruction and materials, but this cluster here will get instruction we know to be ineffective. Politically, our schools and their leaders can’t let real scientifically based research happen in their schools. Too much grief. Too many problems. Too little perceived impact.
So where does this all leave us? At the end of the day, we all seem to be making do with the research we can get, hoping it can be held to some standard when it comes to both methodology and outcomes. We expect it to have enough students in the study so we can disaggregate the data and make some assumptions. We expect to do the best we can with the info we can get.
Today, we see that most “scientifically based” research is cut from the same cloth. No, we aren’t following the medical model established by NICHD’s reading network, nor are we following the letter of the law as called for under NCLB and RF. Some come close, and I would again refer folks to the recent RF impact studies conducted in states such as Idaho and Ohio. The methodology is strong, the data is meaningful. And it shows RF is working.
What we are mostly seeing, though, is outcomes-based data. School X scores XX% on the state reading assessment last year. This year they introduced Y intervention, and scores increased XX%. Is it ideal? No. But it is a definite start. We are a better education community when we are collecting, analyzing, understanding, and applying data. Looking at year-on-year improvement helps us start that learning process and helps us improve our classrooms. It isn’t the solution, but it is an important step to getting there (particularly if we are holding all schools and students to a strong, singular learning standard).
Yes, Kathleen, we do need better research. We know what we need, we know how to get there. But until we demonstrate a need and a sense of urgency for the type of research NCLB and IES are hoping for, we need to take the incremental steps to get us there. Let’s leave the squishy research of days of old dead and buried. We’ve made progress on education research over the past five years. We need to build on it, not destroy it.
No Literacy Crisis?!?
It now seems we are in the full funeral procession for Reading First. In today’s USA Today’s dueling editorials, the nation’s newspaper of record calls for Reading First to be brought back from the dead. USA Today notes (http://blogs.usatoday.com/oped/2008/07/our-view-on-lit.html#more) that the program works, evidenced by the growing number of statements from educators and from recent studies — such as those released last month by CEP — that demonstrate improvement. And RF offices in states from Idaho to Ohio to Alabama have added their voice to save the necessary program.
The latest defense of the program can be found in yesterday’s Boston Herald (http://www.bostonherald.com/news/opinion/op_ed/view.bg?articleid=1105503), in a passionate piece from the head of the Bay State Reading Institute.
Eduflack has said it once, and I’ll say it again. RF works. It has boosted student reading achievement, as demonstrated by student assessments in states across the nation. It has improved reading teacher development, empowering teachers to use data to target specific interventions at the students who need them. And it has focused SEA and LEA spending on programs that work, demanding solid, research-based proof. After flatlining for decades, achievement is on the rise, and improvements are due, in large part, to the direct and indirect impact of Reading First and the embrace of scientifically based reading.
Unfortunately, such results don’t seem to be enough for some people. Case in point — Stephen Krashen’s opposition-editorial (http://blogs.usatoday.com/oped/2008/07/hooked-on-phail.html#more) to USA Today’s cogent stance. Krashen reheats many of the same misguided stances of years past. He embraces the urban legends and conspiracy theories of NRP attacker Elaine Garan. And he reiterates the fallacy that RF is all about phonics, and nothing about reading comprehension (an untruth we all should know to be false).
Krashen raises two points, though, that merit continued discussion. The first is the notion that 99 percent of Americans (adults and kids) can read and write at basic levels. He uses this to say there is no literacy crisis, and thus no need to change the way we teach or increase our worry about student reading achievement. 99 percent? Really?
First, we must look at his term “basic.” In our 50 states, we define reading ability by students who are proficient or better. Basic does not mean proficient. It means a fourth grader reading at a first grade level, or an adult with third grade literacy skills. In today’s society, basic doesn’t cut it. We need proficient readers. And like it or not, 40 percent of today’s fourth graders are not proficient readers. That should signal crisis to every teacher, parent, business leader, or elected official in this country.
More disturbing, though, are Krashen’s closing comments. Of course he applauds the death blow to RF. In doing so, he advocates that the money should be spent on libraries in urban areas. A noble goal, yes, but it demonstrates a complete naiveté when it comes to federal appropriations. The death of RF means the elimination of federal reading funding. It does not mean we get to propose new programs to fund. Nor does not mean we now have a bucket of $300 million or $1 billion of education funding that can now be spent on other programs. The checkbook is closed. The well is dry. This is a bad thing.
For more than a decade now, reading instruction has received hundreds of millions of dollars from the federal government. Funding started under the Clinton Administration and America Reads in 1997. It increased in 2002 with the passage of RF. Since then, billions of dollars has been directed to the LEAs to improve reading instruction in the communities that need it most.
Over the past two years, that money has been cut. This year, Congress is zeroing out the funds. For the first time in more than a decade, the federal government will not be supporting reading instruction in this country. And that is a cryin’ shame. For folks like Dr. Krashen — who have dedicated their professional lives to literacy — they should be embarrassed that RF has been defunded. They should be offended that reading instruction is no longer a federal priority. Instead of celebrating their defeat of RF, they should be outraged that a program supported by everyone from the IRA to the U.S. Department of Education has prematurely come to end. They should be fighting to save reading funding for teachers and schools, not throwing parties because a program supported by their opponents in the faculty senate has been dealt a defeat.
But why should we expect that? That would put kids and teachers above research dollars, professional reputations, and “ideological” camps.
A Display of RF Commitment
Sometimes, we just have to trust our gut. Despite the white noise around us. Despite what the nattering nabobs are saying. We just have to go with what we know, make a decision, stand behind it, and reap the benefits.
That seems to be the MO that the good educators down in Louisiana are following. Yesterday’s Shreveport Times reports that the Caddo School District have committed $1.6 million to continue funding their Reading First programs. And if the feds don’t make the funding available, they will find the money themselves in the district’s general fund. The full story can be found here — http://www.shreveporttimes.com/apps/pbcs.dll/article?AID=2008806050330.
Why? The good folks in Caddo know that RF works. They’ve seen it help their low-performing, high-poverty K-3 classrooms, and they don’t want to lose that help. They recognize that once you find something that works, and I mean really works, you do what is necessary to keep it in place.
The educators and decisionmakers down in Caddo deserve some credit. Despite all of the RF “sky is falling” chickens out their waiving around the IES RF interim study, Caddo knows a good thing when they see it. RF works in schools like theirs and with kids like theirs. It works, and you can’t take that away from them.
Over at Flypaper, Mike Petrilli praises the folks in Caddo for stepping up to the plate and agreeing to fund RF even if Congress won’t. And he’s absolutely right. It’s easy for a district to complain about a federal decision, and bemoan stripped funding and say “if only.” Caddo Interim Superintendent Wanda Gunn is acting, not talking. If the feds won’t do it, she’ll do it herself.
But the situation down in Shreveport also raises one important issue that ed reformers and RF advocates alike need to be mindful of. The Shreveport Times positions this as Caddo spending $1.6 million on “Reading First,” as if the federal funding law were an off-the-shelf basal reading curriculum that school districts can pick up at their next trip to the store. If only it were that easy.
RF provides clear guidelines about the sorts of reading programs that should be implemented in the low-income, low-performing schools most in need of assistance. It requires an educated, savvy superintendent, curriculum director, or reading teacher to take those guidelines, gain an understanding of scientifically based reading research, and make an educated decision on what is best for them, their schools, and their kids.
Despite the growing urban legend, there is no golden list of reading programs that guarantees both federal funding and student success. It falls to educators to make their way through the smoke, move beyond the mirrors, and really identify the most effective, research-based reading programs for their students. Programs that embody both the letter and the intent of the federal law.
It seems like the folks in Caddo have done that, and are putting their money where their mouths are. Here’s hoping other districts will do the same, continuing to build on the gains and successes of the past few years that can only be attributed to SBRR in the classroom.
Swingin’ at an RF Pitch
I know, I know, I promised my Quiotic quest over the IES Reading First implementation study was headed for the bench for a little bit. But after watching so many swing and miss at this RF pitch, Eduflack just has to offer plaudits when someone else makes solid contact and raises some great issues on this study.
Kudos go to Kathleen Kennedy Manzo over at Education Week. Manzo is one of the original RF reporters (along with Greg Toppo), having covered it from the early stages to today. It’s meant that she’s likely been flooded with information, data, research, opinion, and spin over these past six or so years. It’s meant a continuous learning process. And it’s meant having to sort through it all, avoiding the pitches in the dirt and waiting for the good pitch to hit.
Hit it she did. In this week’s Education Week, Manzo’s got a great piece on the IES study. http://www.edweek.org/ew/articles/2008/06/04/39read.h27.html?tmp=1914927477 She explores many of the quality issues that have been raised to date. More importantly, though, she gets Russ Whitehurst to state that no conclusions should be made based on the interim report. Instead, we need to wait for the final.
I, for one, am hoping that means there’s a whole lot of fixing coming in the final report. Of course, I’ve been disappointed before. Regardless, EdWeek and Manzo deserve credit for taking a complicated and growing issue, and reporting on it so that the average educator or the average policymaker understands the issues and knows the tough questions to ask.
Gold stars all around.
Is This the Ed of Our RF Study Quest?
For more than a week now, Eduflack has been a bit of a one-trick pony. Through the ole reform goggles, I’ve been unable to turn away from the issue of Reading First and IES’ interim study of this important law. It may have been a bit much for some, but it was something that just had to be done. Today, nearly 40 percent of fourth graders are still unable to read at grade level or better. We spend billions each year on textbooks and classroom libraries and SES programs. We are expecting nearly half of today’s teachers to retire in the next decade. So if not now, when?
With all of these factors, it only stands to reason that we should do anything and everything we can to ensure our schools — and our kids — are getting reading instruction that works. They need effective learning. How can anyone say that a student with no or poor reading skills has a real chance to succeed in society? They can’t. Reading is the building block for success in K-12, higher ed, career, and life.
For that reason, Eduflack has put the IES interim study under the microscope. We’ve heard from experts such as Reid Lyon, Tim Shanahan, and Richard Allington. We’ve scrutinized the methodology. We’ve pined for what could have been. Now we eagerly await for the next study that Dr. Shanahan has promised is on the way.
We close this chapter of the debate with questions, not with statements of fact. If the last week has taught me anything, it is that we know far less than we should. If these questions are keeping me up at night, they must be keeping others up as well. So I offer these so that the media, policymakers, educators, and influencers can ask them as well, knowing that together we may get some real answers.
* The Reading First law set aside $150 million for research and assessment over the last six years. By most reports, IES spent approximately $30 million. Where is the remainder of this money? What is it going toward? Are we measuring the effectiveness of this reallocation?
* What is the real intent of the IES study? Personally, I think we should be studying ROI for Reading First spending. Six years and billions of dollars later, where is student reading achievement? This study seems to be more process over outcomes.
* How can we measure RF versus non-RF schools or classrooms? Are we suggesting that non-RF schools are not using scientifically based reading in their classes? Of course not. Both buckets are using the same textbooks and have access to the same professional development and the same supplemental materials. The only real difference between RF and non-RF is from whose account the check is being cut.
* Forget how IES has interpreted it, what does the federal law say should be part of this assessment? RF has gotten into some trouble when it comes to the law’s intent (and letter) and its implementation. The law seems pretty clear and comprehensive to me. (Just check out section 1205)
* Why has IES taken a different path? And is there time to get us on the right path?
We need to follow the money here. Had IES spent the full $150 million and gotten a study like this back, advocates and nay-sayers would be screaming from the mountaintops about mismanagement and poor decisions. Yes, we have a bad study. But the nation was given the money to do a great study. Some would even say a $150 million national assessment study would be a researcher’s dream. So why wasn’t that dream fulfilled, particularly after Congress wrote the check to make it a reality? We’ve created a problem that never should have arisen.
A big check. Clear congressional intent. Opportunity to make a lasting, meaningful impact on both education and education research. It all was there. Today, we’re left holding a flawed study, and we still have no clear idea that RF — or more importantly, SBRR — works.
Yes, there is a value to doing an impact study like IES’. Such studies are valuable for the internal agency and for the structure of its future funding opportunities. But we also have a clear need for a study that tells us whether the program is working or not.
We need to get our kids reading. We need them reading at grade level. And we need to identify what works and get it in every classroom across the nation. Whatever it takes. Until we have answers to these questions, though, we may never have a national study that gives us the data — and the guidance — we need to make every child a reader.
I yield the floor and will hold my tongue until more data (and opinion, of course) is presented.
Under the Hood of the IES Reading Study
I know, I know, Eduflack is like a dog with an unbelievably potent bone on this whole IES interim study on Reading First. I can’t help it. Maybe its because I’m a contrarian. Maybe I hate to see folks pile on to something that deserves a good defense. And maybe I’m just practicing insanity, believing that if I keep focusing on the benefits again and again, someone may hear it and change their thinking and their practice.
I come here today not to prosthelatize on RF. Instead, I want to serve as a conduit for needed information. If we’ve learned anything from the back-and-forth on the IES study, it is that there are some real questions with regard to the methodology and the project design. Rather than just trust the salesman that the engine under the study hood is legit, I’ve brought in an expert mechanic of my own.
Today, we hear from the University of Illinois-Chicago’s Tim Shanahan. If you’ve heard of the IES study, you know Tim. A leader on the National Reading Panel, Dr. Shanahan has served on a number of similarly influential groups on reading instruction. He is also the former reading czar of Chicago Public Schools and recently completed his tenure as president of the International Reading Association.
I met Tim a decade ago, when I began my service to the NRP. Immediately, I found that he was one of those rare breeds who knew the research cold, but could explain it to anyone’s grandma so she understood it … thoroughly and completely. Even more, he had the patience and the perseverance to teach this old dog about research methodology and scientific approaches, giving me the foundational understandings I have put to use virtually every day since.
Put simply, there are few researchers I trust more than Dr. Tim Shanahan. He is as straight a shooter as they come. And for our purposes today, Tim was an advisor to the IES study, so he knows of what he speaks. So we asked some questions, he provided far better answers.
EDUFLACK: What does the IES study really say? How strong are the findings?
SHANAHAN: THE IMPLEMENTATION STUDIES INDICATE THAT THE DIFFERENCES BETWEEN RF AND NON-RF SCHOOLS WERE PRETTY MODEST (ABOUT 50 MINUTES OF INSTRUCTIONAL DIFFERENCE PER YEAR IN AMOUNT OF INSTRUCTION), MEANING THAT RF KIDS PROBABLY RECEIVED FEWER THAN 30 HOURS OF ADDITIONAL READING INSTRUCTION EACH YEAR DUE TO THE INTERVENTION. CLEARLY A MODEST INTERVENTION, ESPECIALLY GIVEN THE SIMILARITIES IN CURRICULUM, INSTRUCTIONAL MATERIALS, PROFESSIONAL DEVELOPMENT, AND ASSESSMENTS.Q: How valid are the findings, knowing there may be contamination across groups (that both the RF and non-RF groups may have been doing the same things in the classroom)?
A: MOST SCHOOLS EMPLOY SOME KIND OF COMMERCIAL CORE PROGRAM. WHEN READING FIRST EMPHASIZED THE ADOPTION OF PROGRAMS WITH CERTAIN DESIGNS ALL MAJOR PUBLISHERS CHANGED THEIR DESIGNS TO MATCH THE REQUIREMENTS.
READING FIRST SCHOOLS ALL BOUGHT NEW PROGRAMS IN YEAR 1; ALMOST ALL OTHER TITLE I SCHOOLS ADOPT NEW CORE PROGRAMS EVERY FOUR OR FIVE YEARS. THAT MEANS IN YEAR 1, 100% OF THE RF SCHOOLS GOT A NEW PROGRAM, AND 25% OF THE OTHER SCHOOLS DID. IN YEAR 2, THAT NUMBER WENT TO 50%, IN YEAR THREE 75%. ALL RF SCHOOLS HIRED COACHES IN YEAR 1, SO DID MORE THAN 80% OF THE OTHER SCHOOLS. ETC.
THIS ISN’T A CASE OF SPOT CONTAMINATION, IT WAS INTENTIONAL AND PERVASIVE (IN FACT, IT WAS PART OF THE RF LAW ITSELF—20% OF THE STATE MONEY, THAT MEANS $1 BILLION TOTAL WAS DEVOTED TO GETTING NON-READING FIRST SCHOOLS TO ADOPT THESE REFORMS).
Q: Given that contamination, are there contamination rates that can be tolerated in the design? For example, let’s say 15 percent of the RF and comparison groups received identical programs/PD. Is this level of contamination tolerable? What if there is a 30 percent overlap – is this level tolerable? Are there ways to estimate the degree to which percent contamination will indicate a need to increase sample size?
A: THE PERCENTAGES OF OVERLAP WERE 75-100% DEPENDING ON THE VARIABLE. THE ONLY ONE WHERE WE HAVE ANY KIND OF IDEA ABOUT WHAT IS TOLERABLE IS WITH TIME.FROM PAST RESEARCH, ONE SUSPECTS THAT 100 HOURS OF ADDITIONAL INSTRUCTION WOULD HAVE A HIGH LIKELIHOOD OF GENERATING A LEARNING DIFFERENCE, A 50-60 HOUR DIFFERENCE WOULD STILL HAVE A REASONABLE CHANCE OF RESULTING IN A DIFFERENCE. AT 25-30 HOURS A SMALL DIFFERENCE IN LEARNING MIGHT BE OBTAINED, BUT IT IS MUCH LESS LIKELY (ESPECIALLY IF THE CURRICULA WERE THE SAME).
A: IT [THE IES STUDY] NOT ONLY DID NOT TRY TO AVOID CONTAMINATION, IT COULDN’T POSSIBLY DO IT SINCE THE SOURCES OF THE CONTAMINATION WERE SO PERVASIVE. FIRST, THE FEDERAL POLICY EXPLICITLY CALLED FOR SUCH CONTAMINATION TO BE PUSHED. SECOND, STATES AND LOCAL DISTRICTS MADE THEIR OWN CHOICES (AND THEY FELT ENTICED OR PRESSURED TO MATCH RF).
FOR EXAMPLE, SYRACUSE, NY RECEIVED READING FIRST MONEY FOR SOME SCHOOLS, BUT MANDATED THAT ALL OF ITS SCHOOLS ADOPT THE SAME POLICIES AND PROGRAMS. THERE SHOULD HAVE BEEN NO DIFFERENCES BETWEEN RF AND NON-RF SCHOOLS IN SYRACUSE, THE ONLY DIFFERENCE WOULD BE IN FUNDING STREAM—HOW THE CHANGES WERE PAID FOR, AS THE NON-RF SCHOOLS ATTENDED THE SAME MEETINGS AND TRAININGS, ADOPTED THE SAME BOOKS AND ASSESSMENTS, RECEIVED THE SAME COACHING, PUT IN PLACE THE SAME POLICIES, ETC.
Q: Did the evaluation design describe practices in the comparison groups?
A: YES, THE IMPLEMENTATION STUDIES SHOW THE SIMILARITIES IN PRACTICES AND HOW, OVER TIME, THE PRACTICES THAT WERE SIMILAR AT THE BEGINNING BECAME INCREASINGLY SIMILAR EACH YEAR. THAT WILL BE CLEARER IN THE NEXT STUDY OUTQ: Did the evaluation design account in any way for contamination, crossover, compensatory rivalry, etc.?
A: NO. THE FEDERAL LAW CALLED FOR THE EVALUATION OF READING FIRST IN TERMS OF THE EFFECTIVENESS OF THE INSTRUCTIONAL MODEL, BUT DID NOT CALL FOR A STUDY OF THE IMPACT OF READING FIRST UPON THE ENTIRE EDUCATIONAL SYSTEM.EVEN THOUGH I HAD PERSONALLY MADE A BIG DEAL OUT OF THE PROBLEM FROM THE VERY FIRST STUDY DESIGN MEETING, THE METHODOLOGISTS THOUGHT THEY COULD HANDLE MY PROBLEM SIMPLY BY ACCOUNTING FOR THE RF ROLLOUT EACH YEAR. THEIR ASSUMPTION WAS THAT RF WOULD IMPLEMENT SOME CHANGES IN YEAR 1, OTHERS IN YEAR 2, AND STILL OTHERS IN YEAR 3 AND THAT THIS PATTERN OF IMPLEMENTATION WOULD ALLOW THEM TO EXAMINE A CONTINUING LAG BETWEEN THE RF AND NON-RF SCHOOLS.
I DIDN’T UNDERSTAND THAT THEY WERE THINKING THAT AND THEY NEVER ASKED DIRECTLY ABOUT THAT. LAST YEAR, I FIGURED OUT WHAT THEY WERE THINKING AND I HAD TO EXPLAIN SEVERAL TIMES THAT RF PUT ALL OF ITS REFORMS IN PLACE DURING YEAR 1, WITH NOTHING NEW IN YEARS 2 AND 3, SO IT WOULD BE IMPOSSIBLE TO TEST THE EFFECTS OF DIFFERENT PARTS OF THE IMPLEMENTATION, ETC. USING THEIR APPROACH. I MIGHT HAVE BEEN ABLE TO GET THIS FIXED IF I HAD UNDERSTOOD THAT THEY WERE ASSUMING THAT KIND OF DESIGN (OR IF THEY HAD ASKED ME ABOUT THAT SPECIFICALLY).
Q: Can we assume that the RF group is just like the comparison group except for exposure to RF funding?
A: READ THE IMPLEMENTATION PART OF THE REPORT (AND THERE IS ANOTHER STUDY COMING LATER THAT WILL MAKE THIS CLEARER) AND YOU’LL SEE THE DEGREE OF SIMILARITY IN THE KEY FACTORS BETWEEN THE TWO SETS OF SCHOOLS. I RAISED THIS AS A THEORETICAL PROBLEM ORIGINALLY, BUT THE IMPLEMENTATION STUDY CLEARLY SHOWS THAT CONTAMINATION WAS A BIG PROBLEM (IT CANNOT TELL US WHETHER THE CONTAMINATION CAME FROM THE $1 BILLION FEDERAL EXPENDITURE ON THIS, BECAUSE THE STATES AND LOCAL DISTRICTS OFTEN SIMPLY ADOPTED THE SAME IDEAS.AS ONE ILLINOIS DISTRICT TOLD ME, “IF THIS IS THE RIGHT STUFF TO DO, THEN WE ARE GOING TO DO IT WITH EVERYONE.”
That’s a lot to chew on, but it is a worthwhile meal. Even for the most simple-minded of laypeople (like Eduflack), it is clear that the IES study had no real control group. We had RF schools and non-RF schools, both pools of which were doing similar things with similar materials. How can we compare the two groups as haves and have nots when the only measure of separation is the bucket of money that was paying for the approach?
Dr. Whitehurst, I’ll yield the pulpit to you if you’d like to respond.
Is 100% Proficiency Possible? You Betcha
Since its gaining its moniker, No Child Left Behind has faced growing scrutiny about its goal — ensuring that every student is achieving at grade level. On the reading side of the coin, when NCLB was passed into law, only 60 percent of fourth graders were proficient or better at reading. Two of every five students was struggling at reading. The goal was to get all five of them reading, offering scientifically based interventions to fill the gaps.
Such promises became a punchline for folks. It seemed like some would have felt better if we had said “Only 10 Percent Left Behind” or “Just a Few Left Behind.”
Today’s Washington Post, though, shows that 100 percent proficiency is not just a campaign slogan, it can be a way of life for some schools. Over at the Core Knowledge Blog, they’ve done a good job discussing this very topic, and the fact that a school in Ocean City has already completely fulfilled its AYP obligations. Check it out at http://www.coreknowledge.org/blog/2008/05/28/no-child-no-problem/.
Such gains are not just left to our beachside communities. We are starting to see more and more examples of schools that have cracked the code and have figured out how to get every child reading and get every child performing. Case in point, Pennsylvania’s Souderton Collaborative Charter School.
Full disclosure, I recently came across Souderton as part of my day job. Based in Montgomery County, PA, this K-8 school has clear academic goals. For language arts, that goal is to “read with comprehension, to write with skill, and to communicate effectively and responsibly in a variety of ways/settings.”
To achieve this goal, the school leadership adopted a scientifically based approach to independent reading. The school provides books on topics of interest to the student, at reading levels and content appropriate to the students’ age. In return, the students develop an interest and a passion for reading, developing the skills they need to succeed in ELA and other classroom results.
The result? Success. Don’t believe Eduflack? Take a look at Souderton’s results on the PSSA for 2005-06 — Pennsylvania’s state assessment. Third grade PSSA reading scores — 100% proficient or better. Fourth grade PSSA reading scores — 100% proficient or better. Even seventh grade reading scores — 100% proficient or better. That’s every child reading at grade level.
Souderton achieved this, in part, because they are using approaches that are proven effective. Their reading instruction models the best practices called for by the National Reading Panel and Reading First. They are empowering both students and teachers, inspiring both to achieve. And the results show.
Ocean City and Souderton can’t be the only schools with these sorts of results. While schools don’t have to be 100 percent proficient until 2014, I have a feeling that these two schools are but the tip of the iceberg when it comes to unsung heroes that are achieving despite the white noise of failure and impossibility. We should be modeling behaviors after schools like OC and Souderton. And we, including Eduflack, should be doing a better job uncovering those schools that are doing it right. Finding those schools that are achieving. Throwing the spotlight on those communities where SBRR works, and where student reading proficiency is the norm, not the exception.
Eduflack By Patrick R. Riccards 