The National Reading Panel: Using Research to Create More Literate Students
An invited contribution
University of Illinois at Chicago
After reading these comments, please visit the discussion forum to view readers' comments. To find a list of Reading Online postings related to this commentary, click here.
Controversies over reading education -- the so-called reading wars -- have raged for most of the past decade. In 1997, in response to these disputes, the United States Congress asked that a National Reading Panel (NRP) be established. (Click here to go to an excerpt from the Senate Committee on Appropriations report that accompanied the bill.) The panel was charged with determining what research has shown about the effectiveness of instructional approaches, the readiness of these approaches for translation to practice, and the need for future research. In other words, the panel was asked to decide what works in reading education on the basis of a formal review of research. I am a member of the panel.
P. David Pearson (1999) recently reviewed a series of historical, authoritative reports about reading education, including Learning to Read: The Great Debate (Chall, 1967), Beginning to Read: Thinking and Learning about Print (Adams, 1990), Becoming a Nation of Readers: The Report of the Commission on Reading (Anderson, Hiebert, Scott, & Wilkinson, 1985), and Preventing Reading Difficulties in Young Children (Snow, Burns, & Griffin, 1998, online document). One could easily assume that the NRP will just develop another report similar to these. Making that assumption would be a mistake, however. The National Reading Panel is unique, and the report that it develops will have several qualities and characteristics that will set it apart from the others in this litany.
The fundamental idea behind the federal government's establishment of a review panel is something along these lines: A major controversy is compromising the commonweal (in this case, children are not being taught to read as well as they should be) and undermining public confidence (i.e., trust in schools is declining). To ensure a reasonable standard of quality and to protect respect for public institutions and professions, an authoritative group is appointed to carry out an objective review of the research and to decide upon a standard of practice. The federal government then endorses this standard and benefits are provided to those whose professional practice is consonant with it. For example, in medicine, once such a standard is set, private insurance companies and U.S. government programs such as Medicare will only pay for procedures that are in keeping with the established standard. Physicians also gain some protection against liability if their practices match the standard. Although scientific review panels have a long history in medicine, this approach to resolving an empirical controversy is extraordinary in the annals of reading education -- and, indeed, review panels have never before been tried in any area of education.
Representative Anne M. Northup (a Republican from Kentucky), one of the sponsors of the legislation that called for the panel, told us about her concerns for children whose educational needs are not being met -- an issue she has had personal experience with, as one of her own children had difficulty learning to read. Her hope was that the NRP would identify and support instructional practices that would have higher rates of effectiveness for such children. She also indicated the frustration of several of her constituents -- school board members, teachers, and superintendents -- who felt that the schools were being taken advantage of by those who made claims for widely discrepant materials and methods, always on the basis of purported research. She hoped that the panel's report would help settle some of the seemingly endless wrangling and reduce false information about what research might have to say about reading instruction.
Needless to say, the appointment of the National Reading Panel has itself been controversial. For instance, on Feburary 18, 1998, Education Week ran a story entitled "New National Reading Panel Faulted Before It's Formed" (Manzo, 1998). In that article, Richard Allington, a former member of the Board of Directors of the International Reading Association, expressed his belief that the panel would not be able to do the job: To think that we can create a panel with no staff and little funding ... that is going to be able to provide us with any kind of comprehensiveness or reliability is unlikely.... The public and legislators are being led down a primrose path that suggests that research has the answer." And Kenneth Goodman has made his concerns well known, too. All this, even though the panel has, as yet, come to no conclusions.
One reason for the quick responses is the fact that the National Reading Panel is managed by the National Institute of Child Health and Human Development (NICHD). For some this raises the specter that only NICHD-supported research might be used, or that NICHD staffer Reid Lyon's well publicized conclusions (Grossen, n.d.) or Taylor (1998) will dictate the findings of the panel.
Originally, the panel was to make its report in December 1998, but a request to allow us to proceed through December 1999 was granted; we now have six months from the time of this writing to complete our work. My intention here is to sketch a description of how that work has gone so far and how it is to proceed the rest of the way.
Appointment of the Panel and How We Work
As envisioned in the legislation, the National Reading Panel was to have 15 members, including "leading scientists in reading research, representatives of colleges of education, reading teachers, educational administrators, and parents." The panelists were to be selected by the director of the NICHD, Duane Alexander, in consultation with Secretary of Education Richard Riley. Various professional organizations, government agencies, and individuals put forth recommendations for membership. (For example, I was nominated by the International Reading Association, National Reading Conference, and the Office of Educational Research and Improvement). In all, approximately 300 people received nominations. Each nominee was asked to provide a statement of willingness to take part in this activity, a vita or résumé, and a disclosure of economic interests in programs or materials used for teaching reading. Although some expressed concern that this requirement could "eliminate nearly everyone in the field" (Manzo, 1998), it is essential that national panels be free of such conflicts of interest, as part of their purpose is to restore or preserve public confidence. Many qualified nominees had to be bypassed because of their economic ties to commercial reading programs.
Fifteen panelists were selected, and meetings began in Washington, D.C., in April 1998. Soon after, panelist Robert Glazer resigned; he has not been replaced. Duane Alexander of the NICHD appointed F. William Dommel, Jr., a lawyer who specializes in research ethics, as NRP executive director, and Dr. Donald Langenberg, a physicist and chancellor of the University of Maryland system, serves as chair.
I am frequently asked why this panel is operated by the NICHD instead of the Department of Education -- reading is, after all, an education issue. The most obvious reason is that the NICHD has a history of operating such panels, and the education department does not. It has been useful to have advice and insight from men and women who have dealt before with this surprisingly complicated process.
Colleagues also often wonder what kinds of pressure the panel has been under from NICHD staff, or even from Reid Lyon himself. My sense is that there has been no pressure of this type -- none on me individually, and none on the panel when we have met together. We have been free to proceed largely by our own lights, and advice has more often been solicited by us than offered by the NICHD.
This is not to say that this has been a relaxed process, just one without evident political or ideological effort to bias our findings. In fact, the work has often been difficult and problematic. All of our meetings take place in public, usually with a small audience of bureaucrats, representatives of professional groups, and the occasional interested individual. A stenographer transcribes our exchanges, and everything is audiotaped. Because of the need for microphones and the size of the committee (small for the amount of work, but too large to meet around a single table), we often sit far apart. Not only are we watched while we work, but there is seemingly constant confusion and interruption as we forget to turn on our microphones and are reminded to do so. The drawback to all of this is that the work proceeds slowly, it is difficult to get to know the other panelists, and at times it takes real effort to get something said. The benefit, of course, is that there is a public record of all of our deliberations (summarized in minutes at the NRP website).
Another complication for panelists arose from the public hearings that were held in Chicago, Portland, Houston, New York City, and Jackson, Mississippi, to allow interested citizens to offer their advice and direction. (Minutes of each hearing are posted at the NRP site.) More than 400 people attended those regional meetings, and we heard from 117 presenters, including representatives of the International Reading Association, National Council of Teachers of English, National Institute for Literacy, Learning Disabilities Association of America, National Center for Learning Disabilities, International Dyslexia Association, and the Council for Exceptional Children (links to all of which can be found on the links page of the NRP site), as well as from many individuals and representatives of local agencies. This testimony, taken together with the National Research Council report Preventing Reading Difficulties in Young Children (Snow, Burns, & Griffin, 1998) and our own deliberations, led to the development of a list of approximately 30 topics that the panel might explore. Though we wanted to pursue all of these, we knew that there were some we would never get to. We had to make difficult decisions about priorities, and through a formal ranking and voting procedure we determined what we felt to be the topics that were of greatest importance or that could be explored most profitably. This suggests the need for future panels to examine issues that are certain to go unstudied. For example, one major topic that the panel wanted to pursue concerned the instructional needs of children who come to English as a second language. However, we were asked by the NICHD not to take that on, as future panels would deal directly with the topic of second-language learning.
The issues that we agreed to address include
Because there are only 14 panelists, and not all of them have the professional expertise needed to read and analyze the research studies, it was immediately evident that we needed to divide into various overlapping subgroups. Each of us is a member of one or more such groups, small teams that have taken responsibility for reviewing studies on one or more topics. This makes the work more manageable and will ensure that reviews will be completed on time.
Procedures for the Scientific Review
The panel had to decide upon a set of procedures for reviewing and analyzing studies. We were free to establish the procedures ourselves, though the precedents set by previous panels could be considered. We decided that we would use a meta-analytic review because of its explicit nature. Meta-analysis requires prespecified procedures for identifying and selecting the studies that will be reviewed as well as for analyzing the data they report. By setting such rules we hoped both to ensure that the reviews would be comparable and combinable in the end and to limit the bias that can result when only studies that fit a particular point of view are selected.
It has been suggested that the NRP will consider only research drawn from a single paradigm. There is some truth in this accusation, as our major determinations will require clear experimental or quasi-experimental evidence. This, however, is not because we do not recognize the value of qualitative studies or of other quantitative methodologies, but rather because of the nature of the determinations we have been charged with making. Our job is to decide what works -- what instructional methods, procedures, or programs can be used successfully to improve reading achievement. Experiments and quasi-experiments are the only research methods that try out a technique under real classroom conditions to determine their impact on learning, and it seemed to us unreasonable to indicate that any approach "worked" if such evidence was lacking. Observational data or correlational studies simply cannot answer the questions posed by Congress -- nor do they try to. Such research methods are useful for answering different kinds of questions. This is not to say that we plan to ignore such studies and reports. As I write this, it is still uncertain whether we will employ qualitative evidenceto judge the practicality (as opposed to the effectiveness) of particular approaches, or even whether relevant qualitative evidence exists. We are currently using search procedures that will preserve our capacity to employ qualitative studies (such as ethnographies) if this is deemed useful and desirable.
The panel has agreed upon and shared with Congress an explicit methodological plan that describes virtually every aspect of the study underway. If successfully accomplished, this plan will represent one of the most thorough, careful, and rigorous analyses of reading data ever conducted.
It is reasonable to wonder how our report will differ from Preventing Reading Difficulties in Young Children (Snow, Burns, & Griffin, 1998). One difference is that the latter publication focuses on reading problems -- and then only among young children -- while the NRP report will focus on a broader range of children, including those who do not experience particular problems, from kindergarten through grade 12. The biggest difference, however, is methodological. Preventing Reading Difficulties is quite uneven, with some findings well documented and thoroughly supported by empirical study while others, of apparently equal importance, are presented with little or no evidence. The reader is left to wonder why a particular study was not cited or why another was relied upon heavily when contradictory evidence seems to have been ignored. The report is silent on whether any kind of explicit, replicable review procedures were used in its preparation. With the NRP report readers should be able to replicate our search procedures, review the lists of studies that were considered, see why some studies were rejected from consideration, and even examine how the studies were coded for analysis. This will be, perhaps, the most thorough and explicit review of these topics ever conducted in reading. Because the process has been so open to public inspection and the research and review procedures are so explicit, the findings should go a long way toward allaying public concerns.
How the NRP Report Will Be Used and Other Controversies
Much of the criticism leveled at the National Reading Panel has assumed that its findings are predetermined. In fact, even at this late date, it still is not clear what those findings will be. S. Jay Samuels and I are taking the lead on a couple of the reports, but even with regard to those, I am uncertain what will be recommended to the panel. Moreover, the panel as a whole might reject some of our conclusions if the majority feeling is that they are not sufficiently supported by research or that our claims are not equivalent to those put forth by the other subgroups, in terms of amount or quality of research evidence. On October 13 and 14, the panel will meet to share preliminary versions of the subpanel reviews, and we will start hammering out a consensus of final findings. Those meetings, too, are open to the public, and summaries and transcripts of them will be available soon after they are held.
Probably the most uncertain aspect of the panel's work has to do with what the conclusions will mean. That is, how will the federal government, given the official nature of this review, actually use these findings? Medical panels' results are used to set healthcare policy, but how will an education panel's results be used? Although we are required to present a report to the secretary of health and human services, the secretary of education, and the appropriate congressional committees, what will be done with that report is uncertain. In 1998, soon after the NRP's formation was recommended, Congress passed the Reading Excellence Act to provide funds to poor school districts for professional development focused on effective instructional procedures in reading. It turns out the policymakers were depending on completion of the NRP's analysis by summer 1999, and intended that our findings would determine what those procedures were and how the new funds could be used. Since our work has not been completed, those dollars have been expended according to a different plan.
At this time, no law requires the use of the panel's findings in any way, and no new proposals for using the information in our final report have emerged publicly. What this means is that the NRP findings -- no matter how sound or useful -- may have no legal force at all. However, it is also possible that down the road, policymakers will treat our report as they do those of medical panels, and we could see disbursements of public funds linked to our findings. Although there is no direct requirement for the use of our determinations, I (and I believe the other panelists) are proceeding as if our findings will be used to set public policy. We are therefore striving to be as careful and accurate as possible.
The idea that any panel could, on the basis of research, come up with the definitive and final answer about how all children should be taught to read under all circumstances is patently absurd. There is no possibility of such a result, nor is such a result expected. Even the medical panels, with their supposedly more straightforward biological concerns, do not strive for permanent solutions through their research reviews; instead, they seek the best answer that can currently be determined on the basis of extant data. If they have sufficient evidence, they decide upon what the current standard of practice should be. Other researchers can overturn such determinations -- there are no final answers -- but they must do so by producing contradictory research evidence.
Given the tentative quality of even the best research synthesis, is it reasonable to provide more support to programs that have been proven to be effective? Of course, how you answer that question will likely tell much about your views concerning the National Reading Panel. If you believe that all programs and materials (both the ones you like and those you do not) are deserving of public support as long as some teacher wants to use them, then you probably find the NRP's work to be obnoxious. If, on the other hand, you accept the notion that not all programs are equally deserving of support -- that not everything works equally well -- and that research evidence should play some role in ultimate determinations of what does work, then the idea of the National Reading Panel is probably acceptable.
Both personally and professionally, I accept the idea that schools should try to use procedures that have proven successful in improving children's reading ability. And I believe it is essential that judgments about what those procedures are be made only after careful, thorough, rigorous, and public analysis of bodies of research rather than of single studies. There is nothing wrong with a teacher, a school, a school district, a state, or even the federal government targeting particular effort on what is known to work. However, even with that, I think that it would be a mistake to expend all, or even most, of the funds available for education in this targeted manner. Certainly, there are instructional procedures that work but that have not been studied, and teachers need room to experiment with their own ideas, too. Certainly, even research proven methods have not been studied under all conditions or circumstances.
Some of my colleagues think that the National Reading Panel is a kind of slippery slope -- they fear that it creates a dangerous precedent that will be hard to live with. The NRP's reliance on quantitative, experimental results, they sometimes argue, makes it possible to conclude that only certain types of research evidence have value and, therefore, the federal government might decide to fund only work based on such research paradigms. I think this argument is farfetched. Currently, the U.S. Department of Education's Office of Education Research and Improvement, one of the major funders of reading research, targets particular issues and types of questions to be answered. Right now, for instance, almost all federal research dollars available for studies of reading are devoted to beginning reading (e.g., the Center for the Improvement of Early Reading Achievement), but at one time they all went to comprehension. This has transpired almost without remark from leaders in the reading field, probably because they recognize that researchers are not prohibited from studying whatever they choose -- only that they are not guaranteed any federal money to support their work. Currently, those who study reading among adolescents are unlikely to obtain federal support, but they proceed without it. Most studies of reading do not receive federal support anyway.
We should not be threatened by the notion that certain research methods are better for answering certain questions. After all, most research methods texts recommend that researchers fit their inquiry approaches to the problems being studied. The NRP's approach just carries that recommendation to its logical conclusion, by considering evidence that arises from studies specifically designed to answer particular types of questions. It is possible that policymakers could decide that there is only one kind of research question that they want answered, and they could then concentrate all available federal dollars on such research. Such a decision seems highly unlikely -- and would be an even more remote possibility if our professional organizations would become active in working with policymakers to set research agendas. In any event, the government has never explicitly set such methodological limits in the past, even in fields such as medicine, where research findings have long been used to establish standards of practice.
We need to be able to answer the kinds of questions that the public raises about reading education, and we must be prepared to put the answers we find into practice. Arthur N. Applebee (1999) remarked upon the importance of providing appropriate research-based answers to policymakers when he received the National Council of Teachers of English David H. Russell Award for Distinguished Research in the Teaching of English:
Unless we are willing to conduct research that provides valid answers to public questions and to have our practical actions governed, at least in part, by the findings of such research, we are almost certain to sacrifice future public support for our efforts. When I have spoken at conferences about the work of the National Reading Panel, I have been surprised by the number of teachers and administrators who have gone out of their way to thank us for our efforts. Their gratitude seems to come from their belief that the NRP will help resolve the confusion they feel about the contradictory claims of reading research and the anger they have toward those who make biased research claims to push particular agendas, philosophies, or even commercial programs. The National Reading Panel, through its rigorous and public procedures of research synthesis, is likely to provide the field with it clearest mandate for applying research to practice, for showing the way to use research to make our schools better.
Adams, M. (1990). Beginning to read: Thinking and learning about print. Cambridge, MA: MIT Press.
Anderson, R.C., Hiebert, E.H., Scott, J.A., & Wilkinson, I.A.G. (1985). Becoming a nation of readers: The report of the Commission on Reading. Washington, DC: National Academy of Education, Commission on Education and Public Policy.
Applebee, A.N. (1999). Building a foundation for effective teaching and learning of English: A personal perspective on thirty years of research. Research in the Teaching of English, 33, 352-366.
Chall, J.S. (1967). Learning to read: The great debate. New York: McGraw-Hill.
Grossen, B. (n.d.). 30 years of research: What we now know about how children learn to read. Santa Cruz, CA: Center for the Future of Teaching and Learning.
Manzo, K.K. (1998, February 18). New National Reading Panel faulted before it's formed. Education Week, 27(23), 18. Available at http://www.edweek.com/ew/1998/23nichd.h17.
Pearson, P.D. (1999). Essay book reviews: A historically based review of Preventing Reading Difficulties in Young Children. Reading Research Quarterly, 34, 231-246.
Snow, C., Burns, M., & Griffin, P. (1998). Preventing reading difficulties in young children. Washington, DC: National Academy Press. Available: http://books.nap.edu/catalog/6023.html
Taylor, D. (1998). Beginning to read and the spin doctors of science. Urbana, IL: National Council of Teachers of English.
Timothy Shanahan is professor of urban education and director of the Center for Literacy at the University of Illinois at Chicago. He can be reached at 1040 W. Harrison (M/C 147), Chicago, IL 60607, USA, or by e-mail to firstname.lastname@example.org. His research focuses on reading-writing relations, literacy assessment, and school improvement.
Back to top
Readers who enjoy this posting may also be interested in the following:
Back to top
Reading Online, www.readingonline.org
Posted August 1999; links updated June 2000
© 1999-2000 International Reading Association, Inc. ISSN 1096-1232