Reading Online: Articles -- The Role of Qualitative and Ethnographic Research

In spring 1998, the director of the National Institute of Child Health and Human Development appointed the National Reading Panel (NRP) to carry out a request by the United States Congress to study the effectiveness of various approaches to teaching children how to read. The panel subsequently defined the procedures by which it would conduct its business and the questions that it would use to guide its study. The panel has now published its results, available at www.nationalreadingpanel.org/Documents/default.htm.

The National Reading Conference (NRC), a professional organization dedicated to literacy research, created a committee to respond to the NRP report by addressing the following questions:

Did the panel do what it said it would do to answer the questions it had posed?
What, if anything, did the panel leave out within the questions it created and in defining the questions themselves?
What research traditions, if any, have not been considered in the panel's work?

Following is the first in a planned series of commentaries prepared by members of the NRC committee. It presents one response to the third question.

Martha Rapp Ruddell
Chair, National Reading Conference Committee to Respond to the National Reading Panel
Past President, National Reading Conference

The Role of Qualitative and Ethnographic Research in Educational Policy

Victoria Purcell-Gates

The National Reading Panel, charged with identifying research that could inform policy and practice in reading, has now concluded its exhaustive review. The panel focused its review on experimental and quasiexperimental research which, by necessity, meant that it focused on a small set of questions. I say “by necessity” because the areas of instruction in reading that have been examined by experimental or quasiexperimental research are of themselves few and far between, although certainly of central importance to the learning-to-read process.

The point I wish to argue here is that by limiting the type of research used to address the issue of how to guide policy and practice in reading instruction to experimental and quasiexperimental studies, the panel missed critical areas that have been examined by qualitative and ethnographic research. Further, a clear danger resulting from this limitation to particular types of methodologies is the establishment in the minds of funders and users of research that only experimental and quasiexperimental research provide us with answers to educational problems. A second theme of this commentary, then, is my view that we do not stand a chance of solving problems of learning and teaching if we confine ourselves solely to these methodologies. Rather, it is only by the considered and judicious use of a range of methodologies, rigorously applied and intelligently synthesized, that we as educators, policy makers, and the informed and concerned public can hope to make real advances and reforms to our system and in our procedures for educating all our students to their full potentials.

Proving, Informing, or Both?

There is no doubt that experimental, and to some degree quasiexperimental, research is required to “prove” the effectiveness of an instructional approach, method, or intervention. This to some degree explains the panel's exclusion of other methodologies in its research review (see, e.g., Shanahan, 1999; online document). Identifying the effects of specified instructional techniques can only be accomplished with carefully designed studies that employ random sampling, random assignment to condition, control groups, large Ns, and analysis techniques that allow for the accounting of the myriad confounding variables that are inevitable in the study of socially situated activities such as teaching and learning. Generalization from such studies is bounded and constrained by descriptions of subjects, teachers, teacher training in the technique under study, fidelity of intervention implementation, and evidence of sustainability and scalability.

Clearly, this ideal, requisite though it may be, is difficult to attain, expensive to accomplish, and limited to answering only those questions that can actually be examined with such methodologies (Boruch, 1999, & Gueron, 1999, single online document). Gueron, drawing on her experience with research on training programs and welfare reform, lists eight criteria useful for judging when random assignment is the right tool:

When the key question is “Does the program or reform make a difference?” (questions of program impact)
When the program under study is sufficiently different from business as usual and you can maintain that distinction over time
When the research does not denying people access to an entitlement
When the researchers are addressing an important or unanswered question
When adequate procedures are in place to inform program participants about the study and to ensure confidentiality of data
When there is no easier way to get a good answer
When you can get cooperation
When you have the resources and ability to do a quality study

While experimental and quasiexperimental studies are the gold standard for examining program impact, there are many critical issues facing education and educators that raise other types of questions. For example, most reasonable people would probably agree that a central concern in the field of education is that of differential achievement rates. Throughout the history of public education, children and adults from nonmainstream groups or for low socioeconomic status (SES) have as a whole failed to achieve academic success to the same levels and at the same rate as learners from mainstream, sociopolitically empowered groups (Kaestle, Damon-Moore, Stedman, Tinsley, & Trollinger, 1991). Descriptive and correlational studies (both types of qualitative research) repeatedly have confirmed this disturbing pattern. Other studies using qualitative and quantitative research methods have examined possible causes as subcorrelates -- language differences, cognitive differences, differential access to books, inequalities in schooling, and so on. Clearly, without the ability, or right, to control probable confounding variables or randomly assign to condition (of socioeconomic status?), we can never establish true causation. However, we can gain real insight into the issue by employing nonexperimental methods. Further, with judicious and careful synthesis, we can come to understand the landscape of nonmainstream status and educational achievement to such a degree that we may be able to design experimental studies that examine the impact of educational programs on this SES-related achievement gap.

Many of our pressing and unanswered educational questions are of this type -- deeply embedded in issues of race, culture, class, gender, and family income. For most of these questions, we do not possess sufficient information or insights, gained through systematic, rigorous, nonexperimental research, to design reasonable experimental interventions and research studies that meet the criteria set forth by Gueron (1999), listed above.

LeCompte and Schensul (1999, p. 2) describe ethnography, one type of nonexperimental research that uses both qualitative and quantitative data, as a research method “designed for discovery.” Ethnography, they point out, is particularly suited to gaining insight into questions embedded in social and cultural communities and practices such as education. It is scientific, investigative, uses the researcher as the primary tool of data collection, employs rigorous research methods and data collection techniques to avoid bias and ensure accuracy of data, emphasizes and builds on the perspectives of the people in the research setting, and is inductive, building local theories for testing and adapting for use both locally and elsewhere.

The rigor represented in codified ethnographic research methods produces scientifically valid and reliable data. The same is true for other nonexperimental research designs using qualitative and quantitative methods -- descriptive and correlational methodologies, and different combinations of these. Each type of research has its own criteria for rigor, and each answers different types of questions. Thus, all types of research, from experimental to ethnographic, provide us with important lenses through which to study and examine critical educational questions and issues. Because they are different, each type of research provides us with unique answers and insights, and each type of research presents its own limitations affecting generalizability, types of questions it can address, and the nature of the insights it can offer.

Nonexperimental Research Methods That Inform Policy and Practice

Following is a brief listing of nonexperimental research methodologies that use qualitative and quantitative methods, along with brief descriptions of the types of questions they answer and the considerations researchers and the public should keep in mind when interpreting and applying their results. The purpose of this listing is to demonstrate the “value added” nature of these methodologies in relation to their potential contribution to educational research that can inform policy and practice. Note that this is intended only as a quick description of nonexperimental methodologies; within each category there are many different approaches and standards in relation to researcher-researched relationships, stances, and procedures.

The table below relies heavily on two main sources: Understanding Research in Reading and Writing (Kamil, Langer, & Shanahan, 1985) and Designing and Conducting Ethnographic Research (LeCompte & Schensul, 1999).

Type of Research	Purposes/Uses of Research	Issues for Interpretation
Correlational	To determine relations among variables that are hard or impossible to manipulate To determine the reliability of testing instruments To determine the interrelationships among a number of dependent variables in a given situation	Correlations cannot reveal definitive causation. Effect size is more important than simple correlations. High intercorrelations among multiple measures indicate they are measuring the same thing.
Descriptive	To describe characteristics, properties, or relationships of groups, events, or phenomena without manipulation of the phenomena under study Especially useful to increase awareness of particular problems or to provide baseline data for future studies	Results cannot establish causation. Instruments (surveys, interviews, observations, tests, etc.) must be appropriate for the questions asked. Instruments must have established validity and reliability. Because these studies begin with categories of interest already established, they cannot discover factors of potential importance and interest.
Ethnographic	To define a problem when the problem is not clear To define a problem that is complex and embedded in multiple systems or sectors To identify participants when the participants, sectors, or stakeholders are not yet known or identified To clarify the range of settings where the problem or situation occurs at times when the settings are not fully identified, known, or understood To explore the factors associated with the problem in order to understand it	Research methods must be chosen to provide appropriate information about the problem being studied. Data must be systematically collected and recursively analyzed. Conclusions are applicable primarily to the setting under investigation; application to other settings relies on the degree to which the contexts are similar along all relevant dimensions. Relationships uncovered cannot be interpreted as causative. Conclusions can be used primarily to provide insights, to contribute to theory, and to suggest factors to consider for other types of studies.

The Need for More Than Experiments

Many educational issues and problems clearly require research that draws on multiple perspectives, approaches, and procedures. We simply do not know enough about the complexities of the complex process of learning in schools to design effective experimental studies that will solve all of our problems. We need methodologies that will allow us to probe for insights, for possible operative factors, for new information, before we can begin to think about limiting our research studies to experimental, hypothesis-testing approaches.

On December 2, 1999, a discussion group appointed by the National Reading Conference formulated a sample of questions that have been addressed by nonexperimental research. The group consisted of Yetta Goodman, Judith Green, Jerome Harste, Steven Stahl, Jeffrey Wood, and myself. We offer this sample by way of illustrating the questions, which we consider key to understanding how children learn to read and write in schools, that have been effectively investigated by research using other than experimental and quasiexperimental methodologies.

What resources in terms of experiences, language, cognition, and social and cultural expectations brought to school by learners do they then rely on to interpret and learn from instruction? How do these resources influence how learners “take” from instruction?
What opportunities exist for learning, at both classroom and school levels, for different children in different school settings?
What are the relationships between writing development and reading development?
How do different instructional practices affect instructional outcomes differentially?
What knowledge do teachers draw upon or use to inform their practice?
What are some relationships between classroom talk and learning to read?
What different demands do learners face when reading in different disciplines, such as science, math, or social studies?
How do social interactions among students influence their learning to read?
How do different instructional contexts -- such as teacher-directed group reading, reading to answer questions, or reading on one's own -- influence the strategies readers use to process print for meaning?
What are some relationships between individual differences and instructional outcomes? Do these change in different instructional contexts?

Insights gained from research into these questions are particularly informative, not only to guide development of future experimental studies, but also to present-day considerations of instructional policy. The answers we have gleaned provide the texture for teachers who are considering what research says to them about how to teach each individual child in the classroom in ways that will result in personal growth and achievement for that child. Research into questions such as these helps teachers understand, or at least begin to problem-solve, the inevitable failures with specific children of strategies documented as successful overall in large, generalizable experimental studies.

A Word to Policy Makers and Funding Agencies

This commentary is intended to urge policy makers and funders of research to acknowledge the contributions that nonexperimental research can make to decisions regarding instructional policy as well as to building our knowledge base of learning and teaching. I urge you not to restrict educational research to one or two paradigms. To do so would unacceptably restrict our ability to explore critical issues, insights about which are needed before we can begin to address the complex challenges of educating all learners in this complex, multicultural society of ours to their full potentials. I urge you to acknowledge fully and publicly the need for different lenses to address these challenges. At the same time, I urge you to insist on high research standards within each research paradigm, standards for rigor that will guarantee results that can be trusted and relied upon as they are used, according to their unique purposes, by policy makers, funders of research, school administrators, and individual teachers.

References

Boruch, R. (1999). Can we make education policy on the basis of evidence? What constitutes high quality education research and how can it be incorporated into policymaking? (transcript of a Brookings Press Forum, Washington, DC, December 8, 1999). Washington, DC: Brookings Institution. Available: http://www.brook.edu/comm/transcripts/19991208.htm
Back

Gueron, J. (1999). Can we make education policy on the basis of evidence? What constitutes high quality education research and how can it be incorporated into policymaking? (transcript of a Brookings Press Forum, Washington, DC, December 8, 1999). Washington, DC: Brookings Institution. Available: http://www.brook.edu/comm/transcripts/19991208.htm
Back (1st citation)
Back (2nd citation)

Kaestle, C.F., Damon-Moore, L.C., Stedman, K., Tinsley, K., & Trollinger, W.V., Jr. (1991). Literacy in the United States. New Haven, CT: Yale University Press.
Back

Kamil, M.L., Langer, J.A., & Shanahan, T. (1985). Understanding research in reading and writing. Boston, MA: Allyn & Bacon.
Back

LeCompte, M.D., & Schensul, J.J. (1999). Designing and conducting ethnographic research. Walnut Creek, CA: AltaMira.
Back (1st citation)
Back (2nd citation)

Shanahan, T. (1999, August). The National Reading Panel: Using research to create more literate students. Reading Online. Available: http://www.readingonline.org/critical/shanahan/panel.html
Back

About the Author

Victoria Purcell-Gates is a professor of literacy at Michigan State University (304 Erickson Hall, Department of Teacher Education, East Lansing, MI 48824, USA; e-mail vpgates@msu.edu). Her research spans the methodological continuum from ethnography to experimental, and she has published widely in a variety of academic journals. Her book Other People's Words (Harvard University Press, 1995), based on an ethnographic study of a nonliterate family, won the 1996 University of Louisville Grawemeyer Award for Education for outstanding contribution to education and to “making the world a better place.” Her most recent book, Now We Read, We See, We Speak (Lawrence Erlbaum Associates, 2000), co-written with Robin Waterman, is an ethnography of a Freirean-based women's literacy class in El Salvador. She and coresearcher Nell K. Duke are currently working on an experimental study of explicit teaching of science genre features in second and third grades, funded by the National Science Foundation and the Interagency Education Research Initiative.

Back

Citation: Purcell-Gates, V. (2000, July). The role of qualitative and ethnographic research in educational policy. Reading Online, 4(1). Available: http://www.readingonline.org/articles/purcell-gates/