This column is reprinted from the Technology department of the Journal of Adolescent & Adult Literacy (JAAL). It contains the following sections:

• Author's Message
• Issue of the Month
• Interpretations
• Data View
• Literacy Web Page of the Month
• Glossary
• References

How can we be sure that all the new computers and networks appearing in classrooms around the world will really make a difference for learners? Do we know for certain that the money and time invested in them makes a difference? What criteria should we use and how can we measure success? These questions raise additional questions about differences in the way we use technologies. For every success story, there are other stories about problems or unanticipated negative effects. Why do we so often discover that new technologies remain underused or misused?

Journal articles or newspaper accounts sometimes describe in great detail one classroom in which marvelous learning occurred through the use of some new system, but they usually fail to mention the teachers who merely rewrote their current methods in a new medium without any substantive effect on students' learning. Such reports rarely talk about the larger number who knew about the system but failed to use it, and they never discuss the many who were not interested enough to learn anything about it at all.

Few of us are professional evaluators, but we are all affected by the public arguments made for and against different technologies and their uses. These arguments rely upon more or less formal methods of evaluating what happens when these technologies are used. We need to understand what can be concluded about technology use and what the limits of current methods might be. We are also often called on to interpret specific results, such as that the use of technology within a particular literacy program led to improved learning. We need to understand how to think of such results in larger contexts and to interpret them for ourselves and others.

In this month's column I will consider various aspects of issues related to evaluation, access, and literacy learning.

Back to top
Back to listing of all JAAL Technology columns available in Reading Online

The potential of new information and communication technologies for teaching and learning was recognized early. Programmed instruction as well as the programming language Logo were developed over 30 years ago, and the earliest applications of time-sharing operating systems included teaching. Almost immediately, people began to ask whether the new tools would lead to better, more effective, or more efficient education.

Evaluating the New Information Technologies (Johnston, 1984) made clear why this quite reasonable question would not be so easy to answer. More recent books (Baker & O'Neil, 1994; Roblyer, Castine, & King, 1988) have shown both positive and negative effects, as well as the difficulty of framing that question in a straightforward way. The debate continues in reports available on the Web, such as "Technology Counts '98," published last fall as a special report by Education Week.

In this month's Issue section, I will explore some reasons why information and communication technologies are difficult to evaluate and why these difficulties do not deter us from using the technologies in spite of the lack of solid evaluations.

Ten Reasons Why Information Technologies
Require New Evaluation Approaches

Differences among adopters. Designers of new educational programs know that early adopters of new technology have different experiences than later adopters. The early ones tend to be more adventuresome, more knowledgeable about the area, and often receive more support than those who adopt later. This phenomenon achieves its maximum effect with new technologies. The first users are typically computer science experts relative to those who adopt later, and their patterns of use are strikingly different. This means that evaluations have to be understood with respect to the community of users and cannot be assumed to refer to the technology per se.

Scale effects. Often, when an older technology has been employed in an educational setting its effects can be viewed independently of its use in other settings. The use of a chalkboard in one class is affected only indirectly by its use in another class. But information technologies are often also communication technologies, which connect across settings and produce network effects. Their operation depends directly upon what happens elsewhere. For example, an e-mail discussion group operates very differently with 10 members versus 1,000. Thus, how an innovation scales up becomes critical. Scalability is a factor for any educational innovation, but it can rarely be ignored when we look at information technologies.

Geography. As use scales up, the new media reshape geography through spatially dispersed contexts of use and variations in implementation. A school program involving collaborative data acquisition and analysis may in its very definition be an international program. Looking at other classrooms involved in the program is not just a matter of increasing sample size, but one of observing a single but geographically dispersed context of use.

Media as systems. New information and communication technologies must be understood, not merely as discrete tools but as components of complex systems. The computer on my desk is in some ways akin to a copy machine or a typewriter, but when we consider the variety of software that can alter its mode of operation or transform it into an Internet machine, we see it is but one cog in a gigantic system. Thus, it is not trivial to identify what is being evaluated when we evaluate its use.

Rapid change. Another issue is the newness and changeability of the technology. I have found in a variety of projects to develop technology-based learning that we were tinkering with the software throughout in a way we did not do with text materials. Once something was completed it was usually obsolete because of changes in the companion technologies. This meant we were effectively engaged in perpetual formative evaluation (Bruce & Rubin, 1993). There are few educational programs involving information technologies that remain unchanged for long, often not even until the evaluation report appears.

Trail of use. Despite the ability of information technologies to store massive amounts of data, crucial user information is often unavailable. Many programs do not keep, or they lose, important parts of student work. Electronic files become unreadable with changes in the technology. Students delete files whose corresponding paper version might have been saved. The impermanence of technology use is a challenge for both teachers and evaluators.

Re-creation of the technology. Yet another issue is appropriation of the technology within social practices. Despite the apparent fixedness of technology, it can actually be more malleable in use than a paper-based curriculum. That has led some researchers to turn to situated evaluation approaches.

New roles for teachers and students. New modes of teaching and learning come packaged with new technologies. In fact, many new tools are promoted precisely because they change the role of teacher from disseminator of information to a facilitator in the construction of meaning. They also raise fundamental issues about whether they are being used as tools to help students learn other things, or as phenomena to be understood on their own. These and other issues about roles, curricula, and purposes for learning add to the complexity of evaluating new technologies.

Technical characteristics. There is also a difficult issue of technical characteristics of the technology: features of an interface, the quality of a simulation, or the design of a hypertext. Virtually all programs in education involve materials or technologies of some sort, but new information technologies raise the level of complexity a notch or two and require new evaluation methods.

Access. Finally, one of the most important issues is access to technology. New information technologies entail significant investments not only in equipment, but also in training, support, and changes in traditional practices. Moreover, there is much evidence that technologies reify social stratification, as in the English language dominance of the World Wide Web. Rather than reducing inequities in education, as most of us hope will happen, these technologies may create and solidify inequities as never before. The attribution of goodness to some new approach must therefore be tempered with careful consideration of its accessibility.

The Case for Situated Studies

In a 1949 book, Knowing and the Known, Dewey and Bentley articulated the idea of transaction, which was later applied by Louise Rosenblatt to develop a theory of the reader's response to literature. Transactions provide one way to think about this problem. Rather than conceptualizing the technology as a discrete object that acts on people, they would want to understand the way the technology participates in an organic relationship with living social practices. Transaction moves us away from questions such as "What are the effects of the technology?" toward questions such as "What processes are occurring in the social system in which this technology participates?"

Examination of the unique uses that technology fosters leads the evaluator to more situated studies. One result of such studies of how technologies affect literacy education has been to show how realizations of technologies vary tremendously depending upon the teacher's goals, students' previous experiences with computers, the available support, and the organization's policies with respect to assessment and curriculum. One teacher may use a word processor to create practice lessons on punctuation, while another may develop a yearlong theme study that relies on extensive student writing and revision for publication. These great differences suggest that the teacher's creative role is vital to the successful use of new technologies. It is much more important to understand how people use technologies than simply to measure their effectiveness across broad averages of use.

Another result of situated studies of new technologies has been to show clearly how technologies rarely produce simple, one-step changes. Instead, changes occur over long periods, as teachers and students develop enlarged understandings of what the technologies can do. Both need time to integrate new tools into existing teaching and learning practices.

This research has also shown that the richness of the new technologies -- the access to vast resources on the World Wide Web, the powerful new media, the interactivity -- can sometimes lead to a focus on content or methods in teaching, with less attention to individual learners, thus manifesting Dewey's map in place of the territory. Simply using computers or connecting to the network does not ensure that teaching is easier and more effective or that students will be automatically well prepared to live in the 21st century. Instead, making good use of new technologies increases the demands on teachers, at least initially. Educators face major challenges to use these technologies effectively to expand the possibilities for learning.

Making Decisions in the Face of Uncertainty

These considerations about the complexity of technology may reassure the evaluator, but they don't help the literacy educator trying to make decisions about whether and how to use new technologies. How do we decide what to do? Is technological literacy an essential part of literacy now? Have new technologies become essential tools for learning? How can we think about the apparent conflicts between the classroom and the workplace, between learning for today's needs versus tomorrow's, between using the technologies of today and those likely to appear in the future?

A situated perspective does not tell us whether learning to program in Basic was a good use of student time in the 1980s, but it does suggest that the conventional view of schooling as merely preparation for something else is inadequate. It leads us to think of technologies as integral parts of what we do as literate beings, rather than as isolated tools that are employed to fix problems or as magical devices that can replace good teaching. A deeper understanding of these issues could lead us to rethink how we use technologies to promote literacy and to more informed, if never easy, decision making.

Back to top
Back to listing of all JAAL Technology columns available in Reading Online

Steve Ehrmann pointed out in 1997 that there are always two ways to look at any educational innovation. One is to consider its uniform impact -- that is, to assume that the educator is trying to shape learning in the same way for all learners. The second is to examine the unique uses that emerge when we assume that all learners actively interpret and make use of the resource in their own way.

These two perspectives, both important for almost any evaluation, are particularly important when technology is in use. Most instructional uses of information technology are meant to be empowering -- i.e., to create fresh choices for instructors and learners. When students communicate more, when they work on projects, when they collaborate, the diversity of potential outcomes for learners increases. Any evaluation which uses only the uniform impact perspective will miss some of the most important consequences of this type of innovation.

(From S.C. Ehrmann, How (not) to evaluate a grant-funded technology project. Paper presented at the National Endowment for the Humanities, Washington, DC, April 29, 1997. Available on the Web at classWeb.gatech.edu/fdw/assessment/hownot.htm.)

Back to top
Back to listing of all JAAL Technology columns available in Reading Online

None of the questions about effectiveness of technologies makes sense if those technologies are unavailable. The great discrepancy in personal income within and between countries means that people effectively live in very different technological worlds. Some of this can be seen in the 1998 edition of the Human Development Report from the United Nations:

Inequalities in consumption are stark. Globally, the 20% of the world's people in the highest-income countries account for 86% of total private consumption expenditures; the poorest 20% a minuscule 1.3%. More specifically, the richest fifth:

• consume 45% of all meat and fish (the poorest fifth, 5%)
• consume 58% of total energy (the poorest fifth, less than 4%)
• have 74% of all telephone lines (the poorest fifth, 1.5%)
• consume 84% of all paper (the poorest fifth, 1.1%)
• own 87% of the world's vehicle fleet (the poorest fifth, less than 1%)

Among other things, it is evident that for the more than a billion people worldwide who have no telephone, Internet access is unattainable. Less glaring discrepancies operate even within affluent countries to mean that students in different communities have divergent opportunities to learn through or about the new technologies.

Back to top
Back to listing of all JAAL Technology columns available in Reading Online

Bobby was created at CAST (Center for Applied Special Technology), a not-for-profit organization whose mission is to expand opportunities for people with disabilities through innovative uses of computer technology. It is a Web-based public service that analyzes Web pages for their accessibility to people with disabilities and for their compatibility with various browsers. Bobby follows the guidelines of the W3C Webpage Accessibility Initiative at www.w3.org/WAI/GL/. A Web site that passes the Bobby test may then display the approved icon. Bobby has links to thousands of Web sites, and over 3 million Web pages are tested for addition each month. The Bobby program follows the CAST principles that "universal design for learning" should provide multiple representations of content, multiple options for expression and control, and multiple options for engagement and motivation. It checks to make sure, for example, that the Web site

• provides alternative text for all images,
• provides transcripts for all audio files,
• provides a description and a caption for video information,
• does not use color as the sole means of conveying important information,
• nests headings properly,
• provides a mechanism to allow users to freeze movement or update, and
• uses the LANG attribute to identify the language of the text.

Adhering to these guidelines makes it more likely that the Web site will be useable by people with visual or hearing impairments; by those with older or less expensive equipment; by those who speak other languages; and, as the term "universal design" implies, by anyone.

Back to top
Back to listing of all JAAL Technology columns available in Reading Online

Alternative text for images: the inclusion of text that describes what is in an image so that a user who cannot see clearly can at least read about the image, often using a large font or a speech generator.
Back to article

Formative evaluation: evaluation applied to a program under development in order to find ways to improve it; widely employed for technology-based programs because of the rapid technological changes.
Back to article

Frames: an advanced technique for Web-page design that allows different information to be displayed and manipulated in different portions of the screen; while useful, it makes Web pages inaccessible to those whose browsers cannot process the frames.

Network effects: the phenomenon that the effects of a particular technology may depend on how it is interconnected with other devices, and how those effects depend on the structure, size, and operating characteristics of the network.
Back to article

Situated evaluation: an approach to evaluation of technology use that assumes the technology is not set a priori, but comes into being through use.
Back to article

Situated studies: a research method that explicitly incorporates an analysis of the context in which literacy is practiced or learning occurs.
Back to article

Summative evaluation: evaluation applied to judge the overall effect of a developed program; most suitable for the uniform impacts perspective on a learning program.

Technocentrism: a way of thinking about the use of technology that attributes all important changes to the technology itself.

Transaction: a phenomenon in which mind and reality, or a "knowing" and the "known," are conceived as a unified entity.
Back to article

Web-page accessibility: the degree to which the content of a Web page is available to people in different groups, such as those who speak a language other than the author or who have a low-speed network connection.
Back to article

Back to top
Back to listing of all JAAL Technology columns available in Reading Online

Baker, E.L., & O'Neil, H.F. (1994). Technology assessment in education and training. Hillsdale, NJ: Erlbaum.
Back to article

Bruce, B.C., & Rubin, A.D. (1993). Electronic quills: A situated evaluation of using computers for writing in classrooms. Hillsdale, NJ: Erlbaum.
Back to article

Dewey, J., & Bentley, A.F. (1949). Knowing and the known. Boston: Beacon.
Back to article

Johnston, J. (Ed.). (1984). Evaluating the new information technologies. Washington, DC: Jossey-Bass.
Back to article

Roblyer, M.D., Castine, W.H., & King, F.J. (1988). Assessing the impact of computer-based instruction: A review of the literature. New York: Haworth.
Back to article

Back to top
Back to listing of all JAAL Technology columns available in Reading Online