Transmediation as a Metaphor for New Literacies in Multimedia Classrooms

Ladislaus M. Semali
Judith Fueyo

Teachers in public schools give more time to the development of reading and writing than to facilitating children’s understanding of visual, aural, aesthetic, artistic, digital, or multimedia representations of meanings. The former are the currency of the kingdom, as evidenced by the value placed on standards in the United States and many other countries. Language dominates the way of knowing in mediating sudents’ experiences with the world (Fueyo, 1991). However, because students are bombarded by multimedia, we teachers of English language arts need to find methods and analytical frameworks that help students make sense of and critique the codes and coded messages of our times.

English language arts programs give cursory attention to developing students’ ability to see and hear effectively the multisensory world beyond the printed text (Ferrington, 1994). Those programs that do extend sensory training focus primarily on the visual, as has been noted in the literature of the visual and media literacy movements. However, outside of the schoolhouse the new communication media technologies -- such as fiber optics, holography, interactive video, digital or computerized television, high-definition television, three-dimensional media, videocassette recorders, videodiscs, teletexts, videotexts, teleconferencing, large television screens, video games, video magazines, satellites, and cable television -- stretch the consumer’s visual and auditory abilities beyond their limits (Metallions, 1994).

This article takes the position that educators need to realize that students live in an increasingly multisensory world. Students today are exposed to motion pictures, television, multimedia graphics, advertising, radio, and music in complex natural and human-made environments. The promise of new information technologies lures students to use multiple sign systems. We realize that the invention or production of knowledge outside the walls of the classroom is made possible today through technologies such as the Internet, hypermedia, computer software, and so forth. To put it simply, the authority of teachers and the certainty of what was learned in the classroom and regarded as stable have begun to slip and slide. With the introduction of computers, the Internet, and a plethora of ways to access academic resources that come with these technologies, knowledge is being democratized.

The existence of multimedia calls for multiple forms of literacy, forms that can represent the world of ideas, emotions, and events with multiple symbols.

Back to menu

Multimedia, Hypermedia, and Semiotic Representation

To provide a theoretical context of how multiple symbol systems interact to produce meaning, we borrow from semiotics. At its most basic, semiotics is the study of sign systems: what signs mean, how they relate to one another, and how they are manipulated. As Morris (1971) shows, semiotics has the basic terminology or language that allows us to talk of all signs, whether they use language or nonverbal symbols and whether they are drawn from auditory, visual, tactile, or nonsensory stimuli. The underlying assumption of the study of new literacies is that signs are the basic building blocks of human communication, which takes place in many forms.

Semioticians suggest that a multimedia experience intentionally juxtaposes two or more texts, even if they reflect the same sign system. In the activity of juxtaposing, a new third text is generated that reveals the contrasting belief systems that might have remained hidden if a text were examined alone without connection to all the other signs of the world. In the context of the Internet, for example, the multiple images and texts made possible with hypermedia tools extend representation beyond what has ever been possible with print. Examples of hypermedia might involve students taking and manipulating digital photographs to create collages of meanings and documentaries of experience. Or they could use hypermedia computer tools to combine images, music, video, and word to generate a critique of the beliefs that frame symbolic meaning. Likewise, students could draw pictures, beat drums, and enact living situations as an interpretative response to texts in all media (Semali, in press). These new literacy practices bring to center stage transmedial ways of learning and of producing knowledge and the ways in which such practices become implicated in the semiotic representation of meaning.

Back to menu

Taking the Critical Turn

Semiotic representation can become a way of talking, thinking, and interpreting the sign systems that stand for or represent meaning embedded in texts about race, class, gender, disability, sexual orientation, and other identities. Students and teachers who undertake such a semiotics-based critical approach examine sign systems as vehicles of meaning in a culture. They then invite an interrogation of those meanings via layer on layer of sign systems. Eisner (1997) reminds teachers to reflect on the role that forms of representation -- literal language, visual images, number, and poetry -- play in the creation of what we think.

Here, our central concerns are two:

• What is the relationship between what students know and the signs they encounter in their classrooms (about race, class, gender, disability, and sexual orientation)?
• What meaning do they make of these semiotic systems in their literacy practices?

In this context, semiotic representation is a reflection of the new literacies.

Back to menu

How the New Literacies Reflect Transmedial Experiences

Transmediation in an English language arts classroom means “taking understandings from one system and moving them into another sign system” (Siegel, 1995). When we talk about teachers and students engaging in transmedial experiences, we are referring to their abilities to engage in multiple ways of mediating knowing between sign systems. Such critical engagement does not involve only passive reading and writing; rather, it involves an active textual analysis through a multiplicity of perspectives and critical methods. Suhor (1984) characterized this process of translating or representing content as transmediation -- mediating or representing meaning across sign systems. He perceived transmediation as a syntactic concept since it deals with the structure of sign systems and the relationships between them.

In our own practice, for example, a transmedial experience might mean viewing a film and asking the preservice teachers in our classes to respond by movement, painting, or sculpture. We might read a story and respond with dramatic enactments that combine spoken words and body movement. Often, students embed movement or dance within a verbal response, or vice versa.

Drawing from our experience as teachers, we reflect on analytical frameworks used in our classrooms and the many lessons writing workshops have yielded. For example, we have had students collaborate on group research projects. They brainstorm topics, and align with other students who have similar interests; then they present their research findings to the rest of the class. Students are required to use multiple sign systems in ways they find helpful in the gathering and sense-making stages, as well as in their presentations. Writing, especially discursive writing, need not carry most of the meanings. After all, we are dealing with the generation nursed at the bosom of music videos, video games, the Internet, and “shock jock” radio announcers. Thus, we encouraged them to use multiple literacies to compose videos, graphs, collages, role-play scenarios, interactive games, music, and the like.

As time has passed, it has become necessary to add an explicitly critical stance to these projects, a stance that asks the questions, “Is this the way we want to live together? And how might we make things better?” The objective is to encourage small groups of students to share a common research goal that works toward social justice in more or less explicit ways -- a sort of a marriage of inquiry curriculum and social justice. Following are scenarios that make explicit what we mean by this transmedial marriage of inquiry curriculum and social justice.

Initiating Change

We all sat in the darkened university classroom, waiting for the next group of five undergraduate education students to begin their presentation of a project to “initiate change.” From one corner of the room came a voice speaking in rolling cadences, as if in church. Then a flashlight beam fell on a black face: It was J.D.

I’ve been a nigger all my life.
Oh! So I’ve been told.
My father is a nigger and so was my mother.
Oh! So I’ve been told.
If my father is a nigger and my mother is a nigger
Then my older sisters must be niggers too!
Oh! So I’ve been told.
Therefore there is nothing left for me to be but a nigger.
Right!
Oh! So I’ve been told.

Well, my father is a great man who’s always been there for me
And he has shown me an overwhelming love.
Oh! This I know.
My mother was a beautiful woman who led me the right way
And she always showed me an overwhelming love.
Oh! This I know.
My parents raised me in the way of the righteous and showed a path to glory.
Oh! This I know.
My sisters have guided me through their examples of intellect and courage
And they have both shown me an overwhelming love.
Oh! This I know!
But they are niggers still.
Right!
Just plain old uppity niggers to you.

Now if my mother, and my father, and my two sisters are no more than niggers to you...
Then I am proud to be a nigger, too.
Oh! This I know.

Silence. For what seemed minutes. Then someone raised the lights slowly and the group moved from the corners of the classroom toward the center. They spread a 10-foot sheet of paper on the floor and scattered colored markers across it. One of them announced that their change project was on prejudice on and around campus. They invited us to come forward and “write down all the derogatory terms you know. About women, men, lesbians, gays, Hispanics, African Americans -- all of ’em. Don’t hold back. Come on. No one’s keeping track. Just get it down.”

We moved on the paper -- some of us on all fours, some less aggressively. We wrote those words, words others had called us, words we were ashamed to know. In print we wrote, small and large, and covered that paper. One by one, we retreated to our seats, as if stunned by those orange, pink, green, blue, yellow, black, and red words glaring back at us.

Sound Stories

We agreed that our topic would be neighborhood sounds. Each group of five fourth graders was given a tape recorder. The groups were asked to go inside the buildings and find or make sound effects to tell a 3-minute story only in sounds -- no words. They were given 30 minutes to complete their work.

When students returned to the classroom, we agreed on the order of presentation, lowered the lights, and enjoyed the successive sound stories. Then we talked about how they had decided what sounds to include. My most poignant memory of this discussion was when one little girl asked another, “Is it safe where you live?”

Body Sculptures

I (Judith Fueyo) asked graduate students to read a qualitative research report on women with HIV/AIDS. In order to honor the personal, emotional content of the book, I invited students to respond with their bodies before we began to talk. In groups of four or five, they made sculptures with their bodies. My plan was for us to observe the sculptures, and then begin to talk.

Instead, students were moved beyond words. Tears. Heaving sobs. We had to break for 20 minutes before people could begin to speak.

Writing an Airplane

Chris was a first grader in a classroom where I (Judith) was a researcher. He wanted nothing to do with writing.

Surprisingly, one day he signed up for writing share. He came to the circle with a plane made out of paper towel tubing and construction paper. He faced his classmates, who expected Chris to read something. Instead, he began acting out a story of his plane diving down upon his friends before him. Chris wove among the others, inventing his story as he maneuvered his plane, to the delight of his listeners.

As he spoke, his teacher recorded his story on chart paper, word for word. She hoped Chris and others would read it some time. However, Chris’s story was not of much interest to his classmates; they wanted to learn how to make a plane! So, for the first time, Chris was willing to write something -- the “destructions” his classmates could follow to build their own planes.

Feeling Dehumanized

Undergraduates in a language arts course were reporting on their individual research projects. One woman, whose topic involved the Holocaust, began her report for the class in this way: “Push all the desks to the sides of the room. Squeeze together in the center of the room.” She was not satisfied until we were so squished that we were nearly breathless. Next she ordered us to sway back and forth while she sang, “Ka-chug, ka-ching, ka-chug, ka-ching...” over and over, in an attempt to simulate a railcar heading to the gas ovens.

We were crowded, stepping on one another, breathing in one another’s faces, hot, impatient. Finally, she allowed us to disperse and said, “I wanted you to feel dehumanized.”

Back to menu

Discussion

The scenarios described above capture some of the transmediations in our classrooms. In the first scenario, for example, the word “nigger” is a signifier in a human relationship. Such personification of race yields a critique of self, an opportunity for introspection, and provides a “teachable moment” during which students and teachers can relinquish the fears imposed on them by their cultural context and open up a dangerous topic in a powerful way. We question whether a verbocentric approach to such content would have moved the class toward this moment. What followed that presentation took up much of the remainder of the semester, and involved much energy and trepidation. But it was among the best teaching I (Judith) ever experienced.

Equally, the other scenarios aim to open our eyes to a variety of symbolisms, codes, and conventions that create a sense of community, compassion, and courage to experience deeper, even alternative, meanings that can be critiqued.

Students bring what they know and have learned in their communities into the classroom. As more students gain access to Internet communications and the media technologies of the 21st century, they are thrilled to see themselves as knowledge-makers who find and frame problems worth pursuing, negotiate interpretations, forge new connections, and represent meanings in transmedial ways. Teachers are therefore challenged to consider other methods of teaching and leading students in the understanding of sign systems ushered into the classroom by these multimedia technologies.

In the teaching practice of our classrooms, as demonstrated by the variety of scenarios presented here, different forms of representation do not perform identical cognitive functions. If they did, why would we want to teach how to write poetry, history, fiction, drama, or factual accounts of what we have experienced (Eisner, 1997)? But an emphasis on these multiple forms of representation continues to be seen as innovation; they have not become prominent in setting curricular agendas or in shaping education policy. The classroom scenarios explain how multiple symbol systems engage students in reading the world and the word as they learn how to represent the world they live in. As we reflect on the lessons we gain from multiple forms of representation, we must ask,

• How do students make sense of what they learn?
• What contradictions exist in the knowledge they acquire, invent, or interpret from the texts they read in classrooms?
• What are the different sign systems representing race, for example, they encounter in their classrooms?
• How do students come to learn, interpret, and negotiate the meaning of race from the variety of sign systems they encounter in their school life?

Among the various aims we consider important in education, two are especially so. We would like our children to be well informed-that is, to understand ideas that are important, useful, beautiful, and powerful. And we also want them to have the appetite and ability to think analytically and critically, to be able to use what they know to enhance their own lives and to contribute to a democracy.

It is unfortunate that the resource-rich environments that characterize good preschools and kindergartens are typically neutralized as young children move up into the grades. Upper grades push children to be verbocentric, but words are not as vivid in their minds as images are. We believe, as Eisner (1997) advocates, that we would be doing a better job if we replicated the creative features of kindergarten in the grades, rather than pushing the features of the grades into kindergarten. It is unfortunate that this is not likely to happen in schools as we know them because many teachers think that the transmedial experience of kindergartens is not “serious” enough for the education of upper level students.

Back to menu

Conclusion

Transmediation has the potential to capture the postmodern reality of multiple texts, multiple meanings, and multiple interpretations. Multiple forms of representation provide us with a critical framework for unpacking assumptions that underlie cultural practices. When the school’s curricular agenda is diverse, diverse aptitudes and experience can come into play. Educational equity is provided not merely by opening the doors of the school to the child but by providing opportunities to the child to succeed once he or she arrives. We believe that the habits of mind embedded in the pedagogy of new literacies provide students with the space and ability to imagine and value points of view different from their own. These new ways of seeing and knowing can strengthen, refine, enlarge, and reshape students’ and our own ideas. In this respect, we teachers must strive to encourage students and ourselves to feel comfortable -- and uncomfortable -- with transmedial representations.

Back to menu

References

Eisner, E. (1997, January). Cognition and representation: A way to pursue the American dream. Phi Delta Kappan, 78, 349-353.
Back

Ferrington, G. (1994). Keep your ear-lids open. Journal of Visual Literacy, 14(2), 52-61.
Back

Fueyo, J. (1991). Reading “literate sensibilities”: Resisting a verbocentric writing classroom. Language Arts, 68, 641-648.
Back

Metallions, N. (1994). New communication media technologies: Perceptual, cognitive, and aesthetic effects. Journal of Visual Literacy, 14(2), 41-49.
Back

Morris, C.W. (1971). Writings on the general theory of signs. In T. Sebeok (Ed.), Contributions to the doctrine of signs. The Hague, Netherlands: Mouton.
Back

Semali, L. (in press). Transmediation in the classroom. New York: Peter Lang.
Back

Siegel, M. (1995). More than words: The generative power of transmediation for learning. Canadian Journal of Education, 20(4), 455-475.
Back

Suhor, C. (1984). Towards a semiotics-based curriculum. Journal of Curriculum Studies, 16, 247-257.
Back

About the Authors

		Ladislaus Semali is an associate professor of education in the Curriculum and Instruction Department of the Pennsylvania State University (University Park, PA, USA; e-mail Lms11@psu.edu), specializing in language, media, and literacy education. His research involves critical media literacy, teacher education, and comparative perspectives in crosscultural communication in multicultural classrooms. Among his recent publications are Literacy in Multimedia America: Integrating Media Across the Curriculum (Routledge/Falmer) and Postliteracy in the Age of Democracy (Austin & Winfield). For the fall and winter of 2001-02, he is serving as guest editor of the New Literacies department of Reading Online.
		Judith Fueyo is an associate professor in the College of Education, Pennsylvania State University. She teaches courses in emergent literacy, language arts, writing, teacher research, and qualitative research. Her research focuses on how composing in alternative symbol systems is related to writing development, and the use of portfolios for alternative assessment. She has published in Language Arts, The Writing Teacher, The Journal of Education, and Teaching and Learning: The Journal of Natural Inquiry, and has chapters in many books.

Back to top

To print this article, point and click your mouse anywhere on the article’s text; then use your browser’s print command.

Citation: Semali, L., & Fueyo, J. (2001, December/January). Transmediation as a metaphor for new literacies in multimedia classrooms. Reading Online, 5(5). Available: http://www.readingonline.org/newliteracies/lit_index.asp?HREF=semali2/index.html

Reading Online, www.readingonline.org
Posted December 2001
© 2001 International Reading Association, Inc.   ISSN 1096-1232