Authors' Message

Being a young person today means having to live in multiple worlds. As they move between home, school, and peer group, adolescents have to recalibrate their sense of what actions are appropriate, what is valued, and what is known. The conflicting forces represented in the mass media, on the street corner, and in one's immediate realm of rapid physical and social development create familiar difficulties for many young people.

George Gerbner, founder of the Cultural Environment Movement, along with other social critics, has argued that mass media exacerbate these problems. They see the global marketing strategy of television as creating an especially damaging and alienating environment. Others have pointed to video games or online chat rooms as new dangers for young people. At the same time, many educators have seen the learning potential of the Web, multimedia, and simulations.

The contrasting evaluations of new media remind us how little we actually know about what these different environments mean to adolescents, or to any of us, for that matter. What does it mean to develop and to explore social relationships with others in online environments? Can what one learns in a virtual environment apply to the real world? Can we draw a sharp line between real and virtual worlds?

This month, Michelle Hinn, Kevin Leander, and I explore a unique virtual world, which was created to help educators learn more about teaching adolescents using new technologies. As the project progressed, the simple question "Is this a good way to help students learn?" began to expand into a host of questions about what a simulation is, how to evaluate the use of new media, and how people learn.

The virtual world in question was initially only an object of study for a course for teachers called Evaluation of Information Technologies, which was designed by Michelle, Kevin, and me. But as students began to enter that virtual world and interact with it and within it, they changed it. Soon, it began to take on a life of its own, which we describe here and which we are still attempting to understand.


Back to menu

Issue:
Learning the Art of Evaluation

What's it like to participate in a virtual world? What can one learn by interacting in a space in which participants are both themselves and constructed online personas? We explored these questions as we brought into being a virtual junior high school, and then were swept along in its ever more real life. Our experiences taught us many things about virtual reality, the use of the Web, and how to evaluate innovations.

Our course, Evaluation of Information Technologies, focused on evaluating technology use in instruction. While the course was offered through an online master's program designed for educators, a mixture of 26 online master's degree students and 9 on-campus doctoral students were enrolled. The master's students were all practicing K-12 educators working full time while earning their master's degrees in instructional technology. The doctoral students were enrolled as full-time students in either the Writing Studies or the Technologies for Learning Program.

Our approach. The course had been taught once before as an on-campus doctoral seminar. However, because it was now going to be taught in the online master's degree sequence as a required research/evaluation course, one of our many challenges was how to adapt it for an online environment. But perhaps an even bigger challenge was how to make an evaluation methodologies course interesting and, in particular, interesting in an online context.

Key differences between student histories, communities, and interests were evident from the beginning of the course. The 26 master's students enrolled formed a cohort, for which this course was the third common experience in their degree program. Thus, the master's cohort was fairly accustomed to taking courses online. The doctoral students, on the other hand, were not taking this course as part of any particular sequence, and none had ever taken a course offered entirely online. Most of the doctoral students, however, had taken classes that used technology of some sort (such as an electronic mailing list or a Web board).

Why not have two separate sections for the course, such as one online and one on site? Because of the way that the course emerged. Because the course had been offered once before, it triggered interest amongst the on-campus students, even though it was originally intended for the online master's cohort. We felt that it would be a particularly interesting challenge to include both groups in an online course.

As a response to the challenges of course enrollment, course contexts, and the processes and content we wished the students to experience, we created an online course community in the form of an evaluation center. We modeled the center after offline evaluation centers such as the Evaluation Center at Western Michigan University, led by James Sanders, and the Center for Instructional Reform and Curriculum Evaluation, led by Bob Stake at the University of Illinois at Urbana-Champaign. Our evaluation center would have its own online newsletter, evaluation teams, and projects that spanned on- and offline worlds. Each student in the course was considered to be a member of the evaluation center, and the course instructors would serve as the center facilitators. When we began the course we did not have a name for the evaluation center, preferring to leave it up to the students in the class. One student came up with CITIES (Central Illinois Technology and Instructional Evaluation Service), and the name stuck. CITIES comprised six teams of five or six students each. Because we wished to diversify the teams as much as possible, we assigned the students to teams. Each team had one or two doctoral students and four or five master's students. Team members represented diverse academic disciplines and professional positions in the schools.

As course instructors, the three of us served as another type of team. In discussions of technology in education we often hear a discourse of efficiency in instruction, where teaching more students with fewer instructors and instructional time is held up as a value. However, in creating learning contexts and experiences that are experimental, efficiency should not be a goal at the outset. In our case, we put a lot of time into the course. We monitored and interacted very closely with two teams each while keeping one another informed of problems and successes occurring with our teams. We switched teams every few weeks so that we each had a chance to interact closely with all of the students.

The center had a guiding theme--the words of Lee Cronbach (1982), who said that "evaluation is an art." As an art, because evaluation is also improvisational when one considers multiple contexts and the variety of stakeholder and client needs, we gave each team the surname of a famous jazz artist (e.g., Louis Armstrong, John Coltrane, Duke Ellington).

The virtual evaluation project. A large part of the course centered around a virtual evaluation project that the evaluation center had undertaken of a simulated U.S. school district given a technology grant by a simulated funding body. In this case study, we created a scenario in which our evaluation center received a request from the National Education Foundation (NEF) to conduct an evaluation of an Illinois school district's use of its funds. The school district receives funds from NEF to implement a Learning With Computer Technologies program. During each section, a different evaluation problem arose that each evaluation team worked to solve.

The purpose of the virtual case study was to allow each evaluation team to explore the intricacies of the evaluation of the use of information technologies in a virtual setting. Each student also worked on an individual real-world evaluation project, and the virtual and real evaluations helped give some richer context for textbook readings, which can otherwise be dry. Why not just have the individuals do such real-world evaluations and share them online? A number of different reasons influenced our design. First, the distance between students and instructors and accessibility of real-world evaluation sites was a primary consideration. With all the students online, and most at a fair distance from the home campus, finding a "live" project for the entire class to work on would have been extremely difficult. We also wanted to have a group project in the course as well as individual projects so that we could provide a type of in-class activity for the online course. We wanted something in common that they were all working on, something that we could all learn from. Ethical concerns were also at play: We wanted to ensure that we had prepared students for the ethical issues in a safe arena. Thus, we created a simulation of a virtual school, one that perhaps seemed more real in the end than it had in the beginning.

The scenario. The National Education Foundation is very similar to other government-supported agencies and departments in the United States, like the National Science Foundation and the Department of Education, and it existed solely in our imagination. Or did it? The students had a unique opportunity to interact with one of the representatives of the NEF. We kicked off the online course with one face-to-face meeting before going fully online. This was so that we could introduce the master's cohort to the on-campus doctoral students and explain the design of the course. During this live session, the NEF representative spoke to the members of the CITIES evaluation center, explaining what their charge was, and took questions. The NEF representative was played, in most people's opinions, very accurately in his governmental agency vagueness. The representative was played by a professor at the university who had no prior experience with any of the students in the class. An interesting side note is that, whilst we thought originally that we had created the NEF, we found out that life sometimes does follow fiction when we received an announcement from the "real" National Education Foundation.

The Wynne County Unified School District won a large grant from the National Education Foundation to start up a Learning With Computer Technologies program in their county. Wildwood Junior High School was chosen as the main site for the evaluation, as it received the bulk of the funding. WJHS is a flagship school in the district, and its motto is "Without the students, we wouldn't be here." Sports play a big role at WJHS, and the principal is very supportive of the sports programs there. Because of his love of athletics, the principal takes a "team" attitude in his leadership of the school and tries to remain as accessible as possible to all the students and parents, stating on his homepage that "there's a PAL in principal." A visitor to the school principal's homepage will see an invitation to coffee as well as links to his favorite sports teams.

The staff at Wildwood worked extremely hard on their Web site, and it provided one of the main opportunities that the evaluation center members had for data collection. As the center members looked for more information when they began to focus their evaluations, the Web site often grew. The staff at Wildwood had not thought of including quite a bit of information on their Web site, and so for them the evaluation process was quite a learning experience and helped them communicate better with one another and the public through their Web resources.

Behind the scenes. Eighteen "live" volunteers at multiple campuses served as role-players (some playing multiple parts) for numerous faculty and staff of Wynne County and Wildwood Junior High School. The evaluation center members had access to the "e-mail addresses" of the Wynne County superintendent, the county technology director, and the county's evaluation specialist. At WJHS, the evaluators had access to the principal, the vice principal, 11 teachers, the school technology coordinator, two members of the PTA, and two students. In addition to being available via e-mail, several of the Wynne County and WJHS staff members were available to chat with the evaluation center members live through a synchronous text-based chat tool. This came about later as some teams wanted to "visit" Wildwood.

Wynne County and WJHS had "real" Web sites with which students could interact and from which they could try to glean as much information as possible. We didn't want to provide all information on the Web site because we wanted to encourage each team to interact with the role-players. Not only that, but we didn't even begin to know what "all information" was. We couldn't predict the directions teams would take in their evaluations. When one team would want to see the district's "report card" we would need to provide one. Thus the students helped create the reality of the simulation. We set the stage, and from then on the play became very improvisational. The role-players gave the simulation a new reality, and we no longer controlled it--just like in the real world.

In the beginning, the teams were uncertain about how to go about "talking" with the staff at Wildwood. So one person in one team contacted everyone at Wildwood with "Hi, I'm part of the evaluation team. I love your Web page" (paraphrasing). Another team responded with "I am a member of ____ evaluation team, and we are now ready to initiate electronic communication" (a survey was attached). In turn, the role-players either did not return e-mail at all (to the "Hi" team), and some only curtly responded to the survey (if at all). Both teams wondered why. So, in our commentaries for that week for the evaluation team members, we mentioned that the principal had called us and raised some concerns about the evaluation. Several teachers did not know the scope of the evaluation or even who these people were. Thus, WJHS school personnel were a bit unnerved by the survey without a proper introduction and the barrage of other e-mail. As instructors, we used this breakdown in communication as an occasion (a) to talk about how one initiates dialogue at an evaluation site and (b) to reconsider how real the people might be with whom the students were communicating.


Problems and Successes

Teams. It tended to be the best of times and the worst of times when it came to how team members interacted with one another and with the simulation. We had one team that needed a lot of encouragement and hand-holding in order for members to communicate with one another. On the other hand, members of another team worked amazingly well together, but there was a definite "Let's get things done as efficiently as possible" approach. Although its work was of very high quality, this team didn't spend a lot of time wrangling over process issues, nor did its members spend much time worrying about the authenticity of the simulation. As well as this team functioned to produce end results, perhaps it was the teams whose members didn't work as well together that were the most interesting to observe. Their members were often the most creative with respect to evaluation approaches. These teams questioned the simulation and fought against it. But they also often had the most creative solutions and asked the most probing questions, pushing the inquiry further than the best-of-times group.

Disorderly resources and perspectives. The simulation was constructed with a group of role-players who in many cases did not know one another and had different conceptions of WJHS, evaluation, and the course context and goals. The WJHS Web site was also developed by several different individuals with different perspectives on what the school might be, and these individuals drew upon texts and images from the Web and other disparate sources. Moreover, the course was taught by three instructors who, whilst in constant communication with one another, had many different interactions going with different course members (in their own roles as both instructors and role-players). For course members, the experience of doing the evaluation could often be frustrating due to these disorderly resources and perspectives. WJHS, the Wynne County School District, the NEF, and even the course itself did not make for an entity with a clear perspective. Rather, the simulation created a type of deliberate confusion and set of contradictions that students were directed to puzzle over. For some students, this type of work was particularly challenging and even frustrating; they believed the experience needed to be more clear and manageable. We understand this issue as related to the histories of the students in other, more "tidy" educational experiences and also as a critical response to our own expectations for their creation of order from a disorderly set of texts and experiences.

At the same time, as an indication of success in the project, we believe that a measure of disorder and a number of conflicting perspectives among role-players and Web texts were resources for the simulation. In school or other organizational contexts, participants have limited knowledge of other participants, have widely different perspectives on so-called common goals, and have limited knowledge of others' perspectives and activity. These aspects were in fact built in by the designed disorder of the simulation. A simulation with a well-bounded design team, led by a hierarchical management that had complete knowledge of its own product, would have developed a very different simulation with its own set of strengths, but it would likely not have drawn as strongly on disorder and multiple perspectives.

Authenticity, boundaries, and pleasure. At times, the need for boundaries or sharper distinctions to be made between the virtual and the real emerged from student confusion. For example, when the Wildwood project was first introduced to the class in a face-to-face meeting, one student, showing a great deal of engagement in the activity but also in Wildwood as a location, expressed interest in applying for a job at WJHS. Chip Bruce, who wanted to represent the virtual space by borrowing upon a rhetoric of the "real," while at the same time keeping distinctions clear for course (and even career) purposes, used his own embodied space in the classroom as a response:

(Chip, standing to one side of the room) Okay, I'm over here. This is Wildwood Junior High. It has a technology program that is funded from the National Education Foundation. This semester, evaluators are coming in to find out how this technology program is going so far.

(Chip, moving to other side of the room-across the front) Now I'm over here. I'm in the class on the Evaluation of Information Technologies. As far as I know, Wildwood Junior High School and the National Education Foundation only exist on the Web-they're not real places.

Chip used his embodied positioning in the space of the room as a metaphoric means of making clearer distinctions between the various locations of the project. At the same time, he also slightly blurred these locations ("as far as I know"), as the entire project traded to some extent on the meanings in common between the virtual space of Wildwood, real schools, and real human relationships as well as virtual relations between members of the university class and the role-players of the school.

Because WJHS was shaped by many page designers' and role-players' experiences with schools in different parts of the U.S., it was hard for some students in the class to buy some of the stories and images related to the school. A few times the instructors received comments like "This isn't like a school," and we had to respond with "Wildwood isn't like all schools." In the end, however, we received comments like "You know, we were looking at the Web sites of other schools in some other states and realized that Wildwood is like other schools in a lot of ways."

The lack of believability was also related to the manner in which the Web pages and role-player interactions used humor to play at the boundaries of school life. For example, the school principal, Lee Daniels, was a caricature of a secondary principal whose primary interest was in athletics (having written a master's thesis entitled "We're All on Your Team: Supporting Student Athletes in the Classroom.") Many examples of such humorous play existed in the experience, and pushing boundaries of parody in schooling was one of the pleasures of engaging in the simulation for role-players, instructors, and students alike. At the same time, students had to be willing to enter into this play for the simulation to be meaningful to them. While some students enjoyed and even relished the experience, others seemed to critically distance themselves from it at times as not being "authentic" enough. These positions vis-à-vis the simulation are complex, and we have yet to work them out. One of the complexities, of course, involves understanding not just engaging in simulation, but engaging in simulated "play" as part of school "work," which constructs different relations of power and different consequences for making meaning among instructors, role-players, and students.

Reality of the virtual. A reality of sorts emerged out of the simulation that was not real in the sense that the evaluation teams could go and actually visit WJHS. But there emerged an alternative reality to the simulation, a community of real people doing real things (even if they were simply creating mock-ups) that other real people were evaluating. In some ways, the simulation in the end began to resemble the constant formative nature of most instructional technology projects. As such, the students had to approach the simulation in the same sort of way one might approach a new use of instructional technology. In any evaluation, there exists unknowns. One might go into an evaluation with a set impression of what quality will look like or what the developers say that quality will be and often miss the unexpected, or the unknown. Importantly, the unexpected might be something more wonderful (or more terrible) than what was being deliberately sought after in the first place. With the simulation, we were trying to help the students look beyond what they feel is true from their past experiences and look for the unexpected.

Another general manner in which the simulation took on a real life of its own was that, as a set of texts and resources, WJHS and the evaluations of it were created in dialogue with one another. Wildwood was not fixed in advance of an evaluation of it, but responded to the questions asked of it and was created through them. For instance, Lee Daniels, the principal of Wildwood, had not properly prepared budget documents for the evaluators in advance. However, when asked about them, his postponement could be interpreted as his lack of preparation. Several days later the documents appeared and were posted to the Web site. This dialogic emergence (Holland, Holyoak, Nisbett, & Thagard, 1986) of resources, social contexts, persons, and social practices in the simulation is suggestive of how social life develops across on- and offline worlds.

Final Comments

Information technologies bring in a set of special issues for evaluation (Baker & O'Neil, 1994; Johnston, 1984; Rice, 1984). Among these are technical characteristics of the technologies, including collaborative tools, interface features, and hypertext structures. Additionally, as use scales up, the new media reshape geography through spatially dispersed contexts of use and variations in implementation. Thus, new modes of communication, whose effects modify with use, pose new challenges for understanding collaboration. At the same time, technology-based, collaborative learning projects, such as the one described in this column, typically experience continual revision throughout their implementation, implying that we are effectively engaged in perpetual formative evaluation (Bruce & Rubin, 1993).

Back to menu

Interpretations

Simulacrum and Simulation

Loosely defined, we might term any object or processes of a simulated world as a simulacrum (plural, simulacra). (Simulacrum derives from the Latin simulare, meaning to make like or to put on an appearance of.) Simulations make use of all manner of settings, characters, and action, which may strike us as copies of settings, characters, and action in the real world. However, as it turns out, the idea of a simulacrum as a copy of a real-world object is only one means of conceptualizing simulated and real relations. What are other possibilities, and where might they lead as we reconsider the dynamics of simulation? Below, we offer three contrasting perspectives.

1. Pragmatic: Copying reality. A pragmatic perspective is primarily concerned with the practical needs that drive our desire to create simulations. In the case of the Wildwood simulation, a pragmatic perspective would claim that doing an evaluation of an actual junior high school would be too costly or too difficult to coordinate for the course instructors and students. The technology enables one to surmount such practical difficulties and to create something that is very much like, or perhaps intensifies, encounters in real situations. A pragmatic perspective would generally assume that a simulacrum--such as the principal at WJHS--is a copy of something (in this case, a stereotypical principal, or perhaps caricature) in the real world.

2. Pessimistic: Rupturing with reality. French social theorist Jean Baudrillard (1983) has written much about simulacra and simulation, and he is perhaps best known for theorizing that in postmodern culture images and texts of all kinds have become separated from any relation to reality; signs no longer represent anything "out there" but merely refer to other signs. A postmodern world in which images no longer bear any relation to reality is termed "hyperreality." Disneyland, a self-perpetuating zone of images of images, copies of copies, is a prime example of hyperreality. The simulacra of Disneyland are cut off from anything real, and yet this rupture is not recognized in contemporary life-as-simulation.

3. Productive: Appropriating reality. Like Baudrillard, French theorists Gilles Deleuze (1983) and Felix Guattari (Deleuze & Guattari, 1977) considered the relation of copy to model as a starting point. However, they argued that the relation between simulacrum and model is external and deceptive. In the case of the simulacrum, the process of its production, its inner dynamism, is entirely different from that of its supposed model: Its resemblance to it is merely a surface effect, an illusion (Massumi, 1987). Resemblance in simulation-the ways in which Wildwood looks like the Web page of a junior high school-is only an entrée for considering how reality has been appropriated in the production of the simulacra. In this perspective, the copy/model binary breaks down: Reality cannot be understood apart from the processes of simulation operating behind it, and simulation both draws upon and produces new realities.

Other considerations

While a pragmatic perspective describes something of our own work in creating Wildwood and the simulated evaluation, it tells us relatively little about the relation of simulated and real worlds as they are experienced by players, or about the unfolding of the simulation as an activity and sociotechnical space. Baudrillard's perspective on simulacra-signs made meaningful primarily in relation to other signs-is helpful in understanding the ways in which Wildwood is produced as a world of images and texts that becomes semi-independent from any "real world" we might nominate. Deleuze and Guattari (1977) prompted us to consider, at a deeper level, how Wildwood appropriates, rearranges, and effectively draws upon actual junior high schools and teams of evaluators, but re-creates them in new ways and for new purposes. Copying reality, in this sense, is not the goal of the activity but merely an illusion masking the production of a new educational practice and space.

Back to menu

Literacy Web Site of the Month

Nowhere Road is the Web site of Lloyd Rieber, a professor of instructional technology at the University of Georgia, Athens, USA. The site contains numerous examples of games and simulations for education, including "Nowhere Road--The Game" (where the player assumes the role of a professor who must bike to work "without getting bit, run over, or emotionally scarred for life"), "Space Shuttle Commander" (a simulation designed to introduce Isaac Newton's laws of motion by giving students the role of a commander of a space shuttle), and "SimSchool" (a simulation that examines the philosophical perspectives of objectivism and constructivism and their influence on educational practice). "Nowhere Road" also hosts several papers and electronic texts on building simulations and games for educational settings.

Back to menu

Glossary

Avatar--something in electronic space, often an image but not always, that represents a user in a multiuser situation such as a multiplayer electronic game, simulation, or chat session.

Evaluation--the process of determining the merit, worth, or quality of the object of inquiry.

Role-playing games (RPGs)--games in which each player assumes the role of a character (such as a principal or a student, as in the WJHS simulation) who can interact with other characters within the imaginary world of the game.

Virtual reality--in simplest terms, refers to a simulation of either a real or imagined environment or "world." Virtual reality has been used to describe many different activities including online collaboration through text or video, the imagined environments of a game, and 3D representations of lifelike images displayed through technologies such as CAVEs (Cave Automated Virtual Environments).

Back to menu

References

Baker, E.L., & O'Neil, H.F. (1994). Technology assessment in education and training. Hillsdale, NJ: Erlbaum.
Back

Baudrillard, J. (1983). Simulations (P. Foss, P. Patton, & J. Johnston, Trans.). New York: Semiotexte.
Back

Bruce, B.C., & Rubin, A.D. (1993). Electronic quills: A situated evaluation of using computers for writing in classrooms. Hillsdale, NJ: Erlbaum.
Back

Cronbach, L. (1982). Designing evaluations of educational and social programs. San Francisco: Jossey-Bass.
Back

Deleuze, G. (1983, Winter). Plato and the simulacrum. October, 27, 48-56.
Back

Deleuze, G., & Guattari, F. (1977). Anti-Oedipus (R. Hurley, M. Seem, & H.R. Lane, Trans.). New York: Viking.
Back (1st citation) (2nd citation)

Holland, J.H., Holyoak, K.J., Nisbett, R.E., & Thagard, P.R. (1986). Induction: Processes of inference, learning, and discovery. Cambridge, MA: MIT Press.
Back

Johnston, J. (Ed.). (1984). Evaluating the new information technologies. Washington, DC: Jossey-Bass.
Back

Massumi, B. (1987). Realer than real: The simulacrum according to Deleuze and Guattari. [Online]. Available: http://www.anu.edu/HRC/first_and_last/works/realer.htm
Back

Rice, R.E. (1984). Evaluating new media systems. In J. Johnston (Ed.), Evaluating the new information technologies (pp. 53-71). Washington, DC: Jossey-Bass.
Back

Back to menu

Guest Authors

Hinn is a graduate student in educational psychology at the University of Illinois, Urbana-Champaign. E-mail: hinn@uiuc.edu. Mail: 220 Education Building, 1310 South Sixth Street, Champaign, IL 61820, USA. Leander teaches at Peabody College, Vanderbilt University in Nashville, Tennessee, USA.

Reader comments on this column are welcome. Please send ideas for future discussion, sites for consideration, literacy and student Web pages for sharing, and Glossary suggestions to the Technology Department editor. E-mail: chip@uiuc.edu. Mail: Bertram C. Bruce, Graduate School of Library & Information Science, University of Illinois at Urbana-Champaign, 501 East Daniel Street, MC 493, Champaign, IL 61820, USA.

Back

For an index of JAAL Technology columns available at this site, click here. To print this column, point and click anywhere on the main text; then use your browser's print command.

Citation: Hinn, D.M., Leander, K., & Bruce, B.C. (2001, October). Case studies of a virtual school. Journal of Adolescent & Adult Literacy, 45(2). Available: http://www.readingonline.org/electronic/elec_index.asp?HREF=/electronic/jaal/10-01_Column/index.html

Reading Online, www.readingonline.org
Published October 2001 in the Journal of Adolescent & Adult Literacy
Posted simultaneously in Reading Online
© 2001 International Reading Association, Inc.   ISSN 1096-1232