Day 1, Stream 4

Chair

Lawrence Wallen: School of Design, UTS, Australia

Speakers

Hannah K. Schraffenberger: Leiden University, LIACS, Media Technology Research Group & AR Lab, Royal Academy of Art, The Hague, Netherlands
Ruth Aylett: Heriot-Watt University, Edinburgh, UK
Troy Innocent: Swinburne University of Technology, Australia
Wim van Eck: AR Lab, Netherlands
Gregory Garvey: Quinnipiac University, United States
Christian Clark & Tomas Laurenzo: Facultad de Ingeniería, Universidad de la República, Uruguay

ABSTRACTS

Hannah K. Schraffenberger & Edwin van der Heide: From Coexistence to Interaction: Interrelating the Virtual and the Real in Augmented Reality

In Augmented Reality (AR), virtual content is added to our real environment. This allows for unique relationships between the virtual and the real. So far, much work in the field has focused on scenarios in which virtual and real objects appear to coexist in the same physical space. In contrast, our research looks at AR works in which the virtual and the real not only coexist, but also influence and interact with each other. Influences between the virtual and the real can, for example, take the form of influences that exist between real entities; for example, a real fan might move virtual leaves. However, the virtual does not have to follow the laws of the physical world. As a consequence, we can realise novel interactions that have no equivalent in the physical world, yet appear believable nonetheless. With our work, we explore the possible interactions and influences between the virtual and the real in AR. We present theoretical considerations as well as an initial series of experiments. We believe that relating the virtual and the real by means of interaction can result in exciting artworks as well as novel manifestations of AR. We hope that our research serves as a first step towards works in which the virtual and the real enter an active dialogue.

Ruth Aylett & Sandy Louchart: Interactive Drama in Virtual and Augmented Reality

How do we resolve the paradox of computer-supported interactive drama – that the human participant requires the very freedom to interact that the authored narrative structure denies them? Does the struggle between author and participant inevitably result in winners and losers, or is there a feasible relationship in which both can be satisfied?

This paper reports work around the concept of Emergent Narrative – the development of narrative structure through interaction itself, which can already be seen in non-technological genres such as Live Role-play or Improv. In a succession of systems built both in the virtual world and the virtually-augmented real world, we have explored how far reworking narrative structure as a loop between causal (plot) and affective (character) chaining can produce engaging experiences for participants. We seek to describe dramatic trajectories as patterns of affect and thus to enable virtual actors able to evaluate the possible dramatic impacts of their actions before they take them. A key direction is the development of affective architecture for virtual actors, in which the concept of cognitive appraisal, taken from psychology, can be used to support predictions about the impact of actions on other characters and on human participants. We have designed a ‘double-appraisal’ mechanism that allows this to be implemented in a computationally-feasible manner. The ‘intentional stance’ of interaction between humans can be seen as a source of dramatic regulation and a way of supporting human participants without acting experience in role-play that feels free while being ‘shaped’ so as to reach dramatic goals.

The systems we have built test aspects of the architectures and mechanisms that offer promise for solving the paradox of interactive drama. Using the newly available low-cost interaction technology of Kinect, Wiimote, Dancemat, a path towards technology-supported interactive drama seems a real possibility.

Troy Innocent: Code switching in mixed realities

The codes used in AR systems may act as signifiers of an alternate reality in themselves prior to any technological reading. Mixed realities in urban settings are complex media ecologies that are often traversed in a transmedial way by players and participants. Making AR markers a significant part of the urban landscape by aestheticising them results in an intervention into public space that signifies the presence of an alternate world situated within the real. Recent projects such as Urban Codemakers and noemaflux explore connections between formal abstraction, street art, pervasive gaming and virtual art creating mixed realities with hybrid aesthetics and multiple layers of meaning.

This paper will explore a family of digital media languages that have been constructed or evolved in artificial worlds have been adapted for expression in a range of material forms, such as urban art, lasercut signs, installation, stencils and handheld objects. These forms – expressed in paper, aluminium, acrylic and light – translate digital media aesthetics into tangible, spatial, immediate experiences. They are imprinted onto and into reality.

The languages undergoing this transfer out of the virtual are a form of multimedia, audiovisual iconography, what we refer to here as ‘dynamic ideography’, as it has been theorised in the work of Pierre Lévy (Lévy, 1991). In pre-linguistic cultures visual languages were the primary means of communication. These were carved into rock, sculpted from wood, and painted onto clothing. During the digital revolution of the last century, ideographic forms of communication thrived in the audiovisual world of the computer using movement, icons, colours, gestures and sounds. Inscribed into reality these languages now have an evolved aesthetic that is shaped by their materiality: shaped as glyphs carved into rocks, adapted to woodblock and metal type, drawn as pixels and vectors, and hybridized in mixed realities.

Wim van Eck: Trees as Time Capsules: Extending Airborne Museum Hartenstein to the Forest

The AR Lab is a cooperative effort between the Royal Academy of Art in The Hague, Delft University of Technology and Leiden University. Developing augmented reality installations for museums is one of our specialties, and currently we are working together with the “Airborne Museum Hartenstein”. The museum focuses on the Battle of Arnhem which took place during World War II. It is housed in a villa that served as the Allied headquarters, and heavy fighting took place in the forest in the surrounding area.

While visiting the museum, we noticed that a forester was having trouble cutting down a tree next to the museum, because objects lodged inside the tree hindered his chainsaw. The forester explained that the trees in the forest have a lot of metal fragments inside of them, such as bullets and grenade shrapnel; silent reminders of the heavy fighting which took place during the war. The trees were still young when the bullets hit them, and over the course of some 65 years the trees have completely encapsulated the fragments, thereby also protecting them from environmental influences. When you know what to look for you can still see the spots where bullets and other metal fragments pierced a tree in its younger years.

We find it fascinating how these trees took on the role of living time capsules. The museum itself houses the world’s largest collection of militaria from the Battle of Arnhem; our project will reveal all of the hidden content in the trees just outside of the museum, thus extending the museum in a sense. We are currently developing an augmented reality application running on smartphones which will give the museum’s visitors the opportunity to look inside the trees. At ISEA2013 we will present our first prototype of the application.

Gregory Garvey: Sanity and Mental Health in an Age of Augmented and Virtual Realities

In an age where new technologies such as augmented reality blur the boundaries of the real and virtual how do we achieve a consensus about the manufacture of reality and of mental health? Since the period of the enlightenment science has been the answer. Today we live in era of global warming denial; where political campaigns ignore “fact checkers” and “reality based media;” where we witness the rise of religious extremism and the clash of narratives about reality and civilization; where voices are silenced through terrorist attacks; where as von Clausewitz might have said unmanned drone warfare is politics by other means; where pilots of unmanned vehicles suffer Post Traumatic Stress Disorder and the “collateral damage” is unreported as if those lives never existed; where a war on terror is unceasing and those imprisoned in the gulag of Guantanamo must wait for the cessation of hostilities to receive justice. Add the overlay of virtual worlds and augmented realities, along the real possibility of malware and cyber attacks that might distort what is heard and seen, where does it leave the rational understanding of the “real?” How do we determine if “reality testing is intact?” And whose reality is it anyway?

Edward Castronova argues “the real world is going to lose” – there will be an exodus from the real to the virtual. To understanding the complex nature of the borders and boundaries that separate the real from the virtual I have developed and deployed two survey instruments to study if virtual realities induce dissociative disorders. One is based on the Structured Clinical Interview for Depersonalization – Derealization Spectrum (Mula et al., 2008) and the second is based on the Boundary Questionnaire (Hartmann et al., 2001). This paper reports the results and the implications for our shared consensual hallucination (reality).

Christian Clark & Tomas Laurenzo: CELEBRA

In this paper we present Celebra, a massive, site-specific, interactive, post geographic installation. It comprises a suspended network of two-hundred, one-square-meter led-lit balloons, with an organic aesthetic, that combines the grunginess and DIY-style of the underlying electronics with an extremely refined visual output and interaction scheme.

Although it has been shown in a site-specific manner (a jail in Uruguay reconverted into a museum), presenting an explicit tension between the space and the piece, it also post-geographically deconstructs the relationship artwork-interactor (and interactor-interactor) by allowing real-time bi-directional interaction through internet.

The work is framed within three main axes:
1) media appropriation: the utilisation of technology as a raw medium of artistic production, where artists must (and can, by the modularity of new media) dip arbitrarily down into the technological components, adapting, and creating new ones, producing new new media artworks.

2) the explicitness of interaction. Although all art is interactive (in Duchamp’s words: “The creative act is not performed by the artist alone; the spectator brings the work in contact with the external world by deciphering and interpreting its inner qualifications and thus adds his contribution to the creative act.”) by making the interaction explicit we can work on interaction aesthetics (quoting Myron Krueger “I have been trying to raise interactivity to the level of an art form as opposed to making art work that happened to be interactive”), or we can attempt to create an artistic interaction language.

3) the context. In Eladio Diestes words, “Each problem [...] should be faced with a sort of ingenuity, [...] with an attitude humble and vigilant. It should be thought again, with the basic body of knowledge that is now the heritage of all men”, but assuming it implies a radical change of attitude, embracing its context, its space, and its time.