Abstract:
RelevanceDigital and multimedia environments have profoundly changed the way we communicate in recent decades, with the convergence of technology and media, and the transformation of everyday literacy practices. Recently, virtual reality (VR) technologies have rapidly become more accessible in homes and education, with different meaning-making potentials and affordances for transmediation in textual practices that include immersive storytelling, interactive reading, virtual painting, using three-dimensional models, and other emerging forms of sign-making. Transmediation is a vital learning and knowledge generation process that is of global relevance in multimedia-based digital environments. The rapid evolution of VR technologies and their adoption for learning opens up new possibilities for shifting semiotic content across modes, with underexplored scope for transmediating content in visual, haptic, and auditory ways in immersive media literacy practices. Methods The research investigated users’ creative digital designing involving an immersive virtual painting program with upper elementary students who used a VR head mounted display and sensors. The 44 students were from an elementary school aged between 10 to 12 years (Year 5). Data collection involved video recording of each student creating their virtual design, continuous screen capture, and a recorded think-aloud protocol in the virtual reality learning context. The students' drawings were also compared to the finished virtual design. All data sets were coded to the following four main themes that are distinctive features of immersive virtual reality environments: i) 3D immersion, ii) subjective presence, iii) sensory illusion, and iv) interactivity with motion tracking. FindingsThe findings enliven how students transmediated the same story across written, verbal, and virtual painting modes, tracing key themes of the students’ virtual experience of 3D immersion, presence, sensory illusion, and heightened bodily interactivity. Each of these themes highlight moments of new opportunity and challenges for communicating ideas in the virtual painting mode. For example, student loved that they could be "inside" and "walk around inside" their virtual and three-dimensional creations, but also experienced difficulty finding where to place their virtual brush in a world that had no physical tangibility. Students reassembled and shifted narrative content, sometimes seamlessly, while experiencing ambiguity and complexity about 3D representation in an immersive world. Producing stories across modes invited adaptation and generative thinking to overcome the lack of equivalence between writing, drawing, and virtual painting, involving more than the simple reproduction of story content. ContributionVR technologies can support a range of pedagogies, with clear advantages for experiential and situated learning. This research has shown that VR technologies can support the translation of knowledge across sign-making systems. Even when a task seemingly calls for a simple reproduction of declarative knowledge learned through reading, listening, drawing, and writing, the process of shifting this knowledge into a new mode, such as virtual painting, is never a simple reproduction. Rather, the lack of equivalence invites learners to engage in a generative process of adaption and knowledge transformation. This is because learners must recast knowledge via increasingly complex, multimodal, immersive, and interactive, yet physically intangible, three-dimensional expression planes that now feature in education.