Children’s Transmediation across Modes using Virtual Reality Technology

Year: 2019

Author: Mills, Kathy

Type of paper: Abstract refereed

Focus: Existing research has examined how young children shift meanings across sign systems long before they have mastered formal writing skills, yet virtual reality environments offer new affordances for users to translate semiotic material within and across virtual and material spaces that have not been researched. Transmediation of meanings across modes and media is relevant for literacy learning in schooling, particularly when the Australian Curriculum English requires students (P-10) to compose multimodal texts.

Theoretical Framework:In a digital age, children and youth are socialised into broadened repertoires of digital media conventions, which potentially include new virtual reality textual environments. This presentation extends theories of transmediation that have predominantly explored speech, drawing and writing, to advance new understandings about children’s transmediation of meanings using virtual reality technology. It extends knowledge of transmediation first coined by Suhor (1992), and extended by Semali (2002), Siegel (2006) and others (e.g. Author 2011). Every sign system or mode has unique organisational principles, involving elements or conventions that do not have precisely equivalent meanings when translated across modes. The lack of equivalence between modes is a catalyst for transmediation.

Methods: Nine to twelve-year-old children created 3D, virtual texts with Google Tilt Brush using a virtual reality headset and motion sensors. The creation had to represent an idea or element from their own written story. A multimodal analysis of the video recordings examined how the children shifted meanings from their written stories to 3D virtual reality scenes. The researcher also analysed the auto-recorded, minute-by-minute screencast of the children's’ designing in Google Tilt Brush. Insights on the children’s process of transmediation was also gathered through the analysis of a think-aloud protocol conducted with the children as they “painted” using the VR headset and motion sensors.

Findings:The research demonstrates that translating semiotic content from the materiality of written stories to 3D virtual reality design highlighted a lack of equivalence between the affordances of the sign systems and some similarities that both enabled and constrained the children’s communication of meaning.

Significance: The increasing availability of new digital technologies for creative representation of ideas, such as virtual reality, generates new possibilities for transmediation in young children’s multimodal compositions. Providing opportunities for students to translate literary content to virtual reality environments can enable them to go beyond the simple reproduction of story content to transform meaning and knowledge through a different and currently unexplored virtual materiality.