Engineering an Artistic Experience in VR: Sound and Vision from Chroma

If you could take a break from your engineering tasks and immerse yourself in a world of your own imagination, what world would you create? How would you define a virtual world of sound and vision? How would you weave sound, vision and motion into a compelling or relaxing work of art for yourself and others?  

 You'd be hard pressed to accomplish this by yourself, since multi-disciplinary artistry and room for experimentation would be required to engineer such a feat. You would also benefit from paying attention to Chroma Mixed Media (Chroma), a group of multi-SOCAN award-winning composers and interdisciplinary artists who've teamed up to transform the idea of this type of interactive art installation into a real experience for the public.  

Chroma was founded in 2015 by Katerina Gimon* (composer, vocalist) David Storen (composer, visual artist), and Brian Topp (composer, developer, audio engineer). Based in Vancouver, Canada, Chroma has presented a multitude of projects that explore the relationship between music, visual art, performance, composition, and new technologies.  

 

“What’s unique about Chroma is our diverse backgrounds – in music, art, and technology – which are well-aligned with our vision to explore the artistic possibilities that emerge from combing ‘traditional’ art forms with new technologies.” – Katerina Gimon 

Katerina Gimon, whose compositions have been performed and commissioned by ensembles across Canada, the United States (including Carnegie Hall) and Europe is one-third of Chroma. She was named one of the ‘hot 30 classical musicians under 30’ by the Canadian Broadcasting Company and her reputation as a composer, improviser, and vocalist is growing among contemporary Canadian composition circles.  

David Storen is a three-time SOCAN award winning composer, guitarist, and is the lead artist at IronSky Games. His work as a composer reflects his fascination with the visual arts by continually searching for ways to integrate them into his practice. In 2015 and 2016, Storen received support from the Social Sciences and Humanities Research Council of Canada for a research project that culminated in the development of software for live video manipulation and mixing. His music has been performed across Canada by ensembles such as FAWN Chamber Creative, Standing Wave and Ensemble Paramirabo.  

Engineering.com had a chance to speak with Chroma about their upcoming immersive audio-visual installation which is called Naona.  

What is Naona?

Chroma describes Naona as a work which "places the listener inside a fictional, virtual landscape populated by ancient ruins." As the listener progresses through a series of environments, their movements and interactions within the environments create changes in the musical backdrop. This creates a unique musical journey each time a user experiences Naona.  

 

Naona is being developed for the Oculus Go and will be presented as a virtual reality installation at the Canadian Music Centre in Vancouver from September 12th to 15th.  Besides creating 40 minutes of music and visuals, the members of Chroma have been preparing for analogous challenges, like minimizing the learning curve and maximizing comfort levels for their intended audience, many of whom will be encountering virtual reality for the first time. To help mitigate these challenges, Chroma will be giving demonstrations and working one-on-one with users to get them comfortable with the platform.  

 

“Since the world of artistic exploration in VR is relatively new, we hope Naona will help our audiences understand its creative potential and encourage new exploration.” – Katerina Gimon 

In parallel with Naona, Chroma is developing VR-OSC (Virtual Reality – Open Sound Control), a platform for composing and performing music from within virtual reality. VR-OSC has been the result of Topp’s thesis research at the University of British Columbia and explores several unique and virtual reality specific ways in which a user can control musical parameters.  This platform allows users to create customized and unique performance environments from a variety of simple and complex controllers. These include everything from simple sliders and dials to virtual reality specific controllers such as physics-based controllers, spatial mapping, and multi-modal controllers. VR-OSC works by using Open Sound Control messaging (OSC) to interface with external audio software such as Max/MSP, Ableton Live, and Logic.  VR-OSC is set to release mid 2020.  

 

“VR-OSC allows us to think about how we can interact with music in new and unusual ways. Because the environment is virtual, so many traditional constraints are removed, from how instruments and controllers are laid out, how many instruments we can have, and how we derive meaningful data for controlling musical parameters. In a virtual environment we can easily map position, velocity, direction and collision information to pitch, volume, note length, and many other musical parameters.” – Brian Topp 

Chroma is effectively using new digital technology as a way to meld music composition, performance, visual art, and interactive design into an experimental, cohesive and unique experience. Prior to Chroma, Katerina and Brian were involved in a project called Kicass (in collaboration with the UBC Laptop Orchestra) where they were exploring similar concepts using the Microsoft Kinect. Of the earlier project, Topp said, "we developed a platform for it where we'd have a single Kinect tracking a dancer and streaming her position data over OSC, which other members could tap into. We could use the dancer's position or kinespheric motions to control music and visuals in a reactive way." Brian is also working on a software that allows him to integrate haptic data from a dancer's suit (RUBS/TASTE). They will be presenting the work with dancers, touch suits, motion tracking, live video, interactive electronics at different installations in the coming months, including at the International Computer Music Conference in New York this coming June. 

 

"The technology becomes a way in which you mediate between different art forms." -Brian Topp, Chroma Mixed Media 

 Bottom Line

Knowing that they are turning their music into a digital soundscape and knowing that users will be able to see and change it as a living visual art is an awesome challenge for Chroma and will hopefully yield excitation from the public if they are successful.  

 There are many new worlds, visions, sounds and inventions waiting to be created, and no one can do it alone.  Engineers, like artists, know this all too well.