How we're designing musical instruments with the help of disabled musicians and VR

January 16, 2019 by Franziska Schroeder And Matilde Meireles, The Conversation
Virtual Reality technology opens up new experiences and possibilities in music for people with disabilities. Credit: Performance Without Barriers, Author provided

Most new digital technologies tend to be designed with an able-bodied user in mind. The first desktop computers required fine motor skills to navigate software menus using a mouse, and mobile phones need users to press buttons, swipe screens, and so on. To use such technology a person needs to be fairly dexterous.

In our Performance without Barriers research group, we design digital musical interfaces with disabled in mind. This work engages disabled performers from the start of a new technology and looks at the specific abilities they have. In this way, technologies develop in tandem with them, taking into account their mobility, needs and creative interests.

Current VR technology is designed for the able bodied, but more importantly it often allows only for passive interactions – listening to music performances, such as Elton John's 360 concert, for example, or "riding" a rollercoaster. We were more interested in how disabled musicians can use VR technologies in an active and performative way.

Working together

Our research team consists of a diverse group, including electronic engineers, computer scientists, sonic arts researchers, immersive content designers, a soloist ensemble and a local group that helps disabled musicians perform and compose their own music independently. Together with these musicians, we teamed up with a US software developer, who was designing a VR called the "Infinite Instrument", running on a 360° VR headset called HTC VIVE.

The was developed with able-bodied musicians in mind, so we designed it to take into account different types of mobilities. This led to one of our musicians with cerebral palsy playing a new VR instrument that was specifically designed to take into account her expressive upper body movements. It did not require her to use fine motor precision in her arms or fingers, which she does not possess.

VR headsets are necessarily about what you see. However, we found that the tactile feedback from the hand-held controllers – through which a user accesses a menu and press commands to reach content – allowed this particular musician to play the instrument by feeling and hearing it, rather than seeing it through the headset. Not having to rely on the headset also meant she could maintain visual contact with other musicians during the performance.

Another example of our collaborative VR design was working with a blind performer. Key to this musician were his virtuosic skills on the clarinet. The idea was to build on his expertise and adapt and enhance his musical performance skills using VR technologies.

VR experiences are so concentrated on the visual experience that often they disregard the inherently immersive nature of sound, but it's all around us, a proper 360° experience. Clearly, a visual headset was not relevant or helpful to this blind . We needed to focus on how we could use the other immersive qualities of a space for a music performance.

Our team is based at the Sonic LAB in Queen's, an immersive and fully customisable 3-D sound space – often referred to as the iMAX for the ears – we decided to adapt the VIVE technology to this existing context.

Virtual Reality technology opens up new experiences and possibilities in music for people with disabilities. Credit: Performance Without Barriers, Author provided

For our clarinettist we focused on the VIVE controllers, as they are more tactile than visual. They can also track physical positions in a space, which meant the clarinettist was able to position sounds in the 3-D Sonic LAB by having the VIVE controller strapped to his instrument.

Inclusive thinking and design

These bespoke VR instrument designs featured in a showcase concert in November 2018, where disabled musicians performed alongside musicians from the Hard Rain Soloist Ensemble.

The audience was positioned in a circle closely surrounding the musicians in order to enhance the immersive experience of the performance space.

The disability equipment market worldwide is estimated to increase to more than US$8 billion by 2020, and sales of disabled equipment in the UK have increased over 93% over the last ten years. All technologies, including VR, can be inclusive if the perspective of disabled people is part of their design.

Our approach reveals how new technologies can be developed, that actively engage disabled musicians in music making and demonstrate a commitment to quality of life for disabled musicians. We will continue to design instruments that can be used in VR, but we will now focus on using more affordable systems, with a view to creating a ensemble of disabled and able-bodied musicians.

Explore further: How to dance to a synthetic band

Related Stories

How to dance to a synthetic band

November 16, 2018

Music plays an important role in most people's lives regardless of the genre and in a wide variety of contexts from celebrations and parties to simply providing background while a task is being performed. Until very recently, ...

Study tunes into musician concerns with hearing loss

January 14, 2019

Findings from a Western-led study into the biggest concerns faced by musicians with hearing loss should strike a chord with educators and conductors and result in new ways to lead ensembles, researchers explained.

Music-making for the deaf

November 25, 2015

A Birmingham (UK) researcher is exploring new ways to enhance the experience of deaf musicians with new visual and touch techniques.

Recommended for you


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.