A ubiquitous immersive sound reproduction system

Mobile Maestro enables users to enjoy a high-quality listening experience anytime and anywhere through the collaborative use of multiple mobile devices.

Immersive sound (e.g., 5.1-channel surround sound) reproduction creates a rich, engaging user experience, and people strongly desire high-quality listening experiences in various situations. Imagine a group of friends on a camping trip in a mountain setting. After dinner, they would like to watch an action movie together with immersive surround sound. Alternatively, some movie fans may want to watch their favorite movie scenes at film locations (e.g., watching Hogwarts dining hall scenes from the Harry Potter movies series at the University of Oxford) with a rich sound effect.

The collaborative use of multiple mobile devices opens an opportunity for an in-situ high-quality listening experience. As shown in the accompanying figure, for 5.1-channel surround sound, users can utilize five mobile devices, each of which plays its own audio channel to jointly deliver sound effects. Thanks to the portability of mobile devices, this brings more opportunities to experience immersive sound environments regardless of time and place.

However, in mobile sound environments, it is particularly challenging to reproduce immersive sound compared to conventional sound systems (e.g., home theater systems). A common, traditional requirement for immersive sound reproduction is to high-accurately simulate the difference in the sound arrival times (up to 1ms) and the difference in the sound arrival levels (as accurately as possible). Toward this, the existing systems require that loudspeaker configurations should be pre-defined at specific locations with a specific type of loudspeakers. On the other hand, the mobile sound environment is quite different from the recommended configurations, making it difficult to support accurate arrival times and level representations. The following are three challenges presented when using mobile devices to create a surround sound experience:

1) since each user has different types of mobile devices, non-identical loudspeakers are used;
2) users arbitrarily include devices in the system, not considering specific loudspeaker layouts;
3) users and their devices can freely move, even while multimedia content is being played.

A research group at KAIST, led by Professors Insik Shin and Junehwa Song, developed Mobile Maestro, a sound system that enables users to enjoy an immersive listening experience through the collective use of mobile devices, even in mobile sound environments. Mobile Ensemble is designed to conduct an adaptive, automated, and real-time loudspeaker calibration. The system captures how sounds are transferred from each mobile device to listeners by using acoustic signals. Based on the observations, it synchronizes both the sound arrival time and level and provides rich sounds to listeners. Furthermore, to cope with dynamic changes during playback, Mobile Ensemble regularly repeats the transmission and adjustment with inaudible acoustic signals. The research group showed that Mobile Ensemble maintains synchronization accuracy in sound arrival time in the order of tens of microsecond and holds a high degree of accuracy in sound level equalization in various mobile sound environments. Through a user study with 30 university participants, the research group also showed that Mobile Ensemble helps to create a much more immersive experience for music listening and movie watching.