Abstract:
Broadband networks (Gigabit/s) allow transmission of large amounts of data in real time. They are an ideal conduit for bidirectional communication using high resolution multisensory content between collaborating participants. I will describe an immersive multisensory communication environment created at McGill University's Centre for Interdisciplinary Research in Music, Media and Technology that is used as a sophisticated scientific and creative space used to connect people and to experience music locally, and from a distance. Issues of testing and developing this environment, as well as review of possible applications will also be presented. We consider this to be the medium for next generation 3D multimedia.
Outline:
First two-way communication is described allowing for remote collaboration of musicians. Of particular relevance are acoustic feedback reduction, latency reduction in the transmission, high resolution audio and multisensory stimuli (audio, video and haptics). At the 117th AES convention in San Francisco a real-time audio/video internet streaming was demonstrated from Montreal to San Francisco. Also, a video is shown of a jam session in 2002 with remote telepresence between McGill university and Stanford university for which the audio latency was below 50ms. Vibration was essential to provide a compelling telepresence experience.
Today, a lab is created at McGill with a vibration platform, a 24-channel loudspeaker array including overhead loudspeakers. Elevation is necessary to transcribe the volume and power of sound. The 4x8 feet vibration platform hosts 3 chairs. Three 65'' plasma displays are used for video reproduction. Loudspeaker arrays add to immersiveness. The required multiple channels are provided by 3D spot microphone, room capture (microphone arrays distant to the instrument) or by upmixing systems based on adaptive filters.
One loudspeaker array reproduction technique that allows for a faithful sound reproduction is Wave Field Synthesis (WFS). One important aspect is the instruments' directivity, which is quite different from that of a loudspeaker. To this end, complex models have been derived for sound radiation caption and reproduction. As an example, a piano is recorded with several microphones, and the sound is reproduced by a set of loudspeakers carefully arranged to optimally reproduce the piano sound.
Experiments with capturing vibration from a soundstage and reproducing it on a motion platform have indicated its substantial relevance. Also, an automatic conversion of audio to vibration has been developed. The required synchrony of auditory and haptic stimuli to obtain perceptional fusion has been described. For bass, vibration was considered to be simultaneous with the audio if it was delayed relative to the audio. For percussion instruments, the opposite is true.
Finally, experiments are described on reproduction of reverberation on localization as well as a system that can stitch multiple camera signals into one video rendering in realtime.
Further information on Wieslaw Woszczyk, publications and the work on telepresence can be found at:
http://www.music.mcgill.ca/cirmmt/researchers/woszczyk.html
http://www.music.mcgill.ca/soundrecording/research.html
