AES Journal Forum

Layered Motion and Gesture Sonification in an Interactive Installation

Document Thumbnail

SoundThimble is an interactive sound installation based on the relationship between human motion and virtual objects in 3D space. A Vicon infrared motion-capture system and custom software are used to track, interpret, and sonify the movement and gestures of a performer relative to a virtual object. The authors explore the resulting possibilities for layered sonification dynamics and extended perception and expression in internal tests as well as in a public demo. Experimental evaluation reveals an average object search time of around 60 s, as well as thresholding ranges for effective gesture spotting. The underlying software platform is open source and portable to similar hardware systems, leaving room for extension and variation. This paper presents the pilot application of the proposed framework. Audience members entering the tracking area shift among the roles of game player, sonic performer, and composer/arranger, according to an iterative interaction schema. The central vehicle in all three layers is the “sound-thimble” itself, a virtual object with particular spatial, sonic, and interaction attributes.

JAES Volume 66 Issue 10 pp. 770-778; October 2018
Publication Date:

Click to purchase paper as a non-member or you can login as an AES member to see more options.

No AES members have commented on this paper yet.

Subscribe to this discussion

RSS Feed To be notified of new comments on this paper you can subscribe to this RSS feed. Forum users should login to see additional options.

Start a discussion!

If you would like to start a discussion about this paper and are an AES member then you can login here:

If you are not yet an AES member and have something important to say about this paper then we urge you to join the AES today and make your voice heard. You can join online today by clicking here.

AES - Audio Engineering Society