Towards the design of a natural user interface for performing and learning musical gestures

Edgar Hemery (1) , Sotiris Manitsaris (1) , Fabien Moutarde (1) , Christina Volioti (2) , Athanasios Manitsaris (2)

1 CAOR - Centre de Robotique
2 Multimedia Technologies and Computer Graphics Laboratory

A large variety of musical instruments, either acoustical or digital, are based on a keyboard scheme. Keyboard instruments can produce sounds through acoustic means but they are increasingly used to control digital sound synthesis processes with nowadays music. Interestingly, with all the different possibilities of sonic outcomes, the input remains a musical gesture. In this paper we present the conceptualization of a Natural User Interface (NUI), named the Intangible Musical Instrument (IMI), aiming to support both learning of expert musical gestures and performing music as a unified user experience. The IMI is designed to recognize metaphors of pianistic gestures, focusing on subtle uses of fingers and upper-body. Based on a typology of musical gestures, a gesture vocabulary has been created, hierarchized from basic to complex. These piano-like gestures are finally recognized and transformed into sounds.

Domaines: Vision par ordinateur et reconnaissance de formes [cs.CV]

Edgar Hemery, Sotiris Manitsaris, Fabien Moutarde, Christina Volioti, Athanasios Manitsaris. Towards the design of a natural user interface for performing and learning musical gestures. Procedia Manufacturing, 2015, 3, pp.6329-6336. ⟨10.1016/j.promfg.2015.07.952⟩. ⟨hal-01256374⟩