Towards a Hand Skeletal Model for Depth Images Applied to Capture Music-like Finger Gestures

Arnaud Dapogny (1) , Raoul de Charette (1) , Sotiris Manitsaris (1) , Fabien Moutarde (1) , Alina Glushkova (2)

1 CAOR - Centre de Robotique
2 Multimedia Technology and Computer Graphics Lab

The Intangible Cultural Heritage (ICH) implies gestural knowledge and skills in performing arts, such as music, and its preservation and transmission is a worldwide challenge according to UNESCO. This paper presents an ongoing research that aims at the development of a computer vision methodology for the recognition of music-like complex hand and finger gestures performed in space. This methodology can contribute both to the analysis of classical music playing schools, such as the European and the Russian, and to the finger gesture control of sound as a new interface for musical expression. An implementation of a generic method for building body subpart classification model applied in musical gestures is presented. A robust classification model from a reduced training dataset, as well as a method for spatial aggregation of the classification results, which provides a confidence measure on each hand subpart location is developed. A 80% pixel-wise classification accuracy and 95% ponctual subpart location accuracy is achieved when musical finger gestures with a semi-closed hand are performed in front of the camera and the rotation around camera axis is not too important.

Domaines: Vision par ordinateur et reconnaissance de formes [cs.CV]

Arnaud Dapogny, Raoul de Charette, Sotiris Manitsaris, Fabien Moutarde, Alina Glushkova. Towards a Hand Skeletal Model for Depth Images Applied to Capture Music-like Finger Gestures. 10th Int. Symposium on Computer Music Multidisciplinary Research (CMMR'2013), Oct 2013, Marseille, France. ⟨hal-00875721⟩