Lightweight Palm and Finger Tracking for Real-Time 3D Gesture Control



The following video showcases the freehand 3D gesture control prototype in action. Currently implemented gestures are introduced one by one before applying the gestures to a basic spatial arrangement task.

Paper (IEEE VR 2011)


We present a novel technique implementing barehanded interaction with virtual 3D content by employing a time-of-flight camera. The system improves on existing 3D multi-touch systems by working regardless of lighting conditions and supplying a working volume large enough for multiple users. Previous systems were limited either by environmental requirements, working volume, or computational resources necessary for realtime operation. By employing a time-of-flight camera, the system is capable of reliably recognizing gestures at the finger level in real-time at more than 50 fps with commodity computer hardware using our newly developed precision hand and finger-tracking algorithm. Building on this algorithm, the system performs gesture recognition with simple constraint modeling over statistical aggregations of the hand appearances in a working volume of more than 8 cubic meters. Two iterations of user tests were performed on a prototype system, demonstrating the feasibility and usability of the approach as well as providing first insights regarding the acceptance of true barehanded touch-based 3D interaction.


author    = {Hackenberg, G. and McCall, R. and Broll, W.}, 
booktitle = {Virtual Reality Conference (VR), 2011 IEEE},
title     = {Lightweight palm and finger tracking for real-time 3D gesture control}, 
year      = {2011}, 
month     = {march}, 
pages     = {19 -26}, 
keywords  = {3D multitouch systems;finger tracking;gesture recognition;lightweight palm;real-time 3D gesture control;statistical aggregations;time-of-flight camera;virtual 3D content;gesture recognition;statistical analysis;}, 
doi       = {10.1109/VR.2011.5759431}, 
ISSN      = {1087-8270},}


The following presentation were used at different occasions to introduce the project and summarize research findings respectively development progress. The left presentation was used at the IEEE VR 2011 Conference in Singapore. The right presentation was used at the yearly group seminar of the Chair IV: Software and Systems Engineering of the Technical University of Munich.