IMU2Face: Real-time Gesture-driven Facial Reenactment (1801.01446v1)

Abstract: We present IMU2Face, a gesture-driven facial reenactment system. To this end, we combine recent advances in facial motion capture and inertial measurement units (IMUs) to control the facial expressions of a person in a target video based on intuitive hand gestures. IMUs are omnipresent, since modern smart-phones, smart-watches and drones integrate such sensors, e.g., for changing the orientation of the screen content, counting steps, or for flight stabilization. Face tracking and reenactment is based on the state-of-the-art real-time Face2Face facial reenactment system. Instead of transferring facial expressions from a source to a target actor, we employ an IMU to track the hand gestures of a source actor and use its orientation to modify the target actor's expressions.

• The paper presents a novel system that uses IMU data from hand gestures to control facial reenactment through optimized blendshape weights.
• It integrates established Face2Face techniques with high-rate IMU data processed by flight controller software for smooth, real-time expression transfer.
• The work demonstrates practical applications in VR, digital communication, and animation, highlighting the versatile potential of IMUs.

IMU2Face: Real-time Gesture-driven Facial Reenactment

The paper "IMU2Face: Real-time Gesture-driven Facial Reenactment" authored by Justus Thies, Michael Zollhöfer, and Matthias Nießner, presents a novel system leveraging inertial measurement units (IMUs) for facial reenactment. This work builds upon existing technologies by integrating IMUs to facilitate control of facial expressions in a target video through intuitive hand gestures. This approach reimagines the application of IMUs - commonly found in smartphones, smartwatches, and drones - to influence digital facial expression in real-time video contexts.

Overview and Methodology

IMU2Face extends the capabilities of the real-time Face2Face system, previously developed for facial reenactment in color videos sourced from the web. Whereas traditional systems emphasize transferring expressions between individuals, IMU2Face uniquely uses hand gestures tracked by IMUs to alter facial expressions based on the orientation data of these sensors. The structure of this approach involves analyzing the target actor's facial geometry using established methods from Face2Face, followed by editing these expressions based on the data obtained from an IMU attached to the hand of the source actor.

The process of transferring motion is conducted by determining expression blendshape weights that accurately reflect the orientation provided by the IMU, particularly focusing on the jaw motion. This is achieved through minimizing deformation transfer energy in the jaw region, which is computed by leveraging the orientation of the IMU sensor—representing a direct correlation between hand movement and facial expression alteration.

Technical Details

Key to the system is the real-time analysis and adaptation enabled by the flight controller software, BetaFlight. This software manages the high-rate integration and filtering of IMU data before streaming it to a desktop environment where the reenactment computation occurs. The linear system involved in blending expressions is solved in real time, ensuring that the reenactment remains smooth and responsive to the source actor’s gestures.

Implications and Potential Developments

The implications of IMU2Face are manifold, particularly in the areas of digital communication, entertainment, and virtual reality (VR). The system’s reliance on widely available technology (IMUs) makes it an accessible tool for various applications, from enhancing teleconferencing in VR by addressing occlusion issues to possibly enabling new forms of storytelling and character animation in the digital arts. Moreover, this approach underscores the versatility of IMUs in motion capture beyond their traditional uses.

From a broader perspective, the authors suggest that the principles underlying IMU2Face could be extended to drive other forms of action and reenactment beyond facial expressions. This aligns with ongoing research exploring the convergence of sensor technology and real-time computational graphics, potentially paving the way for increasingly immersive and interactive digital environments.

Conclusion

IMU2Face represents a significant contribution to the field of facial reenactment, showcasing the novel application of IMU technology in transforming how expressions are manipulated and transferred. This system opens avenues for future research to explore how similar methodologies could be adapted for broader sets of actions, thereby enriching the toolset available for creating dynamic, gesture-driven digital experiences.Quantitative evaluation of the system’s performance, accuracy, and user acceptance will be critical for its adoption and further development in practical scenarios.