Abstract
The relationship between muscle activity and resulting facial expressions iscrucial for various fields, including psychology, medicine, and entertainment.The synchronous recording of facial mimicry and muscular activity via surfaceelectromyography (sEMG) provides a unique window into these complex dynamics.Unfortunately, existing methods for facial analysis cannot handle electrodeocclusion, rendering them ineffective. Even with occlusion-free referenceimages of the same person, variations in expression intensity and execution areunmatchable. Our electromyography-informed facial expression reconstruction(EIFER) approach is a novel method to restore faces under sEMG occlusionfaithfully in an adversarial manner. We decouple facial geometry and visualappearance (e.g., skin texture, lighting, electrodes) by combining a 3DMorphable Model (3DMM) with neural unpaired image-to-image translation viareference recordings. Then, EIFER learns a bidirectional mapping between 3DMMexpression parameters and muscle activity, establishing correspondence betweenthe two domains. We validate the effectiveness of our approach throughexperiments on a dataset of synchronized sEMG recordings and facial mimicry,demonstrating faithful geometry and appearance reconstruction. Further, wesynthesize expressions based on muscle activity and how observed expressionscan predict dynamic muscle activity. Consequently, EIFER introduces a newparadigm for facial electromyography, which could be extended to other forms ofmulti-modal face recordings.