Facial motion capture is the process of electronically converting the movements of a person's face into a digital database using cameras or laser scanners.
A facial motion capture database describes the coordinates or relative positions of reference points on the actor's face.
It is a process of using visual or mechanical means to manipulate computer generated characters with input from human faces, or to recognize emotions from a user.
This technology is discussed and demonstrated at CMU,[2] IBM,[3] University of Manchester (where much of this started with Tim Cootes,[4] Gareth Edwards and Chris Taylor) and other locations, using active appearance models, principal component analysis, eigen tracking, deformable surface models and other techniques to track the desired facial features from frame to frame.
These vision based approaches also have the ability to track pupil movement, eyelids, teeth occlusion by the lips and tongue, which are obvious problems in most computer-animated features.
Conversely, systems such as deformable surface models pool temporal information to disambiguate and obtain more robust results, and thus could not be applied from a single photograph.
Markerless face tracking has progressed to commercial systems such as Image Metrics, which has been applied in movies such as The Matrix sequels[6] and The Curious Case of Benjamin Button.
Using digital cameras, the input user's expressions are processed to provide the head pose, which allows the software to then find the eyes, nose and mouth.
Like voice recognition, the best techniques are only good 90 percent of the time, requiring a great deal of tweaking by hand, or tolerance for errors.
Some animators create bones or objects that are controlled by the capture software, and move them accordingly, which when the character is rigged correctly gives a good approximation.
Two notable examples are Snapchat's lens feature and Apple's Memoji[9] that can be used to record messages with avatars or live via the FaceTime app.
Real-time communication applications prioritize low latency to facilitate natural conversation and ease of use, aiming to make the technology accessible to a broad audience.