Modeling the 3D shape of human faces is of prime concern when manipulating and editing faces in still images and video. When it comes to videos, we want to empower users to propagate their edits seamlessly across the entire video. To achieve this, we trained a model using self-supervision and differentiable rendering on real-world videos, enabling unparalleled accuracy and temporal stability.

This powerful model not only captures the 3D geometry of the neutral face shape but also regresses the blendshapes responsible for facial expressions. Additionally, it outputs vital elements such as normal maps, face skin albedo, diffuse lighting, and the 3D translation and rotation of the face. By accurately placing a 3D mesh with thousands of vertices on the face, our technology achieves precise mapping of 2D textures, enabling a wide range of face editing features in images and videos, including for example blemish removal and the addition of makeup.

What sets our solution apart is its remarkable efficiency. Everything is processed directly on the mobile phone using a lightweight model, ensuring that the video editing happens faster than the playback speed. With our innovative approach, we allow our users to edit and transform videos swiftly and effortlessly, unleashing your creativity without any technical constraints.