Infinite 3D Landmarks: Improving Continuous 2D FacialLandmark Detection

Abstract
In this paper, we examine 3 important issues in the practical use of state-of-the-art facial landmark detectors and show how
a combination of specific architectural modifications can directly improve their accuracy and temporal stability. First, many
facial landmark detectors require a face normalization step as a preprocess, often accomplished by a separately-trained neural
network that crops and resizes the face in the input image. There is no guarantee that this pre-trained network performs optimal
face normalization for the task of landmark detection. Thus, we instead analyze the use of a spatial transformer network that
is trained alongside the landmark detector in an unsupervised manner, jointly learning an optimal face normalization and
landmark detection by a single neural network. Second, we show that modifying the output head of the landmark predictor to
infer landmarks in a canonical 3D space rather than directly in 2D can further improve accuracy. To convert the predicted
3D landmarks into screen-space, we additionally predict the camera intrinsics and head pose from the input image. As a side
benefit, this allows to predict the 3D face shape from a given image only using 2D landmarks as supervision, which is useful
in determining landmark visibility among other things. Third, when training a landmark detector on multiple datasets at the
same time, annotation inconsistencies across datasets forces the network to produce a suboptimal average. We propose to add a
semantic correction network to address this issue. This additional lightweight neural network is trained alongside the landmark
detector, without requiring any additional supervision. While the insights of this paper can be applied to most common landmark
detectors, we specifically target a recently-proposed continuous 2D landmark detector to demonstrate how each of our additions
leads to meaningful improvements over the state-of-the-art on standard benchmarks.
Link to publication: studios.disneyresearch.com/20...