Abstract
A persistent challenge in sign language video processing, including the taskof sign language to written language translation, is how we learnrepresentations of sign language in an effective and efficient way that canpreserve the important attributes of these languages, while remaining invariantto irrelevant visual differences. Informed by the nature and linguistics ofsigned languages, our proposed method focuses on just the most relevant partsin a signing video: the face, hands and body posture of the signer. However,instead of using pose estimation coordinates from off-the-shelf pose trackingmodels, which have inconsistent performance for hands and faces, we propose tolearn the complex handshapes and rich facial expressions of sign languages in aself-supervised fashion. Our approach is based on learning from individualframes (rather than video sequences) and is therefore much more efficient thanprior work on sign language pre-training. Compared to a recent model thatestablished a new state of the art in sign language translation on the How2Signdataset, our approach yields similar translation performance, using less than3\% of the compute.