Abstract
Learning to model and reconstruct humans in clothing is challenging due toarticulation, non-rigid deformation, and varying clothing types and topologies.To enable learning, the choice of representation is the key. Recent work usesneural networks to parameterize local surface elements. This approach captureslocally coherent geometry and non-planar details, can deal with varyingtopology, and does not require registered training data. However, naively usingsuch methods to model 3D clothed humans fails to capture fine-grained localdeformations and generalizes poorly. To address this, we present three keyinnovations: First, we deform surface elements based on a human body model suchthat large-scale deformations caused by articulation are explicitly separatedfrom topological changes and local clothing deformations. Second, we addressthe limitations of existing neural surface elements by regressing localgeometry from local features, significantly improving the expressiveness.Third, we learn a pose embedding on a 2D parameterization space that encodesposed body geometry, improving generalization to unseen poses by reducingnon-local spurious correlations. We demonstrate the efficacy of our surfacerepresentation by learning models of complex clothing from point clouds. Theclothing can change topology and deviate from the topology of the body. Oncelearned, we can animate previously unseen motions, producing high-quality pointclouds, from which we generate realistic images with neural rendering. Weassess the importance of each technical contribution and show that our approachoutperforms the state-of-the-art methods in terms of reconstruction accuracyand inference time. The code is available for research purposes athttps://qianlim.github.io/SCALE .