Transformers as Meta-Learners for Implicit Neural Representations

  • 2022-08-04 18:54:38
  • Yinbo Chen, Xiaolong Wang
  • 32

Abstract

Implicit Neural Representations (INRs) have emerged and shown their benefitsover discrete representations in recent years. However, fitting an INR to thegiven observations usually requires optimization with gradient descent fromscratch, which is inefficient and does not generalize well with sparseobservations. To address this problem, most of the prior works train ahypernetwork that generates a single vector to modulate the INR weights, wherethe single vector becomes an information bottleneck that limits thereconstruction precision of the output INR. Recent work shows that the wholeset of weights in INR can be precisely inferred without the single-vectorbottleneck by gradient-based meta-learning. Motivated by a generalizedformulation of gradient-based meta-learning, we propose a formulation that usesTransformers as hypernetworks for INRs, where it can directly build the wholeset of INR weights with Transformers specialized as set-to-set mapping. Wedemonstrate the effectiveness of our method for building INRs in differenttasks and domains, including 2D image regression and view synthesis for 3Dobjects. Our work draws connections between the Transformer hypernetworks andgradient-based meta-learning algorithms and we provide further analysis forunderstanding the generated INRs. The project page with code is at\url{https://yinboc.github.io/trans-inr/} .

 

Quick Read (beta)

loading the full paper ...