Abstract
While in-context learning (ICL) has achieved remarkable success in naturallanguage and vision domains, its theoretical understanding-particularly in thecontext of structured geometric data-remains unexplored. This paper initiates atheoretical study of ICL for regression of H\"older functions on manifolds. Weestablish a novel connection between the attention mechanism and classicalkernel methods, demonstrating that transformers effectively performkernel-based prediction at a new query through its interaction with the prompt.This connection is validated by numerical experiments, revealing that thelearned query-prompt scores for H\"older functions are highly correlated withthe Gaussian kernel. Building on this insight, we derive generalization errorbounds in terms of the prompt length and the number of training tasks. When asufficient number of training tasks are observed, transformers give rise to theminimax regression rate of H\"older functions on manifolds, which scalesexponentially with the intrinsic dimension of the manifold, rather than theambient space dimension. Our result also characterizes how the generalizationerror scales with the number of training tasks, shedding light on thecomplexity of transformers as in-context kernel algorithm learners. Ourfindings provide foundational insights into the role of geometry in ICL andnovels tools to study ICL of nonlinear models.