Abstract
Transformer-based language models have set new benchmarks across a wide rangeof NLP tasks, yet reliably estimating the uncertainty of their predictionsremains a significant challenge. Existing uncertainty estimation (UE)techniques often fall short in classification tasks, either offering minimalimprovements over basic heuristics or relying on costly ensemble models.Moreover, attempts to leverage common embeddings for UE in linear probingscenarios have yielded only modest gains, indicating that alternative modelcomponents should be explored. We tackle these limitations by harnessing the geometry of attention mapsacross multiple heads and layers to assess model confidence. Our approachextracts topological features from attention matrices, providing alow-dimensional, interpretable representation of the model's internal dynamics.Additionally, we introduce topological features to compare attention patternsacross heads and layers. Our method significantly outperforms existing UEtechniques on benchmarks for acceptability judgments and artificial textdetection, offering a more efficient and interpretable solution for uncertaintyestimation in large-scale language models.