Abstract
Foundation models are an emerging research direction in tabular deeplearning. Notably, TabPFNv2 recently claimed superior performance overtraditional GBDT-based methods on small-scale datasets using an in-contextlearning paradigm, which does not adapt model parameters to target datasets.However, the optimal finetuning approach for adapting tabular foundationalmodels, and how this adaptation reshapes their internal mechanisms, remainsunderexplored. While prior works studied finetuning for earlier foundationalmodels, inconsistent findings and TabPFNv2's unique architecture necessitatefresh investigation. To address these questions, we first systematicallyevaluate various finetuning strategies on diverse datasets. Our findingsestablish full finetuning as the most practical solution for TabPFNv2 in termsof time-efficiency and effectiveness. We then investigate how finetuning altersTabPFNv2's inner mechanisms, drawing an analogy to retrieval-augmented models.We reveal that the success of finetuning stems from the fact that aftergradient-based adaptation, the dot products of the query-representations oftest objects and the key-representations of in-context training objects moreaccurately reflect their target similarity. This improved similarity allowsfinetuned TabPFNv2 to better approximate target dependency by appropriatelyweighting relevant in-context samples, improving the retrieval-based predictionlogic. From the practical perspective, we managed to finetune TabPFNv2 ondatasets with up to 50K objects, observing performance improvements on almostall tasks. More precisely, on academic datasets with I.I.D. splits, finetuningallows TabPFNv2 to achieve state-of-the-art results, while on datasets withgradual temporal shifts and rich feature sets, TabPFNv2 is less stable andprior methods remain better.