Abstract
Language models are essentially probability distributions over tokensequences. Auto-regressive models generate sentences by iteratively computingand sampling from the distribution of the next token. This iterative samplingintroduces stochasticity, leading to the assumption that language models makeprobabilistic decisions, similar to sampling from unknown distributions.Building on this assumption, prior research has used simulated Gibbs sampling,inspired by experiments designed to elicit human priors, to infer the priors oflanguage models. In this paper, we revisit a critical question: Do languagemodels possess Bayesian brains? Our findings show that under certainconditions, language models can exhibit near-deterministic decision-making,such as producing maximum likelihood estimations, even with a non-zero samplingtemperature. This challenges the sampling assumption and undermines previousmethods for eliciting human-like priors. Furthermore, we demonstrate thatwithout proper scrutiny, a system with deterministic behavior undergoingsimulated Gibbs sampling can converge to a "false prior." To address this, wepropose a straightforward approach to distinguish between stochastic anddeterministic decision patterns in Gibbs sampling, helping to prevent theinference of misleading language model priors. We experiment on a variety oflarge language models to identify their decision patterns under variouscircumstances. Our results provide key insights in understanding decisionmaking of large language models.