Abstract

Transformer-based LLMs demonstrate strong performance on graph reasoningtasks, yet their internal mechanisms remain underexplored. To uncover thesereasoning process mechanisms in a fundamental and unified view, we set thebasic decoder-only transformers and explain them using the circuit-tracerframework. Through this lens, we visualize reasoning traces and identify twocore mechanisms in graph reasoning: token merging and structural memorization,which underlie both path reasoning and substructure extraction tasks. Wefurther quantify these behaviors and analyze how they are influenced by graphdensity and model size. Our study provides a unified interpretability frameworkfor understanding structural reasoning in decoder-only Transformers.

 

Quick Read (beta)