Abstract
We study compute efficiency of LLM training when using differentparameterizations, i.e., rules for adjusting model and optimizerhyperparameters (HPs) as model size changes. Some parameterizations fail totransfer optimal base HPs (such as learning rate) across changes in modeldepth, requiring practitioners to either re-tune these HPs as they scale up(expensive), or accept sub-optimal training when re-tuning is prohibitive. Evenwhen they achieve HP transfer, we develop theory to show parameterizations maystill exist in the lazy learning regime where layers learn only features closeto their linearization, preventing effective use of depth and nonlinearity.Finally, we identify and adopt the parameterization we call CompleteP thatachieves both depth-wise HP transfer and non-lazy learning in all layers.CompleteP enables a wider range of model width/depth ratios to remaincompute-efficient, unlocking shapes better suited for different hardwaresettings and operational contexts. Moreover, CompleteP enables 12-34% computeefficiency improvements over the prior state-of-the-art.