Abstract
We introduce thoughts of words (ToW), a novel training-time data-augmentationmethod for next-word prediction. ToW views next-word prediction as a corereasoning task and injects fine-grained thoughts explaining what the next wordshould be and how it is related to the previous contexts in pre-training texts.Our formulation addresses two fundamental drawbacks of existing next-wordprediction learning schemes: they induce factual hallucination and areinefficient for models to learn the implicit reasoning processes in raw texts.While there are many ways to acquire such thoughts of words, we explore thefirst step of acquiring ToW annotations through distilling from larger models.After continual pre-training with only 70K ToW annotations, we effectivelyimprove models' reasoning performances by 7% to 9% on average and reduce modelhallucination by up to 10%. At the same time, ToW is entirely agnostic to tasksand applications, introducing no additional biases on labels or semantics.