Abstract
We design a suite of minimal algorithmic tasks that are a loose abstractionof open-ended real-world tasks. This allows us to cleanly and controllablyquantify the creative limits of the present-day language model. Much likereal-world tasks that require a creative, far-sighted leap of thought, ourtasks require an implicit, open-ended stochastic planning step that either (a)discovers new connections in an abstract knowledge graph (like in wordplay,drawing analogies, or research) or (b) constructs new patterns (like indesigning math problems or new proteins). In these tasks, we empirically andconceptually argue how next-token learning is myopic and memorizes excessively;comparatively, multi-token approaches, namely teacherless training anddiffusion models, excel in producing diverse and original output. Secondly, inour tasks, we find that to elicit randomness from the Transformer withouthurting coherence, it is better to inject noise right at the input layer (via amethod we dub hash-conditioning) rather than defer to temperature sampling fromthe output layer. Thus, our work offers a principled, minimal test-bed foranalyzing open-ended creative skills, and offers new arguments for going beyondnext-token learning and softmax-based sampling. We make part of the codeavailable under https://github.com/chenwu98/algorithmic-creativity