Abstract
Large language models (LLMs) have reshaped the landscape of programsynthesis. However, contemporary LLM-based code completion systems oftenhallucinate broken code because they lack appropriate context, particularlywhen working with definitions not in the training data nor near the cursor.This paper demonstrates that tight integration with the type and bindingstructure of a language, as exposed by its language server, can address thiscontextualization problem in a token-efficient manner. In short, we contendthat AIs need IDEs, too! In particular, we integrate LLM code generation intothe Hazel live program sketching environment. The Hazel Language Serveridentifies the type and typing context of the hole being filled, even in thepresence of errors, ensuring that a meaningful program sketch is alwaysavailable. This allows prompting with codebase-wide contextual information notlexically local to the cursor, nor necessarily in the same file, but that islikely to be semantically local to the developer's goal. Completionssynthesized by the LLM are then iteratively refined via further dialog with thelanguage server. To evaluate these techniques, we introduce MVUBench, a datasetof model-view-update (MVU) web applications. These applications serve aschallenge problems due to their reliance on application-specific datastructures. We find that contextualization with type definitions isparticularly impactful. After introducing our ideas in the context of Hazel weduplicate our techniques and port MVUBench to TypeScript in order to validatethe applicability of these methods to higher-resource languages. Finally, weoutline ChatLSP, a conservative extension to the Language Server Protocol (LSP)that language servers can implement to expose capabilities that AI codecompletion systems of various designs can use to incorporate static contextwhen generating prompts for an LLM.