Abstract
The rise of LLMs has deflected a growing portion of human-computerinteractions towards LLM-based chatbots. The remarkable abilities of thesemodels allow users to interact using long, diverse natural language textcovering a wide range of topics and styles. Phrasing these messages is a timeand effort consuming task, calling for an autocomplete solution to assistusers. We introduce the task of chatbot interaction autocomplete. We presentChaI-TeA: CHat InTEraction Autocomplete; An autcomplete evaluation frameworkfor LLM-based chatbot interactions. The framework includes a formal definitionof the task, coupled with suitable datasets and metrics. We use the frameworkto evaluate After formally defining the task along with suitable datasets andmetrics, we test 9 models on the defined auto completion task, finding thatwhile current off-the-shelf models perform fairly, there is still much room forimprovement, mainly in ranking of the generated suggestions. We provideinsights for practitioners working on this task and open new researchdirections for researchers in the field. We release our framework to serve as afoundation for future research.