As neural language models grow in effectiveness, they are increasingly beingapplied in real-world settings. However these applications tend to be limitedin the modes of interaction they support. In this extended abstract, we proposeWordcraft, an AI-assisted editor for story writing in which a writer and adialog system collaborate to write a story. Our novel interface uses few-shotlearning and the natural affordances of conversation to support a variety ofinteractions. Our editor provides a sandbox for writers to probe the boundariesof transformer-based language models and paves the way for futurehuman-in-the-loop training pipelines and novel evaluation methods.