Since language models are used to model a wide variety of languages, it isnatural to ask whether the neural architectures used for the task haveinductive biases towards modeling particular types of languages. Investigationof these biases has proved complicated due to the many variables that appear inthe experimental setup. Languages vary in many typological dimensions, and itis difficult to single out one or two to investigate without the others actingas confounders. We propose a novel method for investigating the inductivebiases of language models using artificial languages. These languages areconstructed to allow us to create parallel corpora across languages that differonly in the typological feature being investigated, such as word order. We thenuse them to train and test language models. This constitutes a fully controlledcausal framework, and demonstrates how grammar engineering can serve as auseful tool for analyzing neural models. Using this method, we find thatcommonly used neural architectures exhibit different inductive biases: LSTMsdisplay little preference with respect to word ordering, while transformersdisplay a clear preference for some orderings over others. Further, we findthat neither the inductive bias of the LSTM nor that of the transformer appearsto reflect any tendencies that we see in attested natural languages.