Stack-augmented recurrent neural networks (RNNs) have been of interest to thedeep learning community for some time. However, the difficulty of trainingmemory models remains a problem obstructing the widespread use of such models.In this paper, we propose the Ordered Memory architecture. Inspired by OrderedNeurons (Shen et al., 2018), we introduce a new attention-based mechanism anduse its cumulative probability to control the writing and erasing operation ofthe memory. We also introduce a new Gated Recursive Cell to compose lower-levelrepresentations into higher-level representation. We demonstrate that our modelachieves strong performance on the logical inference task (Bowman et al.,2015)and the ListOps (Nangia and Bowman, 2018) task. We can also interpret themodel to retrieve the induced tree structure, and find that these inducedstructures align with the ground truth. Finally, we evaluate our model on theStanford SentimentTreebank tasks (Socher et al., 2013), and find that itperforms comparatively with the state-of-the-art methods in the literature.