Reinforcement Learning (RL) agents typically learn memorylesspolicies---policies that only consider the last observation when selectingactions. Learning memoryless policies is efficient and optimal in fullyobservable environments. However, some form of memory is necessary when RLagents are faced with partial observability. In this paper, we study alightweight approach to tackle partial observability in RL. We provide theagent with an external memory and additional actions to control what, ifanything, is written to the memory. At every step, the current memory state ispart of the agent's observation, and the agent selects a tuple of actions: oneaction that modifies the environment and another that modifies the memory. Whenthe external memory is sufficiently expressive, optimal memoryless policiesyield globally optimal solutions. Unfortunately, previous attempts to useexternal memory in the form of binary memory have produced poor results inpractice. Here, we investigate alternative forms of memory in support oflearning effective memoryless policies. Our novel forms of memory outperformbinary and LSTM-based memory in well-established partially observable domains.