Abstract
Market making is a fundamental trading problem in which an agent providesliquidity by continually offering to buy and sell a security. The problem ischallenging due to inventory risk, the risk of accumulating an unfavourableposition and ultimately losing money. In this paper, we develop a high-fidelitysimulation of limit order book markets, and use it to design a market makingagent using temporal-difference reinforcement learning. We use a linearcombination of tile codings as a value function approximator, and design acustom reward function that controls inventory risk. We demonstrate theeffectiveness of our approach by showing that our agent outperforms both simplebenchmark strategies and a recent online learning approach from the literature.