Abstract
This study investigates the use of reinforcement learning to guide a generalpurpose cache manager decisions. Cache managers directly impact the overallperformance of computer systems. They govern decisions about which objectsshould be cached, the duration they should be cached for, and decides on whichobjects to evict from the cache if it is full. These three decisions impactboth the cache hit rate and size of the storage that is needed to achieve thatcache hit rate. An optimal cache manager will avoid unnecessary operations,maximise the cache hit rate which results in fewer round trips to a slowerbackend storage system, and minimise the size of storage needed to achieve ahigh hit-rate. This project investigates using reinforcement learning in cache management bydesigning three separate agents for each of the cache manager tasks.Furthermore, the project investigates two advanced reinforcement learningarchitectures for multi-decision problems: a single multi-task agent and amulti-agent. We also introduce a framework to simplify the modelling ofcomputer systems problems as a reinforcement learning task. The frameworkabstracts delayed experiences observations and reward assignment in computersystems while providing a flexible way to scale to multiple agents. Simulation results based on an established database benchmark system showthat reinforcement learning agents can achieve a higher cache hit rate overheuristic driven algorithms while minimising the needed space. They are alsoable to adapt to a changing workload and dynamically adjust their cachingstrategy accordingly. The proposed cache manager model is generic andapplicable to other types of caches, such as file system caches. This projectis the first, to our knowledge, to model cache manager decisions as amulti-task control problem.