Abstract
Score following is the process of tracking a musical performance (audio) withrespect to a known symbolic representation (a score). We start this paper byformulating score following as a multimodal Markov Decision Process, themathematical foundation for sequential decision making. Given this formaldefinition, we address the score following task with state-of-the-art deepreinforcement learning (RL) algorithms such as synchronous advantage actorcritic (A2C). In particular, we design multimodal RL agents that simultaneouslylearn to listen to music, read the scores from images of sheet music, andfollow the audio along in the sheet, in an end-to-end fashion. All thisbehavior is learned entirely from scratch, based on a weak and potentiallydelayed reward signal that indicates to the agent how close it is to thecorrect position in the score. Besides discussing the theoretical advantages ofthis learning paradigm, we show in experiments that it is in fact superiorcompared to previously proposed methods for score following in raw sheet musicimages.