Abstract
Jamming requires coordination, anticipation, and collaborative creativitybetween musicians. Current generative models of music produce expressive outputbut are not able to generate in an \emph{online} manner, meaning simultaneouslywith other musicians (human or otherwise). We propose ReaLchords, an onlinegenerative model for improvising chord accompaniment to user melody. We startwith an online model pretrained by maximum likelihood, and use reinforcementlearning to finetune the model for online use. The finetuning objectiveleverages both a novel reward model that provides feedback on both harmonic andtemporal coherency between melody and chord, and a divergence term thatimplements a novel type of distillation from a teacher model that can see thefuture melody. Through quantitative experiments and listening tests, wedemonstrate that the resulting model adapts well to unfamiliar input andproduce fitting accompaniment. ReaLchords opens the door to live jamming, aswell as simultaneous co-creation in other modalities.