Abstract
We present Piano Genie, an intelligent controller which allows non-musiciansto improvise on the piano. With Piano Genie, a user performs on a simpleinterface with eight buttons, and their performance is decoded into the spaceof plausible piano music in real time. To learn a suitable mapping procedurefor this problem, we train recurrent neural network autoencoders with discretebottlenecks: an encoder learns an appropriate sequence of buttons correspondingto a piano piece, and a decoder learns to map this sequence back to theoriginal piece. During performance, we substitute a user's input for theencoder output, and play the decoder's prediction each time the user presses abutton. To improve the interpretability of Piano Genie's performance mechanics,we impose musically-salient constraints over the encoder's outputs.