Abstract
Music-to-visual style transfer is a challenging yet important cross-modallearning problem in the practice of creativity. Its major difference from thetraditional image style transfer problem is that the style information isprovided by music rather than images. Assuming that musical features can beproperly mapped to visual contents through semantic links between the twodomains, we solve the music-to-visual style transfer problem in two steps:music visualization and style transfer. The music visualization networkutilizes an encoder-generator architecture with a conditional generativeadversarial network to generate image-based music representations from musicdata. This network is integrated with an image style transfer method toaccomplish the style transfer process. Experiments are conducted onWikiArt-IMSLP, a newly compiled dataset including Western music recordings andpaintings listed by decades. By utilizing such a label to learn the semanticconnection between paintings and music, we demonstrate that the proposedframework can generate diverse image style representations from a music piece,and these representations can unveil certain art forms of the same era.Subjective testing results also emphasize the role of the era label inimproving the perceptual quality on the compatibility between music and visualcontent.