Abstract
In this paper, we propose methods to build a powerful and efficientImage-to-Speech captioning (Im2Sp) model. To this end, we start with importingthe rich knowledge related to image comprehension and language modeling from alarge-scale pre-trained vision-language model into Im2Sp. We set the output ofthe proposed Im2Sp as discretized speech units, i.e., the quantized speechfeatures of a self-supervised speech model. The speech units mainly containlinguistic information while suppressing other characteristics of speech. Thisallows us to incorporate the language modeling capability of the pre-trainedvision-language model into the spoken language modeling of Im2Sp. With thevision-language pre-training strategy, we set new state-of-the-art Im2Spperformances on two widely used benchmark databases, COCO and Flickr8k. Then,we further improve the efficiency of the Im2Sp model. Similar to the speechunit case, we convert the original image into image units, which are derivedthrough vector quantization of the raw image. With these image units, we candrastically reduce the required data storage for saving image data to just 0.8%when compared to the original image data in terms of bits. Demo page:https://ms-dot-k.github.io/Image-to-Speech-Captioning.