Abstract
Existing auto-regressive language models have demonstrated a remarkablecapability to perform a new task with just a few examples in prompt, withoutrequiring any additional training. In order to extend this capability to amulti-modal setting (i.e. speech and language), this paper introduces the Sealmodel, an abbreviation for speech language model. It incorporates a novelalignment method, in which Kullback-Leibler divergence loss is performed totrain a projector that bridges a frozen speech encoder with a frozen languagemodel decoder. The resulting Seal model exhibits robust performance as afew-shot learner on two speech understanding tasks. Additionally, consistencyexperiments are conducted to validate its robustness on different pre-trainedlanguage models.