Efficient Few-Shot Learning Without Prompts

  • 2022-09-22 15:48:11
  • Lewis Tunstall, Nils Reimers, Unso Eun Seo Jo, Luke Bates, Daniel Korat, Moshe Wasserblat, Oren Pereg
  • 32

Abstract

Recent few-shot methods, such as parameter-efficient fine-tuning (PEFT) andpattern exploiting training (PET), have achieved impressive results inlabel-scarce settings. However, they are difficult to employ since they aresubject to high variability from manually crafted prompts, and typicallyrequire billion-parameter language models to achieve high accuracy. To addressthese shortcomings, we propose SetFit (Sentence Transformer Fine-tuning), anefficient and prompt-free framework for few-shot fine-tuning of SentenceTransformers (ST). SetFit works by first fine-tuning a pretrained ST on a smallnumber of text pairs, in a contrastive Siamese manner. The resulting model isthen used to generate rich text embeddings, which are used to train aclassification head. This simple framework requires no prompts or verbalizers,and achieves high accuracy with orders of magnitude less parameters thanexisting techniques. Our experiments show that SetFit obtains comparableresults with PEFT and PET techniques, while being an order of magnitude fasterto train. We also show that SetFit can be applied in multilingual settings bysimply switching the ST body. Our code is available athttps://github.com/huggingface/setfit and our datasets athttps://huggingface.co/setfit .

 

Quick Read (beta)

loading the full paper ...