Abstract
Decoder-only large language model (LLM)-based embedding models are beginningto outperform BERT or T5-based embedding models in general-purpose textembedding tasks, including dense vector-based retrieval. In this work, weintroduce the NV-Embed model with a variety of architectural designs andtraining procedures to significantly enhance the performance of LLM as aversatile embedding model, while maintaining its simplicity andreproducibility. For model architecture, we propose a latent attention layer toobtain pooled embeddings, which consistently improves retrieval and downstreamtask accuracy compared to mean pooling or using the last <EOS> token embeddingfrom LLMs. To enhance representation learning, we remove the causal attentionmask of LLMs during contrastive training. For model training, we introduce atwo-stage contrastive instruction-tuning method. It first applies contrastivetraining with instructions on retrieval datasets, utilizing in-batch negativesand curated hard negative examples. At stage-2, it blends various non-retrievaldatasets into instruction tuning, which not only enhances non-retrieval taskaccuracy but also improves retrieval performance. Combining these techniques,our NV-Embed model, using only publicly available data, has achieved arecord-high score of 69.32, ranking No. 1 on the Massive Text EmbeddingBenchmark (MTEB) (as of May 24, 2024), with 56 tasks, encompassing retrieval,reranking, classification, clustering, and semantic textual similarity tasks.Notably, our model also attains the highest score of 59.36 on 15 retrievaltasks in the MTEB benchmark (also known as BEIR). We will open-source the modelat: https://huggingface.co/nvidia/NV-Embed-v1.