Abstract
Recent large language models (LLMs) have demonstrated exceptional performanceon general-purpose text embedding tasks. While dense embeddings have dominatedrelated research, we introduce the first Lexicon-based EmbeddiNgS (LENS)leveraging LLMs that achieve competitive performance on these tasks. Regardingthe inherent tokenization redundancy issue and unidirectional attentionlimitations in traditional causal LLMs, LENS consolidates the vocabulary spacethrough token embedding clustering, and investigates bidirectional attentionand various pooling strategies. Specifically, LENS simplifies lexicon matchingby assigning each dimension to a specific token cluster, where semanticallysimilar tokens are grouped together, and unlocking the full potential of LLMsthrough bidirectional attention. Extensive experiments demonstrate that LENSoutperforms dense embeddings on the Massive Text Embedding Benchmark (MTEB),delivering compact feature representations that match the sizes of densecounterparts. Notably, combining LENSE with dense embeddings achievesstate-of-the-art performance on the retrieval subset of MTEB (i.e. BEIR).