Abstract
The meaning conveyed by a sentence often depends on the context in which itappears. Despite the progress of sentence embedding methods, it remains unclearhow to best modify a sentence embedding conditioned on its context. To addressthis problem, we propose Condition-Aware Sentence Embeddings (CASE), anefficient and accurate method to create an embedding for a sentence under agiven condition. First, CASE creates an embedding for the condition using aLarge Language Model (LLM), where the sentence influences the attention scorescomputed for the tokens in the condition during pooling. Next, a supervisednonlinear projection is learned to reduce the dimensionality of the LLM-basedtext embeddings. We show that CASE significantly outperforms previouslyproposed Conditional Semantic Textual Similarity (C-STS) methods on an existingstandard benchmark dataset. We find that subtracting the condition embeddingconsistently improves the C-STS performance of LLM-based text embeddings.Moreover, we propose a supervised dimensionality reduction method that not onlyreduces the dimensionality of LLM-based embeddings but also significantlyimproves their performance.