LLM2Vec-Gen: Response-Centric Text Embedding

• LLM2Vec-Gen is a self-supervised framework that transforms frozen decoder-only LLMs into robust text embedders by summarizing the model’s likely responses.
• It appends trainable thought and compression tokens to inputs and uses a dual objective (reconstruction and alignment) to distill latent representations from generated responses.
• The method achieves state-of-the-art improvements on MTEB benchmarks and demonstrates significant gains in safety and reasoning by capturing output-oriented semantics.

LLM2Vec-Gen is a self-supervised framework for converting a frozen decoder-only LLM into a text embedder by changing the target of representation from the input itself to the model’s likely response to that input. Rather than pooling token states to encode literal query semantics, it appends trainable latent suffix tokens, trains them to summarize the model’s own completion in fixed length, distills them toward an unsupervised embedding teacher’s representation of that completion, and uses the resulting latent states as embeddings. In the reported experiments, the method achieves state-of-the-art self-supervised performance on MTEB, improves over the best unsupervised embedding teacher by up to $9.3\%$, yields up to $43.2\%$ reduction in harmful content retrieval, and improves reasoning-intensive retrieval by up to $29.3\%$; the learned embeddings are also decodable into text for interpretability (BehnamGhader et al., 11 Mar 2026).

1. Response-centric representation and the input-output gap

The central claim of LLM2Vec-Gen is that many embedding tasks are not fundamentally input-centric. Standard embedders typically map a text to a vector meant to preserve the semantic content of that text itself, often by pooling hidden states over input tokens. That regime is effective when similarity in the downstream task tracks similarity in the literal input. The paper argues that this assumption fails for retrieval, clustering, classification, pair classification, and STS-like settings, because semantically diverse inputs may need to induce similar downstream outputs. The authors describe this mismatch as the input-output gap (BehnamGhader et al., 11 Mar 2026).

This formulation is motivated by cases in which the relevant semantics emerge only at the level of the anticipated answer. Two sentences with different surface form may imply the same class label; a query and a relevant document may differ lexically but align at the level of the answer they support; a harmful request, when processed by a safety-aligned assistant, may be more appropriately represented by the refusal it should trigger than by the malicious intent it expresses. In this view, embedding the raw query is not always the right objective. The proposed alternative is to embed the model’s potential response.

The paper positions this as an alternative to the dominant recipe of learning task-relevant equivalence classes from paired data via contrastive learning. That conventional route can require curated pairs, inherit biases from the pairing distribution, and collapse semantically distinct texts when they are treated as equally relevant. LLM2Vec-Gen instead derives supervision from unlabeled queries, the student LLM’s own completions, and an unsupervised teacher embedding of those completions. This preserves the self-supervised setting while directly targeting output-oriented semantics (BehnamGhader et al., 11 Mar 2026).

Relative to earlier LLM-based embedders, the conceptual distinction is sharp. LLM2Vec converts decoder-only LLMs into stronger text encoders through bidirectional attention, masked next token prediction, and unsupervised contrastive learning, but it remains input-centric (BehnamGhader et al., 2024). LLM2Vec-Gen retains the use of decoder-only backbones while redefining what the learned representation is supposed to summarize.

2. Architecture, latent tokens, and training objectives

LLM2Vec-Gen appends two kinds of trainable special tokens to each query: thought tokens $\mathtt{t}_1,\dots,\mathtt{t}_m$, described as an intermediate computational buffer, and compression tokens $\mathtt{c}_1,\dots,\mathtt{c}_n$, described as the actual latent response summary. For a query

$\mathbf{q}_i = (q_i^{(1)}, \dots, q_i^{(k)}),$

the model forms

$$\mathbf{x}_i = \mathbf{q}_i \oplus \mathtt{t}_{1:m} \oplus \mathtt{c}_{1:n}.$$

This sequence is processed by the frozen LLM, and the last-layer hidden states of the compression tokens are extracted: $$[\mathbf{h}_i^{1}, \dots, \mathbf{h}_i^{n}] = \text{LLM}(\mathbf{x}_i).$$ Those hidden states support two objectives (BehnamGhader et al., 11 Mar 2026).

The first is a reconstruction objective. The compression-token states are projected by $\text{MLP}_{\text{recon}}$ into latent prompt vectors $\mathbf{p}_i^{1}, \dots, \mathbf{p}_i^{n}$. These vectors are then used as soft prompts in a second forward pass of the same frozen LLM, which is trained to reconstruct the target response autoregressively. The paper interprets this as an information bottleneck: response content must be compressed into a small latent prompt set.

The second is an embedding alignment objective. An external embedding teacher $43.2\%$0 embeds the generated response $43.2\%$1, and the projected latent states are passed through $43.2\%$2, mean pooled, and trained to match the teacher embedding via squared error. The paper characterizes this as JEPA-like in the sense that the model predicts a semantic representation target rather than raw input tokens, although the target comes from an external frozen encoder rather than an internal target encoder (BehnamGhader et al., 11 Mar 2026).

The total loss is the sum of the reconstruction and alignment terms; the paper does not report non-unit weighting coefficients between them. At inference time, only one forward pass is needed: the query is suffixed with the special tokens, the compression-token hidden states are extracted, and the trained projection heads are applied to produce the final embedding. No response generation is required at inference.

A crucial implementation choice is that the LLM backbone remains frozen throughout training. The trainable parameters are only the embeddings of the newly introduced special tokens, $43.2\%$3, and $43.2\%$4. This design is central to the method’s claim of parameter-efficient adaptation.

3. Self-supervised pipeline, data, and implementation

The training pipeline uses only unlabeled queries. The reported setup samples 160K queries from the training split of the single-turn questions from the Tulu instruction-following dataset and explicitly does not use the ground-truth Tulu responses. For each query, the student LLM first generates a response $43.2\%$5. Training then uses two automatically produced signals: the LLM’s own completion and the unsupervised teacher’s embedding of that completion. In the paper’s terminology, this is the basis for calling the method self-supervised (BehnamGhader et al., 11 Mar 2026).

Teacher choice is not incidental. The embedding teacher is an unsupervised LLM2Vec model corresponding to the same backbone family when possible. The authors state two reasons for preferring unsupervised teachers: they are expected to more faithfully represent the semantic content of their inputs, and supervised contrastive encoders may distort the space toward relevance judgments and collapse distinct but equally relevant texts. Empirically, same-family teachers work best, while cross-family teachers degrade results (BehnamGhader et al., 11 Mar 2026).

The default token configuration is 10 thought tokens and 10 compression tokens, appended after the query. The method uses the last-layer hidden states of the compression tokens. Each projection head is a one-layer MLP with the same hidden dimension as the backbone LLM, and the output dimension of $43.2\%$6 matches the teacher embedding dimension.

The appendix reports the following training hyperparameters: AdamW; learning rate 3e-4 for Qwen-3 models and 5e-4 for Qwen-2.5 and Llama models; linear schedule with 100 warmup steps; batch size 32; 1 epoch over 160K samples; maximum sequence length 512 for queries and 512 for responses, with longer responses truncated; bfloat16 precision; and 2 × NVIDIA H100 80GB. Training time for Qwen3-8B is reported as about 3.5 hours. Because the backbone is frozen, the method is highly parameter efficient; for Qwen3-4B, only 13M parameters are trained (BehnamGhader et al., 11 Mar 2026).

A notable evaluation detail is that benchmark inputs are reformulated as generative instructions, such as “Generate text that answers this query,” “Generate text that is semantically similar to this text,” and “Summarize the following passage.” This matches the response-centric training philosophy.

4. Empirical performance on embedding, safety, and reasoning benchmarks

The primary general embedding evaluation is MTEB(eng, v2), comprising 41 tasks across 7 categories: bitext mining, classification, clustering, pair classification, reranking, retrieval, and STS. Ablations use MTEB-Lite. The main headline result is that LLM2Vec-Gen improves over the best unsupervised embedding teacher by up to 9.3% on MTEB. For Qwen-3-8B, LLM2Vec achieves 56.8 average and LLM2Vec-Gen reaches 62.1, which the paper reports as +9.3% relative improvement (BehnamGhader et al., 11 Mar 2026).

The gains are distributed unevenly across task families. On Qwen-3-1.7B, the average improves from 54.8 to 58.6 (+6.9%), with category changes including retrieval 34.9 → 37.0, reranking 39.9 → 44.9, clustering 41.1 → 49.3, classification 71.1 → 74.4, and STS 73.4 → 77.3, while pair classification declines slightly 76.4 → 75.6 and summarization declines 30.4 → 29.0. On Qwen-3-4B, the average changes from 56.8 to 59.9 (+5.5%), with strong improvements in reranking, clustering, classification, and STS, but retrieval declines 41.1 → 38.0 and summarization declines 31.1 → 28.5. On Qwen-3-8B, the average rises from 56.8 to 62.1, with particularly strong gains in clustering, classification, and STS, while retrieval declines slightly 42.7 → 42.2 (BehnamGhader et al., 11 Mar 2026).

This pattern is significant because the paper argues that clustering, classification, and STS are precisely the settings where diverse inputs should map to similar outputs. Retrieval is more mixed, especially for larger models. The authors attribute some of those retrieval declines to imperfect compression: the latent representation may capture the broad topic but miss fine-grained linking entities. Across families and sizes, the method consistently beats the corresponding LLM2Vec teacher by 1.1 to 5.3 points in absolute MTEB average, and the authors state that it closes over 60% of the gap to supervised methods (BehnamGhader et al., 11 Mar 2026).

Two additional benchmarks probe transfer of safety alignment and reasoning. On AdvBench-IR, which measures top-5 accuracy of retrieving a harmful passage from a corpus of 1,796 passages for 520 harmful queries, lower is better. The most striking result is for Qwen-3-1.7B, where harmful retrieval drops from 46.7 to 26.5, a 43.2% reduction. On BRIGHT, a zero-shot reasoning-intensive retrieval benchmark scored with nDCG@10, the largest gain is for Qwen-3-8B, improving from 14.9 to 19.3, which the paper reports as +29.3% (BehnamGhader et al., 11 Mar 2026).

Benchmark	Setting	Reported result
MTEB	Qwen-3-8B	56.8 → 62.1 (+9.3%)
AdvBench-IR	Qwen-3-1.7B	46.7 → 26.5 (-43.2%)
BRIGHT	Qwen-3-8B	14.9 → 19.3 (+29.3%)

5. Ablations, interpretability, and transfer mechanisms

The ablation study on Qwen-3-4B and MTEB-Lite clarifies which components matter. The full model scores 62.4. Using only $43.2\%$7 yields 41.8, whereas using only $43.2\%$8 yields 62.1. The reported interpretation is that alignment is the main driver of embedding quality, while reconstruction contributes little to MTEB score directly but is crucial for grounding and interpretability (BehnamGhader et al., 11 Mar 2026).

Token allocation has a smaller but measurable effect. The default 10 thought + 10 compression configuration reaches 62.4; 0 thought + 20 compression gives 62.0; and 19 thought + 1 compression gives 61.9. A separate analysis shows that increasing the total number of special tokens from 2 to 20 improves performance from 61.6 to 62.6, after which gains are marginal. This supports the claim that thought tokens provide a small benefit as an intermediate computational buffer, but the design is not extremely brittle.

The paper also studies response source, teacher choice, and adaptation strategy. Using the model’s own responses yields 62.4, compared with 61.8 for the original Tulu responses, 62.0 for Qwen3-8B responses, and 61.3 for Gemini-3-flash responses. Teacher compatibility is more important than raw teacher scale: a same-family LLM2Vec-Qwen-3-4B teacher gives 62.4, a stronger in-family LLM2Vec-Qwen-3-8B teacher gives 62.0, and a cross-family LLM2Vec-Llama-3.1-8B teacher drops to 59.7. With LoRA, r=8,\alpha=16 yields 63.0, while r=32,\alpha=64 yields 62.3; the frozen setup remains competitive (BehnamGhader et al., 11 Mar 2026).

Interpretability is one of the paper’s distinctive claims. Because the latent prompt vectors must support next-token prediction of the response, they remain on or near the LM’s natural language manifold. The authors provide two analysis mechanisms. First, the latent prompt vectors can be decoded into text through the same LM machinery used for reconstruction. Second, the final hidden states of compression tokens can be projected into vocabulary space using the pretrained LM head and inspected with a Logit Lens. In the reported examples, harmful prompts decode into refusal-style responses, factual prompts decode into answer-like content, and the top vocabulary items for unsafe queries are words such as “security” or “illegal” rather than harmful-action terms. Conversely, when the model is trained with only $43.2\%$9, the embeddings still work fairly well for retrieval but decode into nonsense. The paper uses this negative result to argue that reconstruction is what makes the embeddings human-readable (BehnamGhader et al., 11 Mar 2026).

The mechanisms behind the reported safety and reasoning transfer are stated explicitly. For safety, if the underlying LLM is safety aligned, then harmful inputs tend to produce refusal or safety-warning completions, and the embedding should summarize those completions rather than the malicious request. For reasoning, the relevant structure may emerge in the model’s completion rather than in the raw query; compressing the potential answer or reasoning trace into latent response tokens can therefore improve alignment with multi-step relevance. The empirical results on AdvBench-IR and BRIGHT are presented as supporting these interpretations.

6. Relation to adjacent work, misconceptions, and open problems

LLM2Vec-Gen sits within a broader line of work on extracting embeddings from generative models, but its representation target and training recipe are distinct. It differs from LLM2Vec, which improves decoder-only embedders through bidirectional attention, masked next token prediction, and unsupervised SimCSE while remaining input-centric (BehnamGhader et al., 2024). It also differs from HyDE, which is described as a generate-then-encode baseline: LLM2Vec-Gen shares the intuition that responses may be more useful than raw queries, but it avoids generation at inference time by internalizing the response into latent tokens (BehnamGhader et al., 11 Mar 2026).

A common misconception would be to treat the method as merely another form of soft prompting. The paper explicitly argues otherwise: soft prompts, prompt tuning, and prefix tuning learn latent vectors to condition or steer generation, whereas LLM2Vec-Gen trains latent suffix tokens specifically as a fixed-length response representation for embedding, then pools them into a single vector for retrieval and similarity tasks (BehnamGhader et al., 11 Mar 2026).

The method also invites comparison with decoder-native unified embedding systems such as GEM, which inserts special bottleneck tokens into a decoder-only LLM, manipulates attention masks so later tokens cannot directly access earlier text, and extracts hidden states at those bottleneck positions as embeddings while preserving autoregressive generation (Zhang et al., 4 Jun 2025). The resemblance is structural, but the objective differs: GEM compresses input text through a bottleneck inside a continued-training decoder, whereas LLM2Vec-Gen compresses the LLM’s own response into latent suffix tokens and keeps the backbone frozen. A plausible implication is that the two methods instantiate different answers to the same unification problem: whether embeddings should summarize the input directly or the response it induces.

Beyond text-only settings, xVLM2Vec shows that a generative multimodal embedder can be multilingualized through self-knowledge distillation on translated inputs, while preserving the original English embedding geometry (Musacchio et al., 12 Mar 2025). In domain-specific retrieval, Learn Before Represent proposes a generative-first bottleneck pipeline in which domain knowledge is acquired through constrained causal generation before contrastive refinement (Liang et al., 16 Jan 2026). These neighboring results suggest that response-centric compression, representational distillation, and bottlenecked generative learning are emerging as complementary design motifs rather than mutually exclusive alternatives.

The limitations identified in the LLM2Vec-Gen paper are substantial. Retrieval is not uniformly improved, especially for larger Qwen-3 models on MTEB. Performance depends on teacher quality and compatibility, with cross-family teachers degrading results. It also depends on response quality, since training uses the LLM’s own completions and the paper explicitly notes that response generator choice affects safety. The fixed latent bottleneck can be lossy, which likely contributes to fine-grained retrieval misses. Training requires two passes in the reconstruction-alignment setup, even though inference is single-pass. The supervised-teacher setting is described as less compelling, because LLM2Vec-Gen generally does not beat the supervised teacher unless LoRA is added. Finally, some implementation details remain unspecified, including explicit weighting coefficients beyond simple loss summation and details of final-embedding normalization (BehnamGhader et al., 11 Mar 2026).

The authors propose three open directions. Full JEPA mode would replace the external teacher with the same frozen LLM encoding the generated response. Latent chaining for fast reasoning would recursively feed compressed latents back into the model with fresh compression tokens to perform multi-step reasoning in latent space. Latent communication between agents would allow multi-agent systems to exchange fixed-length decodable latent embeddings instead of long natural-language messages. These proposals extend the paper’s core premise: the embedding need not represent what the input says, but what the LLM would say in response.