Embedding-based In-Context Prompt Training for Enhancing LLMs as Text Encoders

Abstract: LLMs have been widely explored for embedding generation. While recent studies show that in-context learning (ICL) effectively enhances the representational capability of LLMs by prepending a few task-related demonstrations, it causes substantial token overhead due to the increased sequence length. In this work, we propose EPIC, a novel embedding-based in-context prompt training strategy that leverages ICL to generate high-quality embeddings while reducing computational burden during both training and inference. This approach replaces discrete text demonstrations with their corresponding continuous embeddings, which not only encourages the LLM to align semantically-related text pairs during contrastive learning, but also requires the model to interpret demonstration embeddings as part of the in-context prompt. Consequently, EPIC-trained models achieve excellent embedding performance both with or without in-context prompts at inference time. Comprehensive experiments demonstrate that our method establishes new state-of-the-art results on the MTEB benchmark, surpassing frontier models trained solely on publicly available retrieval data. Extensive ablation studies further validate the effectiveness and necessity of our mechanism.

• The paper introduces EPIC, replacing textual demonstrations with continuous embeddings to reduce token overhead while maintaining strong semantic performance.
• It employs a lightweight MLP and contrastive InfoNCE loss with hard negative mining to ensure robust alignment in the shared latent space.
• Experimental results across 56 datasets show improved accuracy, up to 70% reduced latency, and strong generalization across various model scales.

Embedding-based In-Context Prompt Training for Efficient LLM Text Encoders

Motivation and Background

Text embedding models are pivotal in many NLP tasks requiring semantic representations, including retrieval, classification, clustering, and RAG frameworks. Recent work has repurposed LLMs as powerful embedding generators via contrastive fine-tuning. Techniques such as discrete in-context learning (ICL) demonstrate performance boosts but incur significant token overhead, impacting latency and resource utilization. This paper introduces EPIC, an embedding-based in-context prompt training method that replaces textual demonstrations in prompts with their continuous embedding vectors, leveraging LLM semantic capacity and reducing computational burden.

Figure 1: Illustration of embedding input strategies—EPIC replaces discrete demonstrations with embedding vectors, minimizing token usage.

EPIC Framework: Design and Instantiation

The EPIC architecture builds upon contrastive learning foundations but innovatively modifies prompt construction. Traditional models prepend the input with instructions (and optionally demonstration text pairs) for embedding tasks. EPIC substitutes demonstration text with their embedding representations, using a lightweight MLP layer for format alignment, and concatenates these embedding-based demonstrations with the user query. This design enforces the model to both interpret and produce embeddings in the shared latent space, enhancing semantic alignment and facilitating efficient prompt handling.

Figure 2: EPIC pipeline overview—training uses batch-sampled embedding pairs as demonstrations, which are reused at inference for enhanced embedding quality and efficiency.

During training, demonstration embeddings are randomly sampled from the batch and support both prompt-rich and prompt-free scenarios, robustifying inference for high-throughput retrieval systems. EPIC models are fine-tuned using InfoNCE loss with in-batch and hard negative mining, optimizing temperature-scaled cosine similarity between embeddings.

Experimental Evaluation

The paper presents comprehensive benchmarks on MTEB, using Qwen2.5-7B, Mistral-7B, and LLaMA-3.1-8B as base models. EPIC consistently establishes new state-of-the-art results compared to previous ICL and non-ICL strategies, improving the average score across 56 datasets. Crucially, EPIC-trained models outperform baselines even in the absence of in-context prompts during inference, indicating a strong representational effect beyond prompt conditioning.

Figure 3: Detailed comparison of EPIC and conventional ICL on Mistral-7B—EPIC matches discrete ICL performance with lower token and latency overhead.

EPIC achieves average improvements of up to 1.04 points over standard baselines and surpasses competitive models utilizing bidirectional attention and specialized pooling strategies. Ablation studies show that EPIC is robust across model scales (1B–8B), provides best trade-offs with two-vector demonstration compression, and outperforms both learnable token and soft-prompt strategies.

Figure 4: EPIC performance scales positively with model size (LLaMA-3.2-1B, 3B, 8B), delivering growing gains on larger models.

EPIC also reduces inference latency by up to 70% relative to discrete ICL, as demonstration embeddings are precomputed and reused.

Figure 5: EPIC significantly lowers average per-sample inference latency on MTEB datasets compared to traditional Mistral-7B with EOS pooling.

Strong Claims and Representational Effects

The paper claims and empirically verifies that EPIC-trained models consistently outperform standard contrastive baselines, including in the absence of prompt demonstrations, due to explicit training with and without prompts and the requirement for embedding interpretation. EPIC also alleviates the attention sink phenomenon, allowing the EOS token to aggregate semantic information more efficiently.

The approach contradicts prior bias toward discrete ICL, showing that continuous embeddings are sufficient for effective prompt augmentation, and further that bidirectional attention mechanisms, when combined with EPIC, degrade performance due to attention mismatches.

Practical and Theoretical Implications

EPIC advances embedding model efficiency for LLM architectures, facilitating scalable retrieval/ranking systems and RAG pipelines under strict latency and resource constraints. Its robust training strategy enables strong generalization both with and without prompts, confirming practical utility for high-throughput settings and retrieval-augmented LLM deployments.

From a theoretical perspective, EPIC offers new evidence for the interpretability and compositionality of in-context learning via embeddings, aligning with recent work in vector-based ICL (2605.01372). The necessity to mechanistically explain the emergence of task-relevant representations in embedding ICL remains open, an area ripe for further research.

Future Directions

Potential future developments include:

• Scaling EPIC to larger LLMs (e.g., 30B, 70B) to optimize embedding quality.
• Integrating proprietary/synthetic corpora for broader generalization testing.
• Mechanistic analysis of embedding ICL, elucidating latent representation dynamics.
• Extending EPIC to multilingual and cross-domain settings, leveraging its prompt and model-agnostic design.
• Mitigating inherent social biases and hallucination issues inherited from base LLMs, as noted in the ethical considerations.

Conclusion

EPIC represents a notable advance in in-context prompt engineering for LLM text encoders, combining semantic alignment and computational efficiency. It delivers strong empirical gains on standard benchmarks, robust behavior across prompt configurations, and reduced inference cost, offering practical tools and theoretical insight for future embedding model research in NLP (2605.01372).