- The paper identifies that relying on one prompt leads to significant variability in performance metrics, with some model-task pairs showing up to 45% coefficient of variation.
- The methodology employs 165 prompt-task pairs across 11 tasks to reveal how default prompts can inflate or deflate reported scores, undermining leaderboard integrity.
- It advocates for multi-prompt and distributional evaluation methods to ensure more reliable, reproducible benchmarking of instruction-tuned embedding models.
Instruction Sensitivity and the Reliability of Embedding Model Evaluations
Motivation and Problem Statement
Instruction-tuned embedding models have gained prominence as the backbone for diverse NLP applications, including semantic search, Retrieval-Augmented Generation (RAG), and agentic memory. Contemporary benchmarks, such as MTEB and MMTEB, evaluate these models via a single task-specific prompt, neglecting instruction phrasing variability. This oversight runs counter to the core nature of instruction-based models, which are by design sensitive to prompt wording. As a result, benchmark results might systematically under- or overstate true model quality, directly undermining cross-model comparability and leaderboard integrity.
Experimental Design
The paper conducts a systematic empirical assessment across 6 widely-used open-weight instruction-embedding models: Qwen3-Embedding-0.6B, multilingual-e5-large-instruct, KaLM-embedding-multilingual-mini-instruct-v2.5, and three variants of bge-en-v1.5 (small, base, large). For each of 11 English-only tasks spanning retrieval, classification, clustering, and semantic similarity, the study generates 15 task-specific prompts using LLM-driven synthesis (gpt-oss-120b), resulting in 165 distinct prompt-task pairs and 990 total evaluations. Prompt coherence and task relevance are strictly controlled, and synthetic-human prompt comparison checks validate the realism of the synthetic prompt set.
Key Findings and Results
Prompt Sensitivity and Distributional Effects
- The coefficient of variation (CV) of scores across prompts is nontrivial, with some model-task pairs reaching CVs as high as 30-45%. Median CVs fall in the moderate range (2-6%), but the tails are long, suggesting considerable prompt-induced variability for specific tasks and models.
- The default (reported) prompt frequently yields scores from the extreme ends (tails) of the full distribution. This manifests both as prompt deflation (reported score is a lower outlier that understates typical performance) and prompt inflation (reported score is a high outlier, overstating quality).
- The misalignment between reported (default prompt) and typical (distribution over reasonable prompts) performance is systematic, undermining the representativeness of single-point metrics commonly reported in benchmarks.
Impact on Leaderboards and Benchmark Robustness
- Adversarial prompt selection enables any of the evaluated models to attain the top position in simulated leaderboards. By assigning an optimized prompt to one model and suboptimal or default prompts to others, any model—regardless of its baseline position—can be promoted to first place.
- Even in milder, more realistic scenarios where a single model contributor selects their optimal prompt and others use their default, leaderboard positions shift materially (e.g., bge-large-en-v1.5 rises from rank 5 to rank 3).
- The study quantifies the probability of prompt inflation for benchmark scores, demonstrating that most reported scores fall above the distributional median, consistent with the prevalence of favorable prompt selection in current practice.
Prompt Hacking and Evaluation Integrity
The research identifies and explicitly names the phenomenon of prompt hacking: the selective reporting of a single favorable prompt without modifying model weights or training data. Analogous to p-hacking in scientific research, prompt hacking leverages latent degrees of freedom to inflate apparent performance while remaining compliant with standard benchmarking protocols.
Theoretical and Practical Implications
This study fundamentally challenges the robustness of current benchmarking methodologies for instruction-tuned embedding models:
- Benchmark Reliability: Single-prompt evaluation is statistically unsound for instruction-sensitive models and fails to provide a reliable indication of expected in-the-wild behavior.
- Leaderboard Integrity: The susceptibility of rankings to prompt selection—intentional or incidental—renders current model comparisons (and associated leaderboard claims) unreliable.
- Evaluation Best Practices: To support more faithful, reproducible, and robust model comparisons, the field must transition toward multi-prompt evaluation protocols. Reporting distributions, summary statistics (mean, median, variance), or explicit robustness metrics should become standard. Selective-limited prompt schemes, such as those implemented by the Jina v5 series, may serve as interim solutions.
Limitations and Directions for Future Work
The empirical scope is limited to 6 open-weight models (all ≤0.6B parameters) on English-only tasks; larger models and multilingual evaluation may reveal different sensitivity patterns. Prompt sets, while validated for realism, are synthetically generated and limited to 15 per task. Future work should extend prompt diversity (including more diverse human-authored prompts), scale up to larger closed models (e.g., OpenAI text-embedding-3), and rigorously quantify the trade-offs in multi-prompt evaluation. The utility of prompt sensitivity as a practitioner resource—for instance, through prompt search or ensembling strategies—deserves systematic exploration.
Conclusion
The findings demonstrate that the prevailing approach to benchmarking instruction-tuned embedding models via single prompt evaluation is insufficient and unreliable. Instruction sensitivity can lead to both under- and over-estimation of model quality, and enables manipulation of leaderboard positions through prompt selection alone. To ensure integrity in model evaluation and reporting, the adoption of multi-prompt and distributional evaluation schemes is imperative.
