Instruction-Tuned Embedding Models

• Instruction-tuned embedding models are representation learners that incorporate natural language instructions to generate task-specific vector embeddings from various input types.
• They utilize architectures like shared text encoders and decoder-only extractors to pool token-level features, ensuring that the embeddings reflect the intended task or domain semantics.
• Training regimes combine contrastive, generative, and hybrid objectives, enabling robust performance across retrieval, classification, multimodal modeling, and safety evaluation.

Instruction-tuned embedding models are representation learners in which the embedding is explicitly conditioned on a natural-language instruction that specifies task, domain, or intended similarity relation. In the now-standard formulation, the model computes $f_\theta(x,p)\in\mathbb{R}^d$, where $x$ is the input and $p$ is the prompt or instruction; the same text, graph, image, user profile, video, or audio clip can therefore receive different vector representations under different instructions (Kostiuk et al., 21 May 2026). Across the literature, this idea appears as a “single embedder” that can generate text embeddings tailored to different downstream tasks and domains, as answer-conditioned embeddings derived from generated responses, as joint graph-language or user-language embeddings projected into an LLM space, and as lightweight instruction-specific transformations of pre-computed generic vectors (Su et al., 2022, Peng et al., 2024, Haag et al., 2024, Feng et al., 30 May 2025). The field has consequently expanded from instruction-conditioned text similarity to retrieval, structured data, multimodal modeling, and even brain-alignment and safety analyses of hidden-state geometry (Zhuang et al., 27 May 2025, Oota et al., 9 Jun 2025, Zeng et al., 2024).

1. Conceptual foundations

At its core, an instruction-tuned embedding model treats the instruction as part of the input space rather than as metadata. INSTRUCTOR makes this explicit with a shared encoder that embeds $I\oplus x$, where $I$ is a task-and-domain description and $x$ is the text, and then mean-pools the token representations corresponding to the text portion (Su et al., 2022). The same design principle generalizes to the broader view formalized in prompt-sensitivity work: the embedding function is $f_\theta(x,p)$, and downstream scoring is defined over embeddings that are conditional on prompt choice rather than fixed once and for all (Kostiuk et al., 21 May 2026).

This conditioning changes what “similarity” means. In one line of work, instruction tuning is used to produce embeddings tailored to retrieval, classification, clustering, semantic textual similarity, reranking, and summarization from a single model; in another, “instruction embedding” is defined more narrowly as “a specialized subset of text embedding that prioritizes task identification for instructions over the extraction of sentence-level semantic information” (Choi et al., 9 Jun 2025, Li et al., 2024). The latter distinction is consequential: instructions that are semantically similar may encode different tasks, while surface-diverse instructions may encode the same task category. This suggests that instruction-tuned embedding models are not merely general semantic encoders with longer prompts, but systems whose geometry is meant to reflect task semantics, output format, or user intent.

A second conceptual shift concerns where the task signal is expected to live. In INSTRUCTOR, the instruction is prepended and the encoder is trained contrastively across 330 datasets, including both symmetric and asymmetric tasks, so that task descriptions disambiguate otherwise conflicting supervision (Su et al., 2022). In InBedder, by contrast, the instruction is treated as a question about the input, and the embedding is derived from the model’s answer-side states or from a re-encoding of generated answers; the claim is that texts with the same implicit semantics under an instruction should yield similar answers and therefore similar embeddings (Peng et al., 2024). These two formulations—embed the instructed input directly, or embed the answer implied by the instruction—anchor much of the subsequent design space.

2. Architectural patterns and embedding extraction

The most common architecture is the shared text encoder. INSTRUCTOR uses GTR/T5 encoders and computes

$$f_\theta(I,x)=\mathrm{MeanPool}_x(\mathrm{Encoder}_\theta(I\oplus x)),$$

with pooling restricted to the text tokens rather than the instruction tokens (Su et al., 2022). Decoder-only embedders use a different extraction rule. The LG-AI-Embedding report constructs a prompt from few-shot examples, a task instruction, the target query, and an [EOS] token, then extracts the embedding from the final token representation $e=h_T$ at the [EOS] position (Choi et al., 9 Jun 2025). Work on causal LLM classification adopts the same general idea, using the final token embedding $H_T$ as a holistic sequence representation “analogous to BERT’s [CLS] embedding” (Yousefiramandi et al., 14 Dec 2025).

A second pattern is answer-conditioned extraction. InBedder studies several decoder-side pooling schemes over generated output tokens and finds that, after QA-style tuning, the hidden state at the position that predicts the first generated token—its “1st-gen” representation—is the strongest instruction-aware embedding for the LLaMA-2-7B InBedder model, whereas for a generic chat LLM re-encoding the generated answer with a sentence transformer is more effective (Peng et al., 2024). Multimodal instruction-specific embeddings follow the same decoder-side logic. In the video-audio setting, the feature for a layer $x$0 is

$x$1

the average of hidden states of all generated tokens for clip $x$2 under instruction $x$3 (Oota et al., 9 Jun 2025). The image-based brain-alignment study uses the same principle, averaging generated-token hidden states layerwise and then reducing them with PCA (Oota et al., 26 May 2025).

Instruction tuning has also been extended to non-text modalities through projection into an LLM embedding space. GraphLLaVA encodes a graph $x$4 with a graph transformer, mean-pools node states into $x$5, projects this with a two-layer MLP $x$6, and concatenates the result with the instruction embeddings before decoder-side generation: $x$7 The graph encoder is query-agnostic, but the combined graph-plus-instruction sequence is instruction-sensitive at the LLM level (Haag et al., 2024). A related multimodal pattern appears in InstructUE, where modality-specific encoders and a lightweight adapter map transactions, logs, search, and tabular data into the token space of Qwen2.5-0.5B-Instruct, and the final user embedding is the last-layer hidden state at a special <USER> token (Gao et al., 13 Oct 2025).

A different architectural response to instruction dependence is to avoid re-encoding entirely. GSTransform starts from a generic embedding $x$8, constructs instruction-specific labels on a small sample via LLM-guided summarization, clustering, and label assignment, and trains a linear encoder–decoder transformation over the stored vectors. The instruction is therefore used offline to produce an instruction-specific transformed space, while inference applies only a lightweight linear map to pre-computed embeddings (Feng et al., 30 May 2025).

3. Training objectives and supervision regimes

The dominant training regime remains contrastive multi-task learning. INSTRUCTOR trains on the MEDI mixture with a cosine-based in-batch softmax objective over positive and negative pairs, using task and domain descriptions as instructions; an important empirical result is that combining symmetric and asymmetric datasets hurts without instructions but improves with them, which the paper interprets as evidence that instructions mitigate interference in universal embedding training (Su et al., 2022). LG-AI-Embedding uses a decoder-only Mistral-7B backbone but keeps the same contrastive core, combining instruction prompts and few-shot examples with InfoNCE-style learning, teacher-derived continuous relevance scores, and adaptive margin-based hard-negative mining (Choi et al., 9 Jun 2025).

A second regime uses generative supervision rather than pairwise contrast. InBedder fine-tunes LLMs only on abstractive question answering: the passage becomes the input, the question becomes the instruction, and the answer becomes the target. The embedding objective is therefore implicit in an autoregressive or masked-language-model loss rather than an explicit similarity loss (Peng et al., 2024). GraphLLaVA uses a two-stage generative procedure: first it trains only the projection $x$9 so that graph embeddings land in a usable region of the LLM embedding space via graph-description instructions, and then it instruction-tunes the LLM with LoRA on actual graph questions (Haag et al., 2024). Unsupervised dense retrieval via instruction-tuning follows a comparable sequence: generate synthetic keyword and question responses for each corpus passage, filter them, instruction-tune FLAN-T5 on these self-generated pairs, and then use the tuned encoder for zero-shot dense retrieval (Zeng et al., 2024).

Hybrid objectives that tie language space and representation space are increasingly common. InstructUE combines autoregressive answer generation with a contrastive objective aligning user embeddings and answer embeddings, optimizing

$p$0

with $p$1 in experiments (Gao et al., 13 Oct 2025). This design explicitly uses answers as a semantic bottleneck: similar users under the same instruction should induce similar answer embeddings and therefore similar user representations. A related hybridization appears in instruction-following retrieval. InF-Embed defines an instruction-aware query representation $p$2 from instruction $p$3 and query $p$4, scores passages by $p$5, and trains with multivariate contrastive losses over passages, instructions, and instruction-aware queries using marginal negative sampling (Zhuang et al., 27 May 2025).

Post-hoc transformation methods occupy a distinct corner of the design space. GSTransform does not learn a universal $p$6; instead it learns an instruction-specific encoder–decoder with a supervised contrastive term and a reconstruction term over generic embeddings, using instruction-focused labels derived from an LLM on a small sample (Feng et al., 30 May 2025). This suggests a practical alternative to repeated corpus re-encoding when the latent information is already present in a generic space.

4. Application regimes and domain-specific variants

Instruction-tuned embedding models are now deployed across retrieval, classification, structured data, and user modeling. In retrieval, InF-IR expands traditional $p$7query, passage$p$8 supervision into 38,759 positive $p$9instruction, query, passage$I\oplus x$0 triplets and 77,518 hard negatives, then trains InF-Embed to optimize direct embedding-based retrieval rather than only reranking. Across five instruction-based retrieval benchmarks, the resulting models surpass competitive baselines by 8.1% in p-MRR, a metric designed to capture instruction-following ranking behavior (Zhuang et al., 27 May 2025). A different retrieval-oriented route is unsupervised instruction-tuning of FLAN-T5, where keyword summarization and question generation instructions are used to produce synthetic queries and a Rao–Blackwell-style weighted corpus embedding

$I\oplus x$1

which improves average zero-shot retrieval performance and outperforms several supervised dense retrievers with a smaller model (Zeng et al., 2024).

General-purpose text embedding remains a major target. The LG-AI-Embedding report evaluates a unified instruction-based Mistral-7B system on MTEB (English, v2) with 41 tasks across seven categories and reports 38.93 on summarization, the best score in its comparison table, with an overall mean of 74.12 and a stronger Borda ranking than some higher-mean baselines (Choi et al., 9 Jun 2025). For large-scale deployment, GSTransform attacks the main systems bottleneck—re-encoding a corpus for each instruction—by transforming stored vectors instead. On three instruction-awareness downstream tasks across nine datasets, it reports 6~300x real-time speedups while improving instruction-following embedding quality over state-of-the-art methods (Feng et al., 30 May 2025).

Structured-data and user-centric applications extend the same principle beyond plain text. GraphLLaVA projects permutation-invariant graph representations into the LLM embedding space and significantly outperforms a graph-to-text TinyLlama baseline, with overall yes/no accuracy 62.90% versus 44.41%, while remaining above a 50.11% random baseline and retaining performance on larger graphs where textual graph descriptions exceed about 400 tokens (Haag et al., 2024). InstructUE builds instruction-aware user embeddings from six heterogeneous sources, and its instruction-tuned variant achieves the best AUC/KS on 5/6 real-world evaluation tasks, while also showing that manual instructions can help or hurt depending on alignment with business semantics (Gao et al., 13 Oct 2025). Under resource constraints, classification-specific evidence complicates a simple “instruction-tuning is always best” narrative: for two patent datasets, attaching a classifier head to the final token embedding of a causal LLM significantly outperforms prompt→response instruction tuning in F1-score, even though both use QLoRA and low-rank adaptation (Yousefiramandi et al., 14 Dec 2025).

Instruction embeddings can also be meta-representations of tasks rather than content. The IEB benchmark defines 1,353 task categories over 47,161 instructions, and Prompt-based Instruction Embedding (PIE) trains task-sensitive embeddings with supervised contrastive learning and hard negatives constructed from same-verb or same-noun task confusions. These embeddings improve data selection for instruction tuning, demonstration retrieval for in-context learning, tiny benchmark construction, and dataset task correlation analysis (Li et al., 2024).

5. Multimodal instruction-conditioned embeddings and hidden-state probes

Multimodal instruction-tuned embedding models use the same conditioning principle, but the representations are derived from generated responses to image, video, or audio prompts. In image-based work, InstructBLIP, mPLUG-Owl, and IDEFICS are treated as zero-shot feature extractors: for each image and instruction, the hidden states of generated output tokens are averaged at each layer and used to predict NSD fMRI responses. These instruction-tuned multimodal LLMs outperform a vision-only ViT-H baseline and perform comparably to CLIP-text, while also showing that different instructions preferentially align with different regions of visual cortex; image captioning tends to dominate high-level visual ROIs, dominant-color prompts better explain early visual cortex, and most explained variance is shared between captioning and other instructions (Oota et al., 26 May 2025).

The video-audio setting pushes this farther. Instruction-specific embeddings from six video and two audio instruction-tuned MLLMs are evaluated against fMRI recorded during naturalistic movie watching. Across 13 video task-specific instructions, instruction-tuned video MLLMs significantly outperform non-instruction-tuned multimodal models by 15% and unimodal baselines by 20%, and the resulting representations show both task-specific voxel preference maps and hierarchical layer-to-cortex alignment: early sensory areas align with early layers, while higher-level visual and language regions align more with middle-to-late layers (Oota et al., 9 Jun 2025). This suggests that instruction-conditioned multimodal embeddings are not merely better pooled features, but encode partially disentangled subspaces for narrative, spatiotemporal, perceptual, and affective processing.

Hidden-state geometry also matters for safety. BEEAR studies instruction-tuned LLMs with safety backdoors and shows that triggers induce “relatively uniform drifts in the model’s embedding space.” The method models the backdoor as an additive perturbation in an intermediate hidden-state space, learns a universal embedding perturbation that elicits unsafe behavior, and then adversarially retrains the model so that this perturbation produces safe behavior instead. In experiments, it reduces RLHF-time backdoor attack success from >95% to <1% and instruction-tuning-time malicious-code backdoors from 47% to 0%, without compromising model utility (Zeng et al., 2024). Within the broader topic, this work is significant because it treats the instruction-tuned model’s embedding space not only as a source of downstream features but as the operative medium of both attack and defense.

6. Evaluation methodology, prompt sensitivity, and open issues

A central controversy is whether current evaluation protocols actually measure instruction-tuned embeddings rather than prompt choice. The most direct answer is negative. A large empirical study evaluates 6 embedding models, 11 datasets, and 15 task-specific prompts per dataset, for a total of 990 model–dataset–prompt combinations, and shows that single-prompt evaluation misrepresents the distribution of scores over plausible prompts. The default prompt can understate or overstate performance, leaderboard ranks are not robust to prompt selection, and by choosing prompts favorably any model in the study can be promoted to first place (Kostiuk et al., 21 May 2026). For instruction-tuned embedding models, prompt robustness is therefore not a minor nuisance but a first-class measurement problem.

Earlier zero-shot robustness work reached a similar conclusion for instruction-tuned LMs more generally. Using 319 human-written instructions over more than 80 tasks, it shows that novel but appropriate instruction phrasings consistently degrade performance and increase variance relative to observed training-time phrasings. The proposed mitigation is representation-level: introduce soft prompt embedding parameters and optimize them so that semantically equivalent instructions induce similar output distributions via a KL alignment term, while keeping the base model frozen (Sun et al., 2023). This line of work implies that instruction-tuned embedding models should be evaluated not only for mean downstream performance but also for invariance across semantically equivalent prompts.

Several additional limitations recur across the literature. LG-AI-Embedding is evaluated only on English MTEB(v2), and its teacher-derived soft supervision means that biases or errors in the ANNA IR pipeline propagate to the student model (Choi et al., 9 Jun 2025). GraphLLaVA’s graph encoder is explicitly query-agnostic, which the paper identifies as a representational bottleneck for graph instruction following (Haag et al., 2024). Brain-alignment studies infer task structure through voxelwise encoding and variance partitioning but do not yet provide direct geometric analyses such as CCA or clustering in the embedding space (Oota et al., 9 Jun 2025). In user modeling, naive manual instructions can hurt performance before subsequent instruction tuning corrects them, which makes instruction quality itself a modeling variable rather than a fixed interface (Gao et al., 13 Oct 2025).

Taken together, these results indicate that instruction-tuned embedding models are best understood as conditional representation systems whose quality depends on three linked properties: how faithfully the instruction modulates the embedding, how efficiently that modulation can be deployed at scale, and how stable the resulting geometry remains under paraphrase, domain shift, and adversarial manipulation. The present literature shows clear gains on retrieval, classification, graph understanding, multimodal alignment, and user modeling, but it also shows that prompt sensitivity, task disentanglement, and evaluation protocol remain unresolved core problems.