Instruction-Based Embeddings

• Instruction-based embeddings are vector representations produced under natural language guidance, allowing the same input to be embedded differently based on task intent.
• They typically use methods such as concatenative conditioning, contrastive alignment, and answer-based formulations to modulate representation geometry according to explicit instructions.
• These embeddings are applied across various domains—including text, time series, EEG, and graphs—enabling tailored retrieval, clustering, and control in both unimodal and multimodal settings.

Searching arXiv for papers on instruction-based embeddings and closely related formulations. arXiv search: "instruction-based embeddings" Instruction-based embeddings are vector representations produced under natural-language guidance so that the resulting vectors reflect task intent, use case, domain, target conditions, or other instruction-specified aspects of the input. In the literature, they appear in at least three closely related forms: task-conditioned text embeddings for retrieval, clustering, classification, and evaluation; shared multimodal embeddings that align language with non-text signals such as time series, EEG, graphs, or user logs; and internal contextual hidden states associated with instruction tokens inside multimodal LLMs. Across these settings, the central claim is consistent: the same underlying input can and often should be embedded differently under different instructions, because the instruction specifies what similarity, relevance, control, or grounding should mean in that instance (Su et al., 2022, Peng et al., 2024, Qiu et al., 2 Aug 2025).

1. Conceptual scope and definitions

A foundational formulation appears in "One Embedder, Any Task: Instruction-Finetuned Text Embeddings" (Su et al., 2022). There, an instruction describes the intended task and domain, the text is embedded together with that instruction, and the same encoder is used for both query and document sides with shared weights. The resulting representation is explicitly task-aware: retrieval, clustering, semantic textual similarity, classification, summarization evaluation, and prompt retrieval are all framed as embedding problems whose geometry is modulated by natural-language descriptions of the use case.

A second line of work defines instruction-based embeddings less by downstream similarity and more by the latent structure they should preserve. "Instruction Embedding: Latent Representations of Instructions Towards Task Identification" (Li et al., 2024) argues that, for instruction data, the most important aspect is the task it represents rather than the specific semantics and knowledge information. Its benchmark, IEB, therefore evaluates whether embeddings recover task identity, often expressible as a verb-noun pair such as “write essay” or “generate sentence,” rather than generic sentence-level similarity. This framing sharply distinguishes instruction embeddings from ordinary text embeddings.

A third formulation treats instructions as controllers over internal or multimodal latent spaces. In "Answer is All You Need: Instruction-following Text Embedding via Answering the Question" (Peng et al., 2024), the instruction is treated as a question about the input text, and the embedding is derived from the expected answer. In "Instruction-based Time Series Editing" (Qiu et al., 2 Aug 2025), both a time series and a natural-language instruction are mapped to unit-length vectors on a shared hypersphere, and edits are obtained by decoding an interpolated representation. In "Continual LLaVA: Continual Instruction Tuning in Large Vision-LLMs" (Cao et al., 2024), instruction-conditioned surrogate embeddings select low-rank increments from a pool to adapt a frozen LVLM during stream-task learning. In "Instruction Lens Score" (Lai et al., 12 May 2026), the contextual hidden states of instruction tokens are themselves treated as instruction-based embeddings that implicitly encode image-grounded semantic information and filter hallucinated objects.

These definitions are not interchangeable, but they are structurally related. In each case, the instruction is not merely metadata appended to an input; it changes the latent space, the retrieval criterion, the control variable, or the model-internal evidence used for downstream inference.

2. Learning objectives and mathematical formulations

A dominant training pattern is contrastive alignment. INSTRUCTOR computes an embedding from the concatenation $I_x \oplus x$, mean-pools only the text-token hidden states, and optimizes a bidirectional in-batch contrastive loss with cosine similarity and temperature $\gamma$ (Su et al., 2022). For a text $x$ with instruction $I_x$, the embedding is

$$e_x = f_\theta(I_x, x) = \frac{1}{|M|} \sum_{t \in M} h_t ,$$

where $M$ indexes the token positions belonging to $x$. The training signal is an InfoNCE-style objective over positives and in-batch negatives. This formulation is representative of a large portion of instruction-conditioned retrieval models.

A related but more task-identification-oriented objective appears in PIE (Li et al., 2024). There, same-task instructions form positive pairs, categories sharing the verb or noun supply hard negatives, and supervised contrastive learning is applied with temperature $T = 0.05$. The emphasis is not generic semantic similarity but task separability, which explains why PIE is evaluated by k-means clustering over task categories and by an Instruction Intention Similarity test rather than by conventional sentence similarity alone.

Other systems depart from pure contrastive learning. InBedder fine-tunes LLMs only on abstractive question answering and treats the instruction as a question about the input; the objective is standard autoregressive negative log-likelihood over the answer tokens conditioned on the prompt template containing the input and instruction (Peng et al., 2024). GSTransform instead learns a lightweight instruction-specific transformation over precomputed generic embeddings using a contrastive term plus a reconstruction term,

$$\mathcal{L} = \beta_1 \mathcal{L}_{contr} + \beta_2 \mathcal{L}_{recon},$$

with $\beta_1=\beta_2=1.0$ by default, so that instruction-relevant structure is emphasized without re-encoding the full corpus (Feng et al., 30 May 2025).

Multimodal systems frequently combine alignment with reconstruction or generation. InstructTime optimizes a symmetric InfoNCE alignment loss between time-series and instruction embeddings together with an MSE reconstruction/editing term,

$\gamma$0

using a two-step schedule: contrastive pre-alignment followed by joint optimization with the decoder (Qiu et al., 2 Aug 2025). InstructUE couples autoregressive learning over synthetic instruction-answer pairs with an InfoNCE-style contrastive loss between user embeddings and answer embeddings, yielding

$\gamma$1

with $\gamma$2 empirically (Gao et al., 13 Oct 2025). ELASTIQ uses cosine loss to align instruction-conditioned EEG embeddings to textual label prototypes after spectral-temporal pretraining (Jiang et al., 29 Sep 2025).

The objective choice determines what the embedding is expected to preserve. Contrastive losses emphasize neighborhood structure and discrimination. Autoregressive answer-based formulations bias the embedding toward the latent answer implied by the instruction. Reconstruction terms preserve source fidelity or global structure. Prototype or ordinal losses, as in EEG alignment and emotional speech synthesis, enforce semantic anchoring or ordered geometry rather than mere pairwise similarity (Jiang et al., 29 Sep 2025, Wu et al., 8 Jun 2026).

3. Architectural patterns

Several architectural motifs recur across the literature, even when the target modality changes.

Before the table, it is useful to separate conditioning route from representation type. Some models condition by simple concatenation and shared encoding; some by generating an answer and using answer-side hidden states; some by transforming a precomputed embedding space; and some by constructing a genuinely shared multimodal manifold with an explicit decoder. Others treat the instruction as a routing signal that dynamically selects adapters or low-rank increments.

Pattern	Representative mechanism	Example papers
Concatenative conditioning	Encode $\gamma$3 with one encoder; mean-pool text tokens only	INSTRUCTOR (Su et al., 2022)
Answer-conditioned embedding	Prompt with input and instruction; use generated-answer hidden states, especially first-generation-step hidden state	InBedder (Peng et al., 2024)
Task-identity prompting	Prompt Llama2/BERT so representations attend to task intention; apply supervised contrastive training	PIE (Li et al., 2024)
Guided space transformation	Learn a linear encoder-decoder that transforms precomputed generic embeddings using instruction-labeled supervision	GSTransform (Feng et al., 30 May 2025)
Shared multimodal hypersphere	Map source signal and instruction to unit vectors in a common space, then decode interpolated embeddings	InstructTime (Qiu et al., 2 Aug 2025)
Dynamic instruction-conditioned adapters	Use Sentence-BERT instruction surrogates to select and aggregate low-rank increments injected into the LLM output projection	Continual LLaVA (Cao et al., 2024)
Instruction-modulated cross-attention	Inject instruction embeddings with FiLM and query-based cross-attention before alignment to text prototypes	ELASTIQ (Jiang et al., 29 Sep 2025)

INSTRUCTOR is the canonical text-only example of explicit instruction conditioning by concatenation (Su et al., 2022). InBedder argues that simply embedding the prompt side is insufficiently instruction-aware, and that generation-side hidden states are more faithful to the facet requested by the instruction (Peng et al., 2024). GSTransform makes a different claim: instruction-relevant information is already latent in generic embeddings, so a small transformation learned from an instruction-labeled subset can be more efficient than re-encoding every item for every instruction (Feng et al., 30 May 2025).

Multimodal cases generalize the same design choices. InstructTime uses frozen SentenceTransformer instruction embeddings, parallel MLP heads, multi-resolution 1D CNNs, and an eight-block Transformer-style decoder operating on a two-token sequence $\gamma$4 (Qiu et al., 2 Aug 2025). Continual LLaVA keeps the CLIP vision encoder, visual projector, and Vicuna backbone frozen, and trains only proxy embeddings, low-rank factors, and contextual task weights; the adapted output projection applies

$\gamma$5

ELASTIQ conditions EEG tokens with instruction-derived FiLM parameters, then uses an Instruction-conditioned Q-Former with $\gamma$6 learnable queries to align the resulting representation to textual label embeddings (Jiang et al., 29 Sep 2025). Joint graph-language instruction tuning follows a similar fusion recipe by projecting a graph encoder output into the LLM token space and concatenating graph tokens with instruction tokens before autoregressive answer generation (Haag et al., 2024).

A common misconception is that instruction-based embeddings are only “prompted sentence embeddings.” The cited systems show a wider design space: direct answer-space encoding, dynamic low-rank routing, hyperspherical interpolation, query bottlenecks, and plug-in transformations all satisfy the instruction-based criterion, but they do so by substantially different computational mechanisms.

4. Representation geometry, control, and what instructions encode

The geometry induced by instructions is one of the most distinctive themes in the literature. INSTRUCTOR shows that the same text can be embedded differently under different instructions because the instruction changes the intended task and domain (Su et al., 2022). PIE goes further: it explicitly rejects the assumption that instruction embeddings should optimize generic sentence semantics, and instead prioritizes task identity and interaction style (Li et al., 2024). This is why semantically close instructions may still need to be far apart if they instantiate different task categories.

Several systems make this geometry directly controllable. InstructTime constrains both modalities to unit-norm embeddings on a shared hypersphere and edits by Euclidean interpolation,

$\gamma$7

with $\gamma$8 controlling edit strength (Qiu et al., 2 Aug 2025). The paper reports smooth monotonic changes in RaTS, $\gamma$9DTW, and $x$0RaTS$x$1 as $x$2 varies, yielding an explicit editability-preservation trade-off that diffusion editors in that comparison do not provide.

Instruction-conditioned geometry can also emerge inside generative models rather than in external embedding APIs. "Do LLMs 'know' internally when they follow instructions?" (Heo et al., 2024) identifies an instruction-following direction $x$3 in the residual stream such that a linear probe on first-token representations predicts success versus failure on IFEval-simple with AUROC values of 0.77, 0.83, 0.74, and 0.88 on LLaMA-2-7B-chat, LLaMA-2-13B-chat, Mistral-7B-Instruct-v0.3, and Phi-3-mini-128k-instruct, respectively, for task generalization. The intervention

$x$4

improves success rates relative to random directions, but instruction-type generalization remains near chance, and phrasing modifications align more strongly with $x$5 than task familiarity or instruction difficulty. This directly contradicts any assumption that there is a single universal instruction-following axis.

An even stronger multimodal claim appears in "Instruction Lens Score" (Lai et al., 12 May 2026). There, instruction token embeddings in MLLMs are probed with a Logit Lens and shown to encode visual information while filtering erroneous or misleading signals introduced by visual embeddings. The paper reports that AUROC based on instruction embeddings exceeds that based on image embeddings by at least 8% across models when separating real objects from hallucinated ones. Its final score combines a Calibrated Local Score and a Context Consistency Score,

$x$6

yielding a training-free hallucination detector.

A plausible implication is that instruction-based embeddings are best understood not as a single method but as a family of geometry-shaping devices. Some reshape an embedding space toward task identity, some create a continuous control axis, and some expose latent evidence already stored in instruction tokens after cross-modal fusion.

5. Modalities and application domains

Instruction-based embeddings are no longer limited to text retrieval. In text-centric settings, INSTRUCTOR remains a landmark system: it trains on instructions for 330 diverse tasks and evaluates on 70 embedding tasks, 66 of which are unseen during training, achieving an average improvement of 3.4% compared to the previous best results (Su et al., 2022). InBedder introduces instruction-following text embeddings by answering the question posed by the instruction, and reports significantly improved instruction-following capabilities under its proposed instruction awareness and instruction robustness tests for both LLaMA-2-7B and RoBERTa-large (Peng et al., 2024). GSTransform addresses the computational bottleneck of re-encoding corpora per instruction and reports dramatic speedups of 6–300x in real-time processing on large datasets while improving instruction-following text embedding quality over prior methods across nine datasets (Feng et al., 30 May 2025). In low-resource multilingual alignment, frozen multilingual encoders are evaluated on persona-instruction compatibility across 12 Indian languages; E5-Large-Instruct reaches 27.4% Recall@1 on monolingual retrieval and 20.7% on cross-lingual transfer, BGE-M3 leads reverse retrieval at 32.1% Recall@1, and LaBSE attains 75.3% AUROC for classification (Shah et al., 15 Jan 2026).

Time series work extends the paradigm from similarity to controlled generation. InstructTime is described as the first instruction-based time series editor, accepts free-form instructions, uses multi-resolution encoders to align local and global conditions, and is reported to achieve state-of-the-art editability and preservation in the instruction setting across Synthetic, Air Quality, and NICU Heart Rate datasets (Qiu et al., 2 Aug 2025). In synthetic data it also reports lower MSE and MAE than diffusion editors while providing explicit strength control through interpolation.

Biosignal work uses instructions as semantic priors. ELASTIQ aligns EEG to language via instruction tuning after spectral-temporal pretraining, evaluates on 20 datasets spanning motor imagery, emotion recognition, steady-state visual evoked potentials, covert speech, and healthcare, achieves state-of-the-art performance on 14 of the 20 datasets, and reports average macro-accuracy 0.6678 with average Kappa 0.5391 (Jiang et al., 29 Sep 2025). Its analyses further report that direct inference with instructions improves average balanced accuracy by 1.77% for “Informed Task” and by 2.12% for “Informed Task + Targets” relative to “No Instruction.”

Instruction-based embeddings also function as parameter-efficient controllers in multimodal continual learning. Continual LLaVA introduces COAST with domain-, capability-, and dataset-incremental settings, then augments a frozen LVLM with dual increment embeddings conditioned on the current instruction (Cao et al., 2024). On COAST-domain it reports Avg 37.08% and Forgetting 2.58%, compared with 24.02% and 15.83% for sequential finetuning, while training only about 0.75M parameters versus approximately 6.76B. The same design is evaluated in capability- and dataset-incremental settings and consistently reduces forgetting.

Personalization and user modeling provide another extension. InstructUE produces conditional user vectors $x$7 from heterogeneous behavioral sources, trains with a contrastive-autoregressive framework over a 200 million-triplet UserQA dataset, and reports state-of-the-art results on five of six downstream scenarios, including 98.46 AUC / 0.8972 KS on Game Top-up and 95.69 / 0.7700 on Purchasing Power (Gao et al., 13 Oct 2025). The paper also shows that instruction-aware variants outperform universal embeddings on several tasks and visualizes instruction-aware representations as six well-separated clusters.

Instruction-grounded control has also entered speech and graph generation. EmoInstruct-TTS defines an Emotion2embed space of dimension 896 covering 48 emotional states, trains ICE-Flow to infer embeddings from free-form instructions, and reports higher MOS and emotion similarity than strong baselines in zero-shot emotional TTS evaluations (Wu et al., 8 Jun 2026). For graphs, GraphLLaVA projects a graph embedding into the LLM embedding space and conditions answer generation on the joint graph-instruction representation; it reports overall binary accuracy 62.90% versus 44.41% for a TinyLlama graph-to-text baseline, and GPT-4-as-judge preference 56.0% versus 37.5% (Haag et al., 2024).

A separate application family concerns curation, replay, and diagnosis rather than direct prediction. Dynosaur uses sentence-level instruction embeddings to select diverse replay tasks and reports Stage 3 SUPER-NI Test ROUGE-L 44.8 for Instr. Diverse replay versus 43.3 for No Replay and 43.4 for Full one-shot training (Yin et al., 2023). Monosemantic internal activations derived through sparse autoencoders are then used as instruction-based embeddings for task-specific data selection, where NAS reports average gains over BM25, DSIR, dense embedding baselines, and influence-based selection on several instruction-tuning settings (Ma et al., 19 Mar 2025).

6. Evaluation practices, limitations, and open problems

Evaluation is highly task-specific, which reflects the breadth of the paradigm. Text retrieval and alignment work uses Recall@k, MRR@10, AUROC, AUPRC, ECE, clustering purity, homogeneity, silhouette, Spearman correlation, and downstream accuracy (Shah et al., 15 Jan 2026, Li et al., 2024). Time-series editing uses RaTS, $x$8DTW, MSE, and MAE (Qiu et al., 2 Aug 2025). EEG alignment uses balanced accuracy and Cohen’s Kappa (Jiang et al., 29 Sep 2025). Hallucination detection relies on AUROC and AUPR (Lai et al., 12 May 2026). Continual instruction tuning emphasizes Average Accuracy and Average Forgetting (Cao et al., 2024). These metrics are not interchangeable: some quantify semantic alignment, others controllability, preservation, calibration, robustness, or forgetting.

Several recurring limitations cut across papers. Prompt phrasing remains a first-order variable. PIE reports that prompt design materially affects task-identification performance, and the instruction-following-direction study shows that the learned direction is more closely related to prompt phrasing than to task familiarity or instruction difficulty (Li et al., 2024, Heo et al., 2024). Incorrect instructions can actively distort embeddings: ELASTIQ finds that misleading instructions reconfigure the EEG embedding space toward incorrect semantics, and EmoInstruct-TTS notes instability for contradictory descriptors such as “calm yet highly excited” (Jiang et al., 29 Sep 2025, Wu et al., 8 Jun 2026).

Generalization is also uneven. InstructTime reports that zero-shot generalization to entirely unseen attribute levels is limited, with near-zero RaTS, although 1–10 examples can nearly match training on all levels (Qiu et al., 2 Aug 2025). Continual LLaVA avoids catastrophic forgetting without replay, but its sensitivity to the similarity metric and to dataset-incremental heterogeneity remains explicit (Cao et al., 2024). The Indic multilingual benchmark shows an average drop of about 25% from monolingual to cross-lingual Recall@1, a +6.7 point advantage for Devanagari-pair transfer over cross-script pairs, and consistently weaker performance for Assamese and Odia (Shah et al., 15 Jan 2026). GSTransform is efficient, but it does not claim zero-shot transfer to unseen instructions; each new instruction typically requires its own learned transformation (Feng et al., 30 May 2025). InBedder gives strong instruction awareness, but because embeddings depend on the instruction, corpus-scale systems must re-embed or rerank per instruction (Peng et al., 2024).

A broader controversy concerns universality versus specificity. Some results favor broad instructions: in multimodal brain alignment, image captioning embeddings share most of the explained variance with other instruction-conditioned embeddings (Oota et al., 26 May 2025). Other results favor narrowly task-defined representations: PIE improves data selection and demonstration retrieval precisely because it suppresses generic semantics in favor of task identity (Li et al., 2024). This suggests that “better” instruction-based embeddings are not universally more specialized or more general. Their value depends on whether the target application requires shared semantic coverage, strict task discrimination, explicit control, or robust adaptation under changing instructions.

The field therefore remains plural rather than convergent. What unifies it is not a single model family or loss, but a common principle: embeddings become more useful when the instruction is treated as a first-class signal that defines what the representation should preserve, align, retrieve, edit, or control.