Direct Embedding Optimization (DEO)

• DEO is defined as methods that directly optimize native embeddings—such as query, conditional, or timestep vectors—to control model outputs without full model fine-tuning.
• It employs specific loss functions and optimization strategies (e.g., contrastive, cross-entropy, and gradient ascent) to enhance retrieval, diffusion steering, prompt embedding, and time-series QA.
• The approach maintains most model parameters fixed, offering efficiency and preserved domain structure, yet faces challenges like optimization stability and domain-specific limitations.

Direct Embedding Optimization (DEO) denotes a class of methods that optimize continuous embeddings directly, rather than relying primarily on discrete token search, coordinate-space guidance, or full model fine-tuning. In the cited literature, the term is used for query-time negation-aware retrieval, inference-time steering of protein diffusion models, direct optimization of original prompt-token embeddings in aligned LLMs, and direct timestep embedding for time-series question answering (Lee et al., 10 Mar 2026, Li et al., 5 Feb 2026, Li et al., 27 Apr 2026, Wu et al., 17 Jun 2026). This suggests a shared conceptual pattern: embeddings are treated as the primary control surface for changing model behavior while most of the surrounding model remains fixed.

1. Terminology and scope

The acronym “DEO” is not semantically stable across recent arXiv usage. In several papers it expands to “Direct Embedding Optimization,” but in another paper it expands to “Dimer-Enhanced Optimization,” a curvature-aware training method for escaping saddle points in neural network optimization rather than an embedding-space method (Hu et al., 26 Jul 2025). A related dense-retrieval paper does not use the term DEO at all, but describes “embedding alignment,” which is conceptually the direct optimization of a student encoder’s outputs to match a teacher embedding space (Wang et al., 2023).

Work	Meaning of DEO	Primary optimized object
(Lee et al., 10 Mar 2026)	Direct Embedding Optimization	Query embedding
(Li et al., 5 Feb 2026)	Direct Embedding Optimization	Conditional protein embedding $c=\{s,z\}$
(Li et al., 27 Apr 2026)	Direct Embedding Optimization as realized by PEO	Original prompt-token embeddings
(Wu et al., 17 Jun 2026)	Direct Embedding Optimization as realized in CADE	Timestep embeddings and lightweight projection layers
(Hu et al., 26 Jul 2025)	Dimer-Enhanced Optimization	Model parameters $\theta$

This terminological divergence matters because the substantive methods differ sharply. In retrieval, DEO is training-free and query-time. In protein diffusion, it is inference-time steering of a pretrained conditional denoiser. In prompt attacks, it is a white-box optimization over token embeddings for a fixed prompt. In CADE, it is an architectural and training choice that bypasses tokenization and patching for numerical sequences (Lee et al., 10 Mar 2026, Li et al., 5 Feb 2026, Li et al., 27 Apr 2026, Wu et al., 17 Jun 2026).

2. Common optimization structure

Despite domain differences, the direct-embedding methods share several structural features. First, the optimized variable is an embedding already native to the host model: a query vector $e_u$ in retrieval, a conditional embedding $c=\{s,z\}$ in protein diffusion, a prompt-embedding tensor $E=(e_1,\dots,e_n)$ in LLM jailbreaking, or a timestep-embedding sequence $E_{ts}$ in time-series QA (Lee et al., 10 Mar 2026, Li et al., 5 Feb 2026, Li et al., 27 Apr 2026, Wu et al., 17 Jun 2026). Second, most surrounding parameters remain frozen or minimally modified. The retrieval method keeps the encoder frozen and optimizes only the current query embedding; EmbedOpt steers a pretrained diffusion prior at inference time; PEO does not fine-tune model weights; CADE trains lightweight projection components and uses LoRA for the LLM branch while freezing text anchors for the contrastive branch (Lee et al., 10 Mar 2026, Li et al., 5 Feb 2026, Li et al., 27 Apr 2026, Wu et al., 17 Jun 2026).

The objective is likewise expressed in embedding-native terms. Negation-aware retrieval uses a quadratic contrastive objective that attracts the optimized query embedding toward positive sub-query embeddings, repels it from negative sub-query embeddings, and anchors it to the original query embedding:

$$L(e_u)=\lambda_p \frac{1}{K}\sum_{i=1}^{K}\|e_u-e_{p_i}\|^2-\lambda_n \frac{1}{M}\sum_{j=1}^{M}\|e_u-e_{n_j}\|^2+\lambda_o\|e_u-e_o\|^2.$$

Protein DEO maximizes an experimental reward through the denoiser output by solving a local surrogate

$\max_c R(\hat{x}_\theta(x_t,c,\sigma_t)),$

then denoising with the updated embedding. PEO uses teacher-forced cross-entropy on a structured continuation target plus an $L_2$ anchor to keep optimized prompt embeddings close to their originals:

$$L(E;y)=L_{CE}(y \mid \mathrm{Concat}(E,E_y))+\lambda \|E-E^{(0)}\|_2^2.$$

CADE combines autoregressive cross-entropy with a one-directional supervised contrastive loss against frozen class-name text anchors (Lee et al., 10 Mar 2026, Li et al., 5 Feb 2026, Li et al., 27 Apr 2026, Wu et al., 17 Jun 2026).

A plausible implication is that DEO methods are most attractive when the embedding space already encodes domain structure that is hard to express through discrete or coordinate-level edits. The retrieval paper relies on include/exclude semantics in a pretrained encoder space; EmbedOpt relies on conditional embeddings that encode sequence and coevolutionary signals; PEO exploits the fact that LLMs condition on actual continuous prompt embeddings even when the visible prompt string is unchanged; CADE relies on the LLM input space as the target representation for numerical timesteps (Lee et al., 10 Mar 2026, Li et al., 5 Feb 2026, Li et al., 27 Apr 2026, Wu et al., 17 Jun 2026).

3. Negation-aware and dense retrieval

In "DEO: Training-Free Direct Embedding Optimization for Negation-Aware Retrieval" (Lee et al., 10 Mar 2026), DEO is a training-free query-time method for text and multimodal retrieval. A user query is decomposed into positive components $\theta$0 and negative components $\theta$1 by an LLM-based parser; the reported default parser is GPT-4.1-nano at temperature $\theta$2, and Qwen2.5-1.5B-Instruct was also tested. The base embedding models are frozen: BGE-small-en-v1.5, BGE-large-en-v1.5, and BGE-M3 for text, and OpenAI CLIP, CLIP-laion400m, CLIP-datacomp, and NegCLIP for multimodal retrieval. FAISS is used for indexing, cosine similarity for scoring, and DEO replaces the raw query embedding with the optimized one before nearest-neighbor search (Lee et al., 10 Mar 2026).

The method admits both iterative and closed-form optimization. With $\theta$3 and $\theta$4, the stationary point is

$\theta$5

when $\theta$6. The paper also reports Adam optimization for $\theta$7 steps by default, with $\theta$8–$\theta$9 typically enough and mild degradation beyond approximately $e_u$0 steps. On NegConstraint with BGE-large-en-v1.5, DEO improves MAP from $e_u$1 to $e_u$2 and nDCG@10 from $e_u$3 to $e_u$4, corresponding to gains of $e_u$5 MAP@100 and $e_u$6 nDCG@10. On COCO-Neg, Recall@5 for OpenAI CLIP increases from $e_u$7 to $e_u$8, reported as $e_u$9 (Lee et al., 10 Mar 2026).

A related but terminologically distinct dense-retrieval baseline is "Query Encoder Distillation via Embedding Alignment is a Strong Baseline Method to Boost Dense Retriever Online Efficiency" (Wang et al., 2023). That work does not use the term DEO, but it directly regresses a student query encoder to teacher query embeddings with an MSE objective, keeps the teacher document index fixed, and thereby instantiates a pure embedding-alignment regime. The reported result is that a $c=\{s,z\}$0-layer BERT-based query encoder retains $c=\{s,z\}$1 of full dual-encoder BEIR performance on average, while a $c=\{s,z\}$2-layer student retains $c=\{s,z\}$3; training is unsupervised over approximately $c=\{s,z\}$4 million MS MARCO queries with AdamW, batch size $c=\{s,z\}$5, learning rate $c=\{s,z\}$6, $c=\{s,z\}$7 warm-up steps, and $c=\{s,z\}$8 epoch (Wang et al., 2023). Taken together, these papers show two distinct retrieval uses of direct embedding control: single-query optimization at inference time and student-space alignment during distillation.

4. Inference-time steering of generative and conditional models

"Robust Inference-Time Steering of Protein Diffusion Models via Embedding Optimization" (Li et al., 5 Feb 2026) defines DEO as an inference-time steering paradigm for conditional diffusion models. Its concrete instantiation, EmbedOpt, updates the conditional embedding space rather than the coordinates directly. For AlphaFold 3–style sequence-to-structure models, the optimized condition is $c=\{s,z\}$9, where $E=(e_1,\dots,e_n)$0 is a single embedding of per-residue features and $E=(e_1,\dots,e_n)$1 is a pair embedding of residue-pair features. At each diffusion step, EmbedOpt computes $E=(e_1,\dots,e_n)$2, backpropagates a differentiable reward $E=(e_1,\dots,e_n)$3 to $E=(e_1,\dots,e_n)$4 and $E=(e_1,\dots,e_n)$5, RMS-normalizes the gradients separately, updates the embeddings by a single gradient-ascent step, and then performs the denoising update with the modified condition (Li et al., 5 Feb 2026).

The paper contrasts this embedding-space steering with coordinate-space posterior sampling such as DPS. A first-order Taylor approximation is used to argue that EmbedOpt replaces the DPS-style pull-back with an embedding-space preconditioning term $E=(e_1,\dots,e_n)$6, which the paper associates with smoother and better-conditioned updates. Under local smoothness and small-step assumptions, it also gives an informal monotone surrogate-improvement guarantee. Empirically, the method is evaluated on two benchmarks: cryo-electron microscopy map fitting over $E=(e_1,\dots,e_n)$7 proteins and distance-constrained structure determination over $E=(e_1,\dots,e_n)$8 multi-domain proteins. On the hard cryo-EM target 8H1I, the prior has map correlation coefficient $E=(e_1,\dots,e_n)$9, DPS reaches $E_{ts}$0, and EmbedOpt reaches $E_{ts}$1. Across hyperparameters spanning approximately $E_{ts}$2 to $E_{ts}$3, EmbedOpt shows a broad plateau in map-fitting performance and MolProbity quality, whereas DPS exhibits a brittle sweet spot; EmbedOpt also remains effective when diffusion steps are reduced from $E_{ts}$4 to $E_{ts}$5, corresponding to an approximately $E_{ts}$6 speedup (Li et al., 5 Feb 2026).

This usage of DEO is notable because it treats the conditioning embedding as a means of “moving the prior” rather than overpowering it. The paper argues that when targets lie in low-density regions of the pretrained prior, coordinate-space guidance must use aggressive weighting and can leave the learned manifold, whereas embedding optimization shifts the model’s conditional preferences so that subsequent denoising steps naturally move toward higher-likelihood structures while remaining closer to the manifold of valid proteins (Li et al., 5 Feb 2026).

5. Prompt embedding optimization in aligned LLMs

"Adaptive Prompt Embedding Optimization for LLM Jailbreaking" (Li et al., 27 Apr 2026) presents Prompt Embedding Optimization (PEO) as a concrete realization of DEO in a white-box jailbreak setting. Instead of appending discrete adversarial suffixes or soft prompts, PEO directly optimizes the continuous embeddings of the original prompt tokens. The attacker has white-box access to tokenizer, embedding layer, forward pass, and gradients with respect to prompt-token embeddings, but model weights are not fine-tuned and each prompt is attacked independently. The optimization target is a teacher-forced structured continuation associated with the harmful behavior, used as an optimization scaffold rather than as the evaluation criterion (Li et al., 27 Apr 2026).

The method optimizes prompt embeddings $E_{ts}$7 with Adam at learning rate $E_{ts}$8, gradient clipping norm $E_{ts}$9, and $$L(e_u)=\lambda_p \frac{1}{K}\sum_{i=1}^{K}\|e_u-e_{p_i}\|^2-\lambda_n \frac{1}{M}\sum_{j=1}^{M}\|e_u-e_{n_j}\|^2+\lambda_o\|e_u-e_o\|^2.$$0 iterations per pass, using the anchored cross-entropy loss

$$L(e_u)=\lambda_p \frac{1}{K}\sum_{i=1}^{K}\|e_u-e_{p_i}\|^2-\lambda_n \frac{1}{M}\sum_{j=1}^{M}\|e_u-e_{n_j}\|^2+\lambda_o\|e_u-e_o\|^2.$$1

with $$L(e_u)=\lambda_p \frac{1}{K}\sum_{i=1}^{K}\|e_u-e_{p_i}\|^2-\lambda_n \frac{1}{M}\sum_{j=1}^{M}\|e_u-e_{n_j}\|^2+\lambda_o\|e_u-e_o\|^2.$$2 in final runs. Later passes use structured continuation targets augmented with heuristic composite response scaffolds, and only prompts that fail advance to the next pass. The schedule cold-starts every pass from $$L(e_u)=\lambda_p \frac{1}{K}\sum_{i=1}^{K}\|e_u-e_{p_i}\|^2-\lambda_n \frac{1}{M}\sum_{j=1}^{M}\|e_u-e_{n_j}\|^2+\lambda_o\|e_u-e_o\|^2.$$3, which the paper motivates as a way to escape failed basins, keep the $$L(e_u)=\lambda_p \frac{1}{K}\sum_{i=1}^{K}\|e_u-e_{p_i}\|^2-\lambda_n \frac{1}{M}\sum_{j=1}^{M}\|e_u-e_{n_j}\|^2+\lambda_o\|e_u-e_o\|^2.$$4 anchor referenced to the original prompt, and bound cumulative drift. For text-level reporting, optimized embeddings are projected to the nearest vocabulary token by cosine similarity; empirically, perturbations remain within the Voronoi cell of the original token for all evaluated models, so nearest-token projection recovers the exact original prompt text, reported as $$L(e_u)=\lambda_p \frac{1}{K}\sum_{i=1}^{K}\|e_u-e_{p_i}\|^2-\lambda_n \frac{1}{M}\sum_{j=1}^{M}\|e_u-e_{n_j}\|^2+\lambda_o\|e_u-e_o\|^2.$$5 text change in every evaluated cell (Li et al., 27 Apr 2026).

Evaluation uses AdvBench and HarmBench text-test against Vicuna-7B-v1.3, Llama-2-7B-Chat, Qwen3-4B-Instruct, and Llama-3.2-3B-Instruct. The primary metric is ASR-Judge, requiring both GPT-5.4 and Claude Opus 4.6 to judge the response harmful, with overall micro-averaged agreement across valid rows of $$L(e_u)=\lambda_p \frac{1}{K}\sum_{i=1}^{K}\|e_u-e_{p_i}\|^2-\lambda_n \frac{1}{M}\sum_{j=1}^{M}\|e_u-e_{n_j}\|^2+\lambda_o\|e_u-e_o\|^2.$$6. On AdvBench ASR-Judge, PEO reports $$L(e_u)=\lambda_p \frac{1}{K}\sum_{i=1}^{K}\|e_u-e_{p_i}\|^2-\lambda_n \frac{1}{M}\sum_{j=1}^{M}\|e_u-e_{n_j}\|^2+\lambda_o\|e_u-e_o\|^2.$$7 on Vicuna, $$L(e_u)=\lambda_p \frac{1}{K}\sum_{i=1}^{K}\|e_u-e_{p_i}\|^2-\lambda_n \frac{1}{M}\sum_{j=1}^{M}\|e_u-e_{n_j}\|^2+\lambda_o\|e_u-e_o\|^2.$$8 on Llama-2, $$L(e_u)=\lambda_p \frac{1}{K}\sum_{i=1}^{K}\|e_u-e_{p_i}\|^2-\lambda_n \frac{1}{M}\sum_{j=1}^{M}\|e_u-e_{n_j}\|^2+\lambda_o\|e_u-e_o\|^2.$$9 on Qwen3, and $\max_c R(\hat{x}_\theta(x_t,c,\sigma_t)),$0 on Llama-3.2, outperforming nanoGCG, SPT, and BEAST in all four model cells; on HarmBench text-test, it reports $\max_c R(\hat{x}_\theta(x_t,c,\sigma_t)),$1, $\max_c R(\hat{x}_\theta(x_t,c,\sigma_t)),$2, $\max_c R(\hat{x}_\theta(x_t,c,\sigma_t)),$3, and $\max_c R(\hat{x}_\theta(x_t,c,\sigma_t)),$4, again leading every model–benchmark cell (Li et al., 27 Apr 2026). The paper also documents substantial divergence between ASR-Match and ASR-Judge, including false positives from off-topic substitution, degenerate repetition, cross-lingual refusals, and fictional framing, and false negatives caused by safety/legal words triggering substring heuristics despite harmful content (Li et al., 27 Apr 2026).

The significance of this result lies in a counterintuitive property emphasized by the paper: even when the visible prompt string is preserved exactly after nearest-token projection, the model still conditions on the actual continuous embeddings at inference. Small but targeted perturbations to those embeddings can therefore alter early hidden states and attention patterns enough to change the autoregressive continuation, while leaving the user-visible text unchanged (Li et al., 27 Apr 2026).

6. Direct timestep embedding for time-series question answering

"Beyond Tokenization: Direct Timestep Embedding and Contrastive Alignment for Time-Series Question Answering" (Wu et al., 17 Jun 2026) treats DEO as the direct optimization of timestep embeddings in the LLM input space, bypassing both text tokenization of raw numbers and patch/window encoders. In CADE, each timestep is mapped first by a point-wise linear encoder to a $\max_c R(\hat{x}_\theta(x_t,c,\sigma_t)),$5-dimensional feature and then by a two-layer MLP projector with GELU and LayerNorm to the LLM embedding dimension $\max_c R(\hat{x}_\theta(x_t,c,\sigma_t)),$6. For a univariate series, the paper writes

$\max_c R(\hat{x}_\theta(x_t,c,\sigma_t)),$7

followed by

$\max_c R(\hat{x}_\theta(x_t,c,\sigma_t)),$8

The sequence $\max_c R(\hat{x}_\theta(x_t,c,\sigma_t)),$9 is then assigned exact timestep positions in the LLM, preserving index-level addressability without patching or padding (Wu et al., 17 Jun 2026).

CADE supplements the direct embedding pathway with a one-directional supervised contrastive loss on classification samples. Lowercased class names are tokenized, their frozen LLM input embeddings are averaged into anchors $L_2$0, timestep embeddings are mean-pooled to $L_2$1, and a memory queue stores previous $L_2$2 pairs. The supervised contrastive loss aligns current time-series representations with anchors for the same class while keeping the anchors and LLM embedding layer frozen. The total objective is

$L_2$3

with $L_2$4 reported as the best overall balance in ablation and a memory bank size of $L_2$5 robust across $L_2$6 to $L_2$7. The implementation uses Qwen-3-0.6B with LoRA fine-tuning, $L_2$8, learning rate $L_2$9, cosine scheduling, batch size $$L(E;y)=L_{CE}(y \mid \mathrm{Concat}(E,E_y))+\lambda \|E-E^{(0)}\|_2^2.$$0, training for $$L(E;y)=L_{CE}(y \mid \mathrm{Concat}(E,E_y))+\lambda \|E-E^{(0)}\|_2^2.$$1 steps on a single NVIDIA A100 40GB (Wu et al., 17 Jun 2026).

On the Time-MQA benchmark, CADE reports improvements over BPE tokenization across six tasks. Selected results include forecasting FCR $$L(E;y)=L_{CE}(y \mid \mathrm{Concat}(E,E_y))+\lambda \|E-E^{(0)}\|_2^2.$$2 versus $$L(E;y)=L_{CE}(y \mid \mathrm{Concat}(E,E_y))+\lambda \|E-E^{(0)}\|_2^2.$$3, forecasting Own-MSE $$L(E;y)=L_{CE}(y \mid \mathrm{Concat}(E,E_y))+\lambda \|E-E^{(0)}\|_2^2.$$4 versus $$L(E;y)=L_{CE}(y \mid \mathrm{Concat}(E,E_y))+\lambda \|E-E^{(0)}\|_2^2.$$5, imputation FCR $$L(E;y)=L_{CE}(y \mid \mathrm{Concat}(E,E_y))+\lambda \|E-E^{(0)}\|_2^2.$$6 versus $$L(E;y)=L_{CE}(y \mid \mathrm{Concat}(E,E_y))+\lambda \|E-E^{(0)}\|_2^2.$$7, imputation Own-MSE $$L(E;y)=L_{CE}(y \mid \mathrm{Concat}(E,E_y))+\lambda \|E-E^{(0)}\|_2^2.$$8 versus $$L(E;y)=L_{CE}(y \mid \mathrm{Concat}(E,E_y))+\lambda \|E-E^{(0)}\|_2^2.$$9, anomaly detection $\theta$00 versus $\theta$01, classification $\theta$02 versus $\theta$03, true/false $\theta$04 versus $\theta$05, and MCQ $\theta$06 versus $\theta$07 (Wu et al., 17 Jun 2026). Against patch-based ITFormer, the paper highlights that ITFormer had FCR $\theta$08 on forecasting and imputation, which it interprets as evidence for the value of per-timestep indexability. An ablation further reports that even a Frozen Random Linear encoder outperformed BPE on every task, suggesting that the continuous-token interface itself is a major driver of gains (Wu et al., 17 Jun 2026).

In this setting, DEO is not a query-time optimization of a single instance but a representational design principle and training regime. The paper’s central claim is that direct embedding of numerical timesteps preserves metric structure and exact temporal indexing that BPE and patch compression discard (Wu et al., 17 Jun 2026).

7. Comparative properties, limitations, and acronym collisions

Across these instantiations, DEO is associated with several recurring advantages. Query-time retrieval DEO requires no model updates and no index changes; protein DEO is reported to be more robust than DPS across hyperparameters spanning two orders of magnitude and to permit fewer diffusion steps; prompt DEO avoids the combinatorial token space and preserves the visible prompt string exactly after nearest-token projection; CADE avoids tokenization and patching while preserving exact index-level access (Lee et al., 10 Mar 2026, Li et al., 5 Feb 2026, Li et al., 27 Apr 2026, Wu et al., 17 Jun 2026). This suggests that direct optimization becomes especially useful when the native embedding space already captures the relevant inductive bias better than the baseline action space.

The limitations are equally domain-specific. Negation-aware retrieval depends on decomposition quality and can over-repel with very large $\theta$09 or too many optimization steps; the paper reports degradation beyond approximately $\theta$10 steps and notes difficulties with implicit negation and entangled concepts (Lee et al., 10 Mar 2026). EmbedOpt requires a differentiable reward and a differentiable path from embeddings to denoiser output, and the paper notes that shallow MSAs or targets outside the training distribution can lead to reward overfitting with compromised physics (Li et al., 5 Feb 2026). PEO requires white-box access to embeddings and gradients, and residual failure modes include empty formatting, off-topic substitutions, creative-genre reframing, and cross-lingual refusals (Li et al., 27 Apr 2026). CADE inherits the $\theta$11 attention cost of native-resolution LLM processing and still trails a larger instruction-tuned LLM on format compliance and reasoning-heavy MCQ (Wu et al., 17 Jun 2026).

A final source of confusion is nomenclatural rather than methodological. "Dimer-Enhanced Optimization: A First-Order Approach to Escaping Saddle Points in Neural Network Training" uses the acronym DEO for a periodic dimer rotation and projected-gradient scheme over model parameters $\theta$12, with one extra gradient evaluation every $\theta$13 steps and default settings such as $\theta$14 and $\theta$15 (Hu et al., 26 Jul 2025). That method does not introduce or use the expansion “Direct Embedding Optimization.” For encyclopedia purposes, the acronym therefore names two separate research lines: one centered on direct manipulation of embedding variables, and another centered on first-order curvature estimation for training dynamics (Hu et al., 26 Jul 2025).