---
title: Direct Embedding Optimization (DEO)
url: https://www.emergentmind.com/topics/direct-embedding-optimization-deo
type: topic
---

# Direct Embedding Optimization (DEO)

Direct Embedding Optimization (DEO) denotes a class of methods that optimize continuous embeddings directly, rather than relying primarily on discrete token search, coordinate-space guidance, or full model fine-tuning. In the cited literature, the term is used for query-time negation-aware retrieval, inference-time steering of protein diffusion models, direct optimization of original prompt-token embeddings in aligned LLMs, and direct timestep embedding for time-series question answering [2603.09185] [2602.05285] [2604.24983] [2606.18986]. This suggests a shared conceptual pattern: embeddings are treated as the primary control surface for changing model behavior while most of the surrounding model remains fixed.

## 1. Terminology and scope

The acronym “DEO” is not semantically stable across recent arXiv usage. In several papers it expands to “Direct Embedding Optimization,” but in another paper it expands to “Dimer-Enhanced Optimization,” a curvature-aware training method for escaping saddle points in neural network optimization rather than an embedding-space method [2507.19968]. A related dense-retrieval paper does not use the term DEO at all, but describes “embedding alignment,” which is conceptually the direct optimization of a student encoder’s outputs to match a teacher embedding space [2306.11550].

| Work | Meaning of DEO | Primary optimized object |
|---|---|---|
| [2603.09185] | Direct Embedding Optimization | Query embedding |
| [2602.05285] | Direct Embedding Optimization | Conditional protein embedding $c=\{s,z\}$ |
| [2604.24983] | Direct Embedding Optimization as realized by PEO | Original prompt-token embeddings |
| [2606.18986] | Direct Embedding Optimization as realized in CADE | Timestep embeddings and lightweight projection layers |
| [2507.19968] | Dimer-Enhanced Optimization | Model parameters $\theta$ |

This terminological divergence matters because the substantive methods differ sharply. In retrieval, DEO is training-free and query-time. In protein diffusion, it is inference-time steering of a pretrained conditional denoiser. In prompt attacks, it is a white-box optimization over token embeddings for a fixed prompt. In CADE, it is an architectural and training choice that bypasses tokenization and patching for numerical sequences [2603.09185] [2602.05285] [2604.24983] [2606.18986].

## 2. Common optimization structure

Despite domain differences, the direct-embedding methods share several structural features. First, the optimized variable is an embedding already native to the host model: a query vector $e_u$ in retrieval, a conditional embedding $c=\{s,z\}$ in protein diffusion, a prompt-embedding tensor $E=(e_1,\dots,e_n)$ in LLM jailbreaking, or a timestep-embedding sequence $E_{ts}$ in time-series QA [2603.09185] [2602.05285] [2604.24983] [2606.18986]. Second, most surrounding parameters remain frozen or minimally modified. The retrieval method keeps the encoder frozen and optimizes only the current query embedding; EmbedOpt steers a pretrained diffusion prior at inference time; PEO does not fine-tune model weights; CADE trains lightweight projection components and uses LoRA for the LLM branch while freezing text anchors for the contrastive branch [2603.09185] [2602.05285] [2604.24983] [2606.18986].

The objective is likewise expressed in embedding-native terms. Negation-aware retrieval uses a quadratic contrastive objective that attracts the optimized query embedding toward positive sub-query embeddings, repels it from negative sub-query embeddings, and anchors it to the original query embedding:
$$
L(e_u)=\lambda_p \frac{1}{K}\sum_{i=1}^{K}\|e_u-e_{p_i}\|^2-\lambda_n \frac{1}{M}\sum_{j=1}^{M}\|e_u-e_{n_j}\|^2+\lambda_o\|e_u-e_o\|^2.
$$
Protein DEO maximizes an experimental reward through the denoiser output by solving a local surrogate
$$
\max_c R(\hat{x}_\theta(x_t,c,\sigma_t)),
$$
then denoising with the updated embedding. PEO uses teacher-forced cross-entropy on a structured continuation target plus an $L_2$ anchor to keep optimized prompt embeddings close to their originals:
$$
L(E;y)=L_{CE}(y \mid \mathrm{Concat}(E,E_y))+\lambda \|E-E^{(0)}\|_2^2.
$$
CADE combines autoregressive cross-entropy with a one-directional supervised contrastive loss against frozen class-name text anchors [2603.09185] [2602.05285] [2604.24983] [2606.18986].

A plausible implication is that DEO methods are most attractive when the embedding space already encodes domain structure that is hard to express through discrete or coordinate-level edits. The retrieval paper relies on include/exclude semantics in a pretrained encoder space; EmbedOpt relies on conditional embeddings that encode sequence and coevolutionary signals; PEO exploits the fact that LLMs condition on actual continuous prompt embeddings even when the visible prompt string is unchanged; CADE relies on the LLM input space as the target representation for numerical timesteps [2603.09185] [2602.05285] [2604.24983] [2606.18986].

## 3. Negation-aware and dense retrieval

In "DEO: Training-Free Direct Embedding Optimization for Negation-Aware Retrieval" [2603.09185], DEO is a training-free query-time method for text and multimodal retrieval. A user query is decomposed into positive components $P=\{p_i\}$ and negative components $N=\{n_j\}$ by an LLM-based parser; the reported default parser is GPT-4.1-nano at temperature $0.1$, and Qwen2.5-1.5B-Instruct was also tested. The base embedding models are frozen: BGE-small-en-v1.5, BGE-large-en-v1.5, and BGE-M3 for text, and OpenAI CLIP, CLIP-laion400m, CLIP-datacomp, and NegCLIP for multimodal retrieval. FAISS is used for indexing, cosine similarity for scoring, and DEO replaces the raw query embedding with the optimized one before nearest-neighbor search [2603.09185].

The method admits both iterative and closed-form optimization. With $\mu_p=\frac{1}{K}\sum_i e_{p_i}$ and $\mu_n=\frac{1}{M}\sum_j e_{n_j}$, the stationary point is
$$
e_u^*=\frac{\lambda_p\mu_p-\lambda_n\mu_n+\lambda_o e_o}{\lambda_p-\lambda_n+\lambda_o}
$$
when $\lambda_p-\lambda_n+\lambda_o>0$. The paper also reports Adam optimization for $20$ steps by default, with $20$–$50$ typically enough and mild degradation beyond approximately $100$ steps. On NegConstraint with BGE-large-en-v1.5, DEO improves MAP from $0.6299$ to $0.7327$ and nDCG@10 from $0.7139$ to $0.7877$, corresponding to gains of $+0.1028$ MAP@100 and $+0.0738$ nDCG@10. On COCO-Neg, Recall@5 for OpenAI CLIP increases from $0.4792$ to $0.5392$, reported as $+6.00\%$ [2603.09185].

A related but terminologically distinct dense-retrieval baseline is "Query Encoder Distillation via Embedding Alignment is a Strong Baseline Method to Boost Dense Retriever Online Efficiency" [2306.11550]. That work does not use the term DEO, but it directly regresses a student query encoder to teacher query embeddings with an MSE objective, keeps the teacher document index fixed, and thereby instantiates a pure embedding-alignment regime. The reported result is that a $2$-layer BERT-based query encoder retains $92.5\%$ of full dual-encoder BEIR performance on average, while a $4$-layer student retains $96.2\%$; training is unsupervised over approximately $8$ million MS MARCO queries with AdamW, batch size $128$, learning rate $10^{-4}$, $1000$ warm-up steps, and $1$ epoch [2306.11550]. Taken together, these papers show two distinct retrieval uses of direct embedding control: single-query optimization at inference time and student-space alignment during distillation.

## 4. Inference-time steering of generative and conditional models

"Robust Inference-Time Steering of Protein Diffusion Models via Embedding Optimization" [2602.05285] defines DEO as an inference-time steering paradigm for conditional diffusion models. Its concrete instantiation, EmbedOpt, updates the conditional embedding space rather than the coordinates directly. For AlphaFold 3–style sequence-to-structure models, the optimized condition is $c=\{s,z\}$, where $s \in \mathbb{R}^{L \times C_s}$ is a single embedding of per-residue features and $z \in \mathbb{R}^{L \times L \times C_z}$ is a pair embedding of residue-pair features. At each diffusion step, EmbedOpt computes $\hat{x}_0=\hat{x}_\theta(x_t,c_t,\sigma_t)$, backpropagates a differentiable reward $R(\hat{x}_0)$ to $s_t$ and $z_t$, RMS-normalizes the gradients separately, updates the embeddings by a single gradient-ascent step, and then performs the denoising update with the modified condition [2602.05285].

The paper contrasts this embedding-space steering with coordinate-space posterior sampling such as DPS. A first-order Taylor approximation is used to argue that EmbedOpt replaces the DPS-style pull-back with an embedding-space preconditioning term $J_cJ_c^\top \nabla_{\hat{x}_0}R(\hat{x}_0)$, which the paper associates with smoother and better-conditioned updates. Under local smoothness and small-step assumptions, it also gives an informal monotone surrogate-improvement guarantee. Empirically, the method is evaluated on two benchmarks: cryo-electron microscopy map fitting over $77$ proteins and distance-constrained structure determination over $24$ multi-domain proteins. On the hard cryo-EM target 8H1I, the prior has map correlation coefficient $0.42$, DPS reaches $0.58$, and EmbedOpt reaches $0.93$. Across hyperparameters spanning approximately $0.01$ to $1.0$, EmbedOpt shows a broad plateau in map-fitting performance and MolProbity quality, whereas DPS exhibits a brittle sweet spot; EmbedOpt also remains effective when diffusion steps are reduced from $200$ to $50$, corresponding to an approximately $4\times$ speedup [2602.05285].

This usage of DEO is notable because it treats the conditioning embedding as a means of “moving the prior” rather than overpowering it. The paper argues that when targets lie in low-density regions of the pretrained prior, coordinate-space guidance must use aggressive weighting and can leave the learned manifold, whereas embedding optimization shifts the model’s conditional preferences so that subsequent denoising steps naturally move toward higher-likelihood structures while remaining closer to the manifold of valid proteins [2602.05285].

## 5. Prompt embedding optimization in aligned LLMs

"Adaptive Prompt Embedding Optimization for LLM Jailbreaking" [2604.24983] presents Prompt Embedding Optimization (PEO) as a concrete realization of DEO in a white-box jailbreak setting. Instead of appending discrete adversarial suffixes or soft prompts, PEO directly optimizes the continuous embeddings of the original prompt tokens. The attacker has white-box access to tokenizer, embedding layer, forward pass, and gradients with respect to prompt-token embeddings, but model weights are not fine-tuned and each prompt is attacked independently. The optimization target is a teacher-forced structured continuation associated with the harmful behavior, used as an optimization scaffold rather than as the evaluation criterion [2604.24983].

The method optimizes prompt embeddings $E=(e_1,\dots,e_n)$ with Adam at learning rate $3\times 10^{-4}$, gradient clipping norm $5$, and $T=100$ iterations per pass, using the anchored cross-entropy loss
$$
L(E;y)=L_{CE}(y \mid \mathrm{Concat}(E,E_y))+\lambda \|E-E^{(0)}\|_2^2,
$$
with $\lambda=10^{-4}$ in final runs. Later passes use structured continuation targets augmented with heuristic composite response scaffolds, and only prompts that fail advance to the next pass. The schedule cold-starts every pass from $E^{(0)}$, which the paper motivates as a way to escape failed basins, keep the $L_2$ anchor referenced to the original prompt, and bound cumulative drift. For text-level reporting, optimized embeddings are projected to the nearest vocabulary token by cosine similarity; empirically, perturbations remain within the Voronoi cell of the original token for all evaluated models, so nearest-token projection recovers the exact original prompt text, reported as $0\%$ text change in every evaluated cell [2604.24983].

Evaluation uses AdvBench and HarmBench text-test against Vicuna-7B-v1.3, Llama-2-7B-Chat, Qwen3-4B-Instruct, and Llama-3.2-3B-Instruct. The primary metric is ASR-Judge, requiring both GPT-5.4 and Claude Opus 4.6 to judge the response harmful, with overall micro-averaged agreement across valid rows of $88\%$. On AdvBench ASR-Judge, PEO reports $78.65\%$ on Vicuna, $71.73\%$ on Llama-2, $72.69\%$ on Qwen3, and $75.58\%$ on Llama-3.2, outperforming nanoGCG, SPT, and BEAST in all four model cells; on HarmBench text-test, it reports $49.69\%$, $45.62\%$, $37.50\%$, and $50.00\%$, again leading every model–benchmark cell [2604.24983]. The paper also documents substantial divergence between ASR-Match and ASR-Judge, including false positives from off-topic substitution, degenerate repetition, cross-lingual refusals, and fictional framing, and false negatives caused by safety/legal words triggering substring heuristics despite harmful content [2604.24983].

The significance of this result lies in a counterintuitive property emphasized by the paper: even when the visible prompt string is preserved exactly after nearest-token projection, the model still conditions on the actual continuous embeddings at inference. Small but targeted perturbations to those embeddings can therefore alter early hidden states and attention patterns enough to change the autoregressive continuation, while leaving the user-visible text unchanged [2604.24983].

## 6. Direct timestep embedding for time-series question answering

"Beyond Tokenization: Direct Timestep Embedding and Contrastive Alignment for Time-Series Question Answering" [2606.18986] treats DEO as the direct optimization of timestep embeddings in the LLM input space, bypassing both text tokenization of raw numbers and patch/window encoders. In CADE, each timestep is mapped first by a point-wise linear encoder to a $d_{ts}$-dimensional feature and then by a two-layer MLP projector with GELU and LayerNorm to the LLM embedding dimension $d_{llm}$. For a univariate series, the paper writes
$$
h_t=W_{lin}x'_t+b_{lin},
$$
followed by
$$
\tilde{z}_t=\mathrm{GELU}(W_1h_t+b_1), \qquad z_t=\mathrm{LayerNorm}(W_2\tilde{z}_t+b_2).
$$
The sequence $E_{ts}=\{z_1,\dots,z_T\}$ is then assigned exact timestep positions in the LLM, preserving index-level addressability without patching or padding [2606.18986].

CADE supplements the direct embedding pathway with a one-directional supervised contrastive loss on classification samples. Lowercased class names are tokenized, their frozen LLM input embeddings are averaged into anchors $a_c$, timestep embeddings are mean-pooled to $\bar{z}$, and a memory queue stores previous $(\bar{z}^{(k)},y^{(k)})$ pairs. The supervised contrastive loss aligns current time-series representations with anchors for the same class while keeping the anchors and LLM embedding layer frozen. The total objective is
$$
L=L_{ce}+\lambda_\tau L_{sc},
$$
with $\lambda=0.1$ reported as the best overall balance in ablation and a memory bank size of $512$ robust across $32$ to $2048$. The implementation uses Qwen-3-0.6B with LoRA fine-tuning, $d_{ts}=384$, learning rate $5\times 10^{-5}$, cosine scheduling, batch size $32$, training for $2000$ steps on a single NVIDIA A100 40GB [2606.18986].

On the Time-MQA benchmark, CADE reports improvements over BPE tokenization across six tasks. Selected results include forecasting FCR $0.598$ versus $0.46$, forecasting Own-MSE $296{,}897$ versus $440{,}626$, imputation FCR $0.785$ versus $0.65$, imputation Own-MSE $25{,}210$ versus $2{,}399{,}043$, anomaly detection $0.8625$ versus $0.5975$, classification $0.8075$ versus $0.72$, true/false $0.7675$ versus $0.6775$, and MCQ $0.5315$ versus $0.471$ [2606.18986]. Against patch-based ITFormer, the paper highlights that ITFormer had FCR $=0$ on forecasting and imputation, which it interprets as evidence for the value of per-timestep indexability. An ablation further reports that even a Frozen Random Linear encoder outperformed BPE on every task, suggesting that the continuous-token interface itself is a major driver of gains [2606.18986].

In this setting, DEO is not a query-time optimization of a single instance but a representational design principle and training regime. The paper’s central claim is that direct embedding of numerical timesteps preserves metric structure and exact temporal indexing that BPE and patch compression discard [2606.18986].

## 7. Comparative properties, limitations, and acronym collisions

Across these instantiations, DEO is associated with several recurring advantages. Query-time retrieval DEO requires no model updates and no index changes; protein DEO is reported to be more robust than DPS across hyperparameters spanning two orders of magnitude and to permit fewer diffusion steps; prompt DEO avoids the combinatorial token space and preserves the visible prompt string exactly after nearest-token projection; CADE avoids tokenization and patching while preserving exact index-level access [2603.09185] [2602.05285] [2604.24983] [2606.18986]. This suggests that direct optimization becomes especially useful when the native embedding space already captures the relevant inductive bias better than the baseline action space.

The limitations are equally domain-specific. Negation-aware retrieval depends on decomposition quality and can over-repel with very large $\lambda_n$ or too many optimization steps; the paper reports degradation beyond approximately $100$ steps and notes difficulties with implicit negation and entangled concepts [2603.09185]. EmbedOpt requires a differentiable reward and a differentiable path from embeddings to denoiser output, and the paper notes that shallow MSAs or targets outside the training distribution can lead to reward overfitting with compromised physics [2602.05285]. PEO requires white-box access to embeddings and gradients, and residual failure modes include empty formatting, off-topic substitutions, creative-genre reframing, and cross-lingual refusals [2604.24983]. CADE inherits the $O(T^2)$ attention cost of native-resolution LLM processing and still trails a larger instruction-tuned LLM on format compliance and reasoning-heavy MCQ [2606.18986].

A final source of confusion is nomenclatural rather than methodological. "Dimer-Enhanced Optimization: A First-Order Approach to Escaping Saddle Points in Neural Network Training" uses the acronym DEO for a periodic dimer rotation and projected-gradient scheme over model parameters $\theta$, with one extra gradient evaluation every $f$ steps and default settings such as $\eta_{rot}=10^{-3}$ and $\alpha=5.0$ [2507.19968]. That method does not introduce or use the expansion “Direct Embedding Optimization.” For encyclopedia purposes, the acronym therefore names two separate research lines: one centered on direct manipulation of embedding variables, and another centered on first-order curvature estimation for training dynamics [2507.19968].

Source: https://www.emergentmind.com/topics/direct-embedding-optimization-deo