EmbedOpt: Optimizing Embedding Spaces

• EmbedOpt is an embedding-space optimization framework applied in both CTR prediction and protein diffusion, treating embeddings as primary, trainable variables.
• In CTR prediction, it combines learnable row pruning and evolutionary search to build dense, memory-efficient tables while maintaining high accuracy.
• In protein diffusion, it steers conditional embeddings via gradient updates to maximize experimental likelihoods, enhancing robustness in structure prediction.

Searching arXiv for the cited papers to ground the article in the current literature. EmbedOpt denotes embedding-space optimization frameworks that appear in two distinct settings in the supplied arXiv literature. In click-through rate prediction, the query label overlaps with "OptEmbed: Learning Optimal Embedding Table for Click-through Rate Prediction," which learns a compact and hardware-friendly embedding table by combining feature-value pruning with field-wise dimension search (Lyu et al., 2022). In protein diffusion, "Robust Inference-Time Steering of Protein Diffusion Models via Embedding Optimization" uses EmbedOpt as an inference-time procedure that updates conditional embeddings to maximize experimental likelihoods while keeping the pretrained denoiser fixed (Li et al., 5 Feb 2026). The two usages share a common premise: the embedding representation is treated as the primary optimization variable rather than as a static by-product of model design.

1. Scope and nomenclature

The supplied literature uses closely related names for two different technical programs. The CTR paper is titled "OptEmbed" but is explicitly identified as "EmbedOpt" in the query; the protein diffusion paper uses "EmbedOpt" as the method name. In both cases, the optimization target is an embedding object, but the object itself, the optimization objective, and the deployment regime differ substantially.

Usage in the supplied literature	Domain	Central objective
OptEmbed	CTR prediction	Learn an optimal embedding table with pruning and field-wise dimension search
EmbedOpt	Protein diffusion	Steer diffusion inference by optimizing conditional embeddings against experimental likelihoods

A common misconception is to treat these two methods as a single algorithmic family with interchangeable mechanics. The supplied descriptions do not support that reading. OptEmbed operates on embedding tables indexed by categorical feature values and is concerned with model performance, memory usage, and dense deployment (Lyu et al., 2022). EmbedOpt operates on single-residue and pairwise conditioning embeddings in AlphaFold 3–style diffusion models and is concerned with posterior-like steering under experimental constraints (Li et al., 5 Feb 2026).

2. OptEmbed as optimal embedding-table learning for CTR prediction

OptEmbed formulates embedding-table learning as a constrained optimization problem over field-wise tables $$E_{(1)} \in \mathbb{R}^{|f_{(1)}| \times D_{(1)}}, \ldots, E_{(n)} \in \mathbb{R}^{|f_{(n)}| \times D_{(n)}}$$ and network weights $W$, with the objective of minimizing standard log-loss on a CTR dataset $\mathcal{D}$ (Lyu et al., 2022). The formulation is

$$\min_{\{E_{(i)}\}, W} \; L_{CE}(\mathcal{D} \mid \{E_{(1)}, \ldots, E_{(n)}\}, W)$$

subject to

$$\sum_{i=1}^n |f_{(i)}| \le |f|,\qquad D_{(i)} \le D \;\; \text{for all } i=1 \ldots n.$$

The loss is

$$L_{CE}(\mathcal{D}) = -\frac{1}{|\mathcal{D}|}\sum_{(x,y)\in\mathcal{D}} [y \log \hat y + (1-y)\log(1-\hat y)],$$

with

$$\hat y = \sigma\!\left(F_{\text{interaction}}\!\left(\text{concat}_i(E_{(i)} x_{(i)})\right)\right).$$

The paper states that this constrained problem simultaneously prunes redundant rows, assigns flexible dimensions $D_{(i)}$, and respects hardware-friendly dense layout. That combination distinguishes it from prior approaches that assign various embedding dimensions for feature fields, reduce the number of embeddings, or mask embedding-table parameters. The paper argues that those alternatives either still require a large amount of memory, suffer performance degradation, or produce sparse tables that are hard to deploy (Lyu et al., 2022).

A central mechanism is learnable row pruning. OptEmbed introduces an embedding matrix $E \in \mathbb{R}^{|f| \times D}$, a binary mask $m_e \in \{0,1\}^{|f|}$, and a field-wise threshold vector $W$0. For feature-value row $W$1, belonging to field $W$2,

$W$3

where $W$4 if $W$5 and $W$6 otherwise. The effective table is $W$7, so a row is zeroed if $W$8. Since the unit-step is non-differentiable, the method uses a "long-tail estimator" $W$9 during back-propagation. The paper describes the resulting second term as a "structure gradient" that moves thresholds and recovers rows if needed (Lyu et al., 2022).

To encourage pruning, the method adds the sparsity regularizer

$\mathcal{D}$0

and optimizes the joint objective

$\mathcal{D}$1

where $\mathcal{D}$2 trades off accuracy versus sparsity.

3. Supernet search, evolutionary selection, and dense deployment in OptEmbed

OptEmbed treats each field’s embedding dimension $\mathcal{D}$3 as a network choice and avoids exhaustive search over $\mathcal{D}$4 by constructing a one-shot supernet with ordinal weight-sharing (Lyu et al., 2022). A single super-embedding $\mathcal{D}$5 is maintained, and a per-field dimension mask $\mathcal{D}$6 selects the first $\mathcal{D}$7 columns for field $\mathcal{D}$8 via

$\mathcal{D}$9

During supernet training, each field’s $$\min_{\{E_{(i)}\}, W} \; L_{CE}(\mathcal{D} \mid \{E_{(1)}, \ldots, E_{(n)}\}, W)$$0 is sampled uniformly from $$\min_{\{E_{(i)}\}, W} \; L_{CE}(\mathcal{D} \mid \{E_{(1)}, \ldots, E_{(n)}\}, W)$$1 for every minibatch. The stated purpose is to equally train all candidate architectures, so architecture-related parameters and learnable thresholds are trained simultaneously in one supernet. The training loop updates $$\min_{\{E_{(i)}\}, W} \; L_{CE}(\mathcal{D} \mid \{E_{(1)}, \ldots, E_{(n)}\}, W)$$2, $$\min_{\{E_{(i)}\}, W} \; L_{CE}(\mathcal{D} \mid \{E_{(1)}, \ldots, E_{(n)}\}, W)$$3, and $$\min_{\{E_{(i)}\}, W} \; L_{CE}(\mathcal{D} \mid \{E_{(1)}, \ldots, E_{(n)}\}, W)$$4 using the loss $$\min_{\{E_{(i)}\}, W} \; L_{CE}(\mathcal{D} \mid \{E_{(1)}, \ldots, E_{(n)}\}, W)$$5, where $$\min_{\{E_{(i)}\}, W} \; L_{CE}(\mathcal{D} \mid \{E_{(1)}, \ldots, E_{(n)}\}, W)$$6 (Lyu et al., 2022).

After supernet training, the method fixes $$\min_{\{E_{(i)}\}, W} \; L_{CE}(\mathcal{D} \mid \{E_{(1)}, \ldots, E_{(n)}\}, W)$$7 and $$\min_{\{E_{(i)}\}, W} \; L_{CE}(\mathcal{D} \mid \{E_{(1)}, \ldots, E_{(n)}\}, W)$$8 and applies evolutionary search over $$\min_{\{E_{(i)}\}, W} \; L_{CE}(\mathcal{D} \mid \{E_{(1)}, \ldots, E_{(n)}\}, W)$$9 to find a dimension mask $$\sum_{i=1}^n |f_{(i)}| \le |f|,\qquad D_{(i)} \le D \;\; \text{for all } i=1 \ldots n.$$0 that minimizes validation loss. The population consists of $$\sum_{i=1}^n |f_{(i)}| \le |f|,\qquad D_{(i)} \le D \;\; \text{for all } i=1 \ldots n.$$1 random masks. Each mask is evaluated by validation AUC, the top-$$\sum_{i=1}^n |f_{(i)}| \le |f|,\qquad D_{(i)} \le D \;\; \text{for all } i=1 \ldots n.$$2 masks are retained, and new candidates are generated by crossover and mutation. The paper reports that each generation costs $$\sum_{i=1}^n |f_{(i)}| \le |f|,\qquad D_{(i)} \le D \;\; \text{for all } i=1 \ldots n.$$3 for inference only, stops after $$\sum_{i=1}^n |f_{(i)}| \le |f|,\qquad D_{(i)} \le D \;\; \text{for all } i=1 \ldots n.$$4 generations, uses typical population size $$\sum_{i=1}^n |f_{(i)}| \le |f|,\qquad D_{(i)} \le D \;\; \text{for all } i=1 \ldots n.$$5, and therefore requires overall cost of approximately $$\sum_{i=1}^n |f_{(i)}| \le |f|,\qquad D_{(i)} \le D \;\; \text{for all } i=1 \ldots n.$$6–$$\sum_{i=1}^n |f_{(i)}| \le |f|,\qquad D_{(i)} \le D \;\; \text{for all } i=1 \ldots n.$$7 fast forward-passes (Lyu et al., 2022).

Deployment is organized around dense extraction rather than runtime sparsity. After search, the method obtains $$\sum_{i=1}^n |f_{(i)}| \le |f|,\qquad D_{(i)} \le D \;\; \text{for all } i=1 \ldots n.$$8 and $$\sum_{i=1}^n |f_{(i)}| \le |f|,\qquad D_{(i)} \le D \;\; \text{for all } i=1 \ldots n.$$9, then physically slices field-wise dense matrices by collecting feature values with $$L_{CE}(\mathcal{D}) = -\frac{1}{|\mathcal{D}|}\sum_{(x,y)\in\mathcal{D}} [y \log \hat y + (1-y)\log(1-\hat y)],$$0 and keeping only the first $$L_{CE}(\mathcal{D}) = -\frac{1}{|\mathcal{D}|}\sum_{(x,y)\in\mathcal{D}} [y \log \hat y + (1-y)\log(1-\hat y)],$$1 columns for each surviving row. The resulting production tables are $$L_{CE}(\mathcal{D}) = -\frac{1}{|\mathcal{D}|}\sum_{(x,y)\in\mathcal{D}} [y \log \hat y + (1-y)\log(1-\hat y)],$$2 small dense matrices $$L_{CE}(\mathcal{D}) = -\frac{1}{|\mathcal{D}|}\sum_{(x,y)\in\mathcal{D}} [y \log \hat y + (1-y)\log(1-\hat y)],$$3. The paper states that no additional indirection or sparsity metadata is needed and that standard GPU kernels for dense lookup and gather remain unchanged (Lyu et al., 2022). This directly addresses the objection that pruning embedding parameters typically leads to sparse embedding tables that are hard to deploy.

4. EmbedOpt as inference-time steering in protein diffusion models

The protein-diffusion EmbedOpt method is defined for AlphaFold 3–style diffusion models whose denoiser $$L_{CE}(\mathcal{D}) = -\frac{1}{|\mathcal{D}|}\sum_{(x,y)\in\mathcal{D}} [y \log \hat y + (1-y)\log(1-\hat y)],$$4 accepts two kinds of conditioning embeddings: single-residue embeddings $$L_{CE}(\mathcal{D}) = -\frac{1}{|\mathcal{D}|}\sum_{(x,y)\in\mathcal{D}} [y \log \hat y + (1-y)\log(1-\hat y)],$$5 and pairwise embeddings $$L_{CE}(\mathcal{D}) = -\frac{1}{|\mathcal{D}|}\sum_{(x,y)\in\mathcal{D}} [y \log \hat y + (1-y)\log(1-\hat y)],$$6 (Li et al., 5 Feb 2026). These embeddings are produced by the PairFormer or Evoformer module from one-hot or learned sequence representations, multiple-sequence alignment statistics, and optional template features; template features, if used, are held fixed. Preprocessing consists of building an MSA, feeding MSA plus sequence through transformer blocks to yield $$L_{CE}(\mathcal{D}) = -\frac{1}{|\mathcal{D}|}\sum_{(x,y)\in\mathcal{D}} [y \log \hat y + (1-y)\log(1-\hat y)],$$7 and $$L_{CE}(\mathcal{D}) = -\frac{1}{|\mathcal{D}|}\sum_{(x,y)\in\mathcal{D}} [y \log \hat y + (1-y)\log(1-\hat y)],$$8, concatenating them into $$L_{CE}(\mathcal{D}) = -\frac{1}{|\mathcal{D}|}\sum_{(x,y)\in\mathcal{D}} [y \log \hat y + (1-y)\log(1-\hat y)],$$9, and applying no further normalization at inference time.

The method is motivated by inverse problems in which one seeks biomolecular conformations that are both physically plausible and consistent with experimental measurements. Let $$\hat y = \sigma\!\left(F_{\text{interaction}}\!\left(\text{concat}_i(E_{(i)} x_{(i)})\right)\right).$$0 denote atomic coordinates, let $$\hat y = \sigma\!\left(F_{\text{interaction}}\!\left(\text{concat}_i(E_{(i)} x_{(i)})\right)\right).$$1 denote experimental data, let $$\hat y = \sigma\!\left(F_{\text{interaction}}\!\left(\text{concat}_i(E_{(i)} x_{(i)})\right)\right).$$2 be the pretrained diffusion prior, and define the reward

$$\hat y = \sigma\!\left(F_{\text{interaction}}\!\left(\text{concat}_i(E_{(i)} x_{(i)})\right)\right).$$3

EmbedOpt seeks to adapt $$\hat y = \sigma\!\left(F_{\text{interaction}}\!\left(\text{concat}_i(E_{(i)} x_{(i)})\right)\right).$$4 so that samples from $$\hat y = \sigma\!\left(F_{\text{interaction}}\!\left(\text{concat}_i(E_{(i)} x_{(i)})\right)\right).$$5 maximize $$\hat y = \sigma\!\left(F_{\text{interaction}}\!\left(\text{concat}_i(E_{(i)} x_{(i)})\right)\right).$$6. The paper notes an ideal posterior over embeddings,

$$\hat y = \sigma\!\left(F_{\text{interaction}}\!\left(\text{concat}_i(E_{(i)} x_{(i)})\right)\right).$$7

but does not optimize this expression directly at inference time. Instead it uses a greedy, step-wise surrogate (Li et al., 5 Feb 2026).

At diffusion step $$\hat y = \sigma\!\left(F_{\text{interaction}}\!\left(\text{concat}_i(E_{(i)} x_{(i)})\right)\right).$$8, the experimental negative log-likelihood is

$$\hat y = \sigma\!\left(F_{\text{interaction}}\!\left(\text{concat}_i(E_{(i)} x_{(i)})\right)\right).$$9

and the surrogate embedding loss is

$D_{(i)}$0

If

$D_{(i)}$1

then the update is written as reward ascent,

$D_{(i)}$2

or equivalently as gradient descent on $D_{(i)}$3,

$D_{(i)}$4

In practice, the update is normalized by the RMS of the gradient separately for $D_{(i)}$5 and $D_{(i)}$6, with constant base learning rate $D_{(i)}$7 and

$D_{(i)}$8

The supplied pseudocode specifies the per-step sequence: denoise to obtain $D_{(i)}$9, compute gradients with respect to $E \in \mathbb{R}^{|f| \times D}$0 and $E \in \mathbb{R}^{|f| \times D}$1, RMS-normalize them, update the embeddings, and then apply the diffusion step

$E \in \mathbb{R}^{|f| \times D}$2

optionally with a step-scale factor (Li et al., 5 Feb 2026).

A common misconception is to treat this procedure as conventional coordinate-based posterior sampling. The paper explicitly presents EmbedOpt as an alternative inference-time approach that steers diffusion models in the conditional embedding space, and reports comparisons against the coordinate-based posterior sampling method DPS rather than subsuming it into DPS (Li et al., 5 Feb 2026).

5. Empirical findings

For CTR prediction, OptEmbed is evaluated on Criteo, Avazu, and KDD 2012, using AUC, LogLoss, and Sparsity $E \in \mathbb{R}^{|f| \times D}$3, and comparing against original CTR models such as DeepFM, DCN, FNN, and IPNN, as well as AutoDim, AutoField, QR, and PEP (Lyu et al., 2022). In the DeepFM example on Criteo, the reported AUC values are 0.8104 for the original model, 0.8093 for AutoDim, 0.8101 for AutoField, 0.8084 for QR, 0.7980 for PEP, and 0.8105 for OptEmbed, with OptEmbed achieving 96.8% sparsity. The paper summarizes this as 0.8105 on Criteo while pruning 96.8% embeddings and reports +0.1–0.15 ppt AUC with approximately 50–62% sparsity on Avazu and KDD12. Ablation on DeepFM at Criteo reports OptEmbed-E with AUC 0.8104 and sparsity 62.7%, OptEmbed-D with AUC 0.8103 and sparsity 55.5%, and full OptEmbed with AUC 0.8105 and sparsity 96.8%, which the paper interprets as showing synergy between pruning and dimension search. It also reports that without re-training final $E \in \mathbb{R}^{|f| \times D}$4, AUC drops by approximately 0.002–0.003. Training time including supernet training, evolutionary search, and re-training is about 1.2× that of the original model. At batch size 2048, inference latency is 2.7 ms/batch for the original model, 3.5 ms for PEP, and 1.6 ms for OptEmbed (Lyu et al., 2022).

For protein diffusion, EmbedOpt is validated on two benchmarks: cryo-electron microscopy map fitting with 77 test systems and distance-constraint folding with 24 systems and $E \in \mathbb{R}^{|f| \times D}$5 pairs (Li et al., 5 Feb 2026). In cryo-EM map fitting, difficulty is defined as $E \in \mathbb{R}^{|f| \times D}$6. Across an $E \in \mathbb{R}^{|f| \times D}$7 sweep, the best-achieved final map correlation $E \in \mathbb{R}^{|f| \times D}$8 on easy systems is approximately $E \in \mathbb{R}^{|f| \times D}$9 for DPS and $m_e \in \{0,1\}^{|f|}$0 for EmbedOpt; on hard systems it is approximately $m_e \in \{0,1\}^{|f|}$1 for DPS and $m_e \in \{0,1\}^{|f|}$2 for EmbedOpt. Averaged over all systems, the mean $m_e \in \{0,1\}^{|f|}$3 is 0.86 for DPS and 0.91 for EmbedOpt. MolProbity scores remain below 2.0 for EmbedOpt even at high $m_e \in \{0,1\}^{|f|}$4, whereas DPS blows up above $m_e \in \{0,1\}^{|f|}$5. In distance-constraint folding, the reward is

$m_e \in \{0,1\}^{|f|}$6

and both methods reach median 20 satisfied constraints out of 20, with comparable minima of total distance violation near 0; MolProbity is reported as below 1.8 for EmbedOpt and about 1.5 for DPS in the sweet spot but above 3.0 outside it (Li et al., 5 Feb 2026).

The robustness results are particularly prominent in the protein setting. The paper sweeps $m_e \in \{0,1\}^{|f|}$7 from 0.01 to 1.0, a 100× range. For cryo-EM, DPS peaks at $m_e \in \{0,1\}^{|f|}$8 and collapses for $m_e \in \{0,1\}^{|f|}$9 or $W$00, while EmbedOpt maintains a flat plateau of $W$01 for $W$02. In the distance benchmark, EmbedOpt maintains 20/20 constraints for $W$03, whereas DPS does so only in $W$04. When reducing the number of diffusion steps under the constraint $W$05, EmbedOpt retains 20, 20, 18, and 12 satisfied constraints at $W$06, while DPS retains 20, 15, 5, and 0. MolProbity remains below 2.0 for EmbedOpt down to $W$07, whereas DPS exceeds 4.0 for $W$08 (Li et al., 5 Feb 2026). This suggests that embedding-space steering can be more tolerant to aggressive inference compression than coordinate-based guidance.

6. Engineering considerations, limitations, and broader implications

The deployment logic of OptEmbed is centered on compact dense tables. The paper states that memory becomes $W$09 bytes, up to 97% smaller; standard GPU kernels remain unchanged; and end-to-end latency is reduced by 30–50% on all benchmarks (Lyu et al., 2022). The main engineering claim is therefore not only parameter reduction but compatibility with existing dense inference kernels. A related misconception is that any pruning-based compression will necessarily incur sparse-runtime overhead. The OptEmbed formulation explicitly avoids that outcome by extracting dense field-wise matrices after search.

In the protein-diffusion setting, EmbedOpt adds one additional gradient backpropagation per diffusion step for $W$10, at roughly 1.2× the cost of unguided sampling, and does not require extra model fine-tuning or Monte Carlo chains (Li et al., 5 Feb 2026). The paper also states that it does not store the whole diffusion trajectory for backpropagation, only a single forward and backward pass per step. Stability is supported by a theoretical trust-region guarantee, identified as Proposition 3.1, which shows that with sufficiently small $W$11 and $W$12, the surrogate reward is monotonic; empirically, the method yields smooth reward ascent and avoids the oscillations seen in DPS (Li et al., 5 Feb 2026).

The stated limitations differ sharply across the two domains. In CTR prediction, the principal concern motivating OptEmbed is the tension among model performance, memory usage, and deployability, especially under vast feature vocabularies and field-wise heterogeneity (Lyu et al., 2022). In protein diffusion, the paper notes that if the experimental target lies completely outside the manifold spanned by embeddings, such as a novel fold or missing coevolution signal, EmbedOpt may overfit $W$13 to satisfy the reward while producing unphysical structures. A proposed remedy is post-hoc energy relaxation by classical force-field minimization, though the paper adds that this may not fully restore viability. Proposed future directions include an explicit embedding-space regularizer such as $W$14 or a VAE bottleneck, ensemble inference by optimizing a batch of embeddings, and a hybrid two-stage procedure that first uses EmbedOpt to shift the prior and then applies coordinate-based DPS for fine local sampling (Li et al., 5 Feb 2026).

Taken together, the supplied literature presents "EmbedOpt" not as a single canonical algorithm but as a recurring design choice: optimize embeddings directly when the embedding object is the most controllable locus of inductive bias. In CTR prediction, that choice appears as joint pruning and dimension search over categorical embedding tables (Lyu et al., 2022). In protein diffusion, it appears as inference-time steering in conditional embedding space under experimental likelihoods (Li et al., 5 Feb 2026). A plausible implication is that embedding optimization functions as a unifying abstraction across otherwise unrelated systems, with dense deployability the critical issue in recommender models and robustness of guided sampling the critical issue in biomolecular generative modeling.