Foundational Optimization Embeddings

• Foundational Optimization Embeddings are a collection of methods that optimize the embedding space itself, facilitating reuse across domains like MIP and SAT.
• They employ techniques such as graph autoencoders, optimized Gaussian samplers, and layer selection to enhance clustering, prediction, and inference performance.
• This approach challenges conventional practices by demonstrating that intermediate or deterministic representations can offer superior structural fidelity and operational efficiency over last-layer methods.

Searching arXiv for the cited topic and closely related papers. arXiv search query: "Foundational Optimization Embeddings Forge SAT Johnson-Lindenstrauss optimization embeddings" In the literature surveyed here, the phrase foundational optimization embeddings encompasses several related but non-identical practices: pre-training reusable structural representations for mixed-integer programming (MIP), transferring those representations to Boolean satisfiability (SAT), constructing deterministic Johnson–Lindenstrauss (JL) embeddings by optimizing over distributions rather than fixed matrices, selecting intermediate foundation-model layers whose frozen embeddings best support a task, and directly optimizing embeddings used for prompts, tables, or logic-grounded geometric models (Shafi et al., 28 Aug 2025, Pal et al., 16 Apr 2026, Tsikouras et al., 2024, Batra et al., 21 May 2026, Hou et al., 5 Aug 2025, Lyu et al., 2022, Kulmanov et al., 2019). Taken together, these works suggest that the topic is not a single algorithmic recipe but a family of methods in which the embedding itself—or the space from which it is extracted—is treated as the primary optimization object.

1. Conceptual scope and recurrent design pattern

A common thread across these works is the replacement of narrowly task-specific representation learning with a more general embedding substrate that can be reused, transferred, or derandomized. In Forge, the foundational object is an unsupervised, instance-level representation of MIP problems learned from static bipartite graphs without requiring optimal solutions (Shafi et al., 28 Aug 2025). In the SAT transfer study, the same pretrained representation is reused without any architectural changes or supervised fine-tuning, and the claim is that the embeddings remain meaningful even when the domain changes from optimization to decision problems (Pal et al., 16 Apr 2026). In the JL setting, the foundational move is different: the paper does not pretrain a representation on many instances, but it replaces direct optimization over projection matrices with optimization over a larger space of Gaussian samplers and then shrinks variance to zero, yielding a deterministic embedding (Tsikouras et al., 2024).

A second recurring motif is that the useful representation need not coincide with the most obvious one. The single-cell study argues that “where” to extract features matters as much as “what” the model learns, because intermediate layers can outperform final-layer embeddings and the optimum depends on task and cellular context (Civale et al., 16 Apr 2026). The LOES work makes a parallel claim at a broader level: task-relevant information is distributed non-monotonically across depth, so transfer should be posed as selecting a compact task-discriminative subspace under isotropy, redundancy, and simplicial-geometry constraints rather than simply taking the last hidden state (Batra et al., 21 May 2026).

This body of work therefore rejects several default assumptions. One is that a final layer is automatically the best embedding. Another is that direct optimization over a deterministic embedding object is necessarily viable. A third is that foundational representations must remain within the domain in which they were pretrained. Each of these assumptions is explicitly challenged in at least one of the cited papers.

2. Foundational graph embeddings for optimization instances

The clearest use of the term in combinatorial optimization is Forge, which introduces a vector-quantized graph autoencoder for MIP bipartite graphs and frames its output as a reusable instance-level representation (Shafi et al., 28 Aug 2025). A MIP of the form

$$f(x)=\min\{c^Tx\mid Ax\leq b,\ x\in \mathbb{R}^n,\ x_j\in \mathbb{Z}\ \forall j\in I\}$$

is encoded as a bipartite graph with variable nodes, constraint nodes, weighted edges for nonzero coefficients, and only static input-instance features. Constraint nodes receive 4 features, variable nodes receive 6 features, and these are padded into a 10-dimensional node-feature vector.

Forge applies two GraphSAGE layers, vector-quantizes the node embeddings into discrete codes, replaces each code by its codeword embedding, and decodes both graph structure and node features. Its training loss is

$$\mathcal{L}=\mathcal{L}_{Rec}+\mathcal{L}_{Codebook}+\mathcal{L}_{Commitment},$$

with reconstruction, codebook, and commitment terms defined over node embeddings and selected codewords. The paper explicitly interprets the codebook objective as k-means-like clustering in latent space. The central representation is not the dense readout of node states but the distribution of discrete codes across an instance. If the codebook has size $k$, the instance embedding is a $k$-dimensional count vector; the paper likens this to a vocabulary-like representation of optimization instances.

This representation is evaluated in both unsupervised and supervised settings. In the unsupervised setting, Forge is trained on 600 MIPLIB instances plus two augmented variants of each instance formed by randomly dropping 5% and 10% of constraints, for 1,800 instances total. Tested on 1,050 unseen D-MIPLIB instances spanning 21 problem type-difficulty pairs, the code-distribution embedding achieves NMI = 0.843 under $k$-means with 21 clusters, compared with 0.790 for two-hop label propagation and 0.087 for mean readout. In supervised fine-tuning, the same pretrained encoder is reused for integrality gap prediction and warm-start variable prediction, and the resulting predictions improve Gurobi and also improve PS-Gurobi after dimensionality reduction of node embeddings from 1024 to 64 dimensions via PCA (Shafi et al., 28 Aug 2025).

The practical significance is that the foundational representation is learned without dependency on solution labels, yet is later adapted to solver-facing tasks. This is a specific answer to a long-standing bottleneck in ML-for-optimization: collecting large numbers of hard-to-solve labeled instances.

3. Transfer from optimization to SAT

The SAT transfer paper extends the foundational-embedding idea beyond optimization by asking whether a MIP-pretrained representation can organize decision problems in a fully unsupervised way (Pal et al., 16 Apr 2026). The key technical bridge is a structural one: SAT formulas are recast into the same bipartite constraint–variable graph format used for MIPs. A CNF formula

$F=C_1\land\cdots\land C_m$

is encoded as a 0–1 MIP in which each clause becomes

$\sum_{t=1}^k y_{jt}\ge 1,$

with

$$y_{jt}=x_i \quad \text{if } \ell_{jt}=x_i,\qquad y_{jt}=1-x_i \quad \text{if } \ell_{jt}=\lnot x_i.$$

The paper states that this encoding preserves satisfiability exactly.

Three transfer settings are studied. Forge-Mip applies the pretrained MIP model to SAT instances with no architectural changes and no feature redesign. Forge-Mip-Sat still reuses the pretrained MIP weights but substitutes SAT-specific clause and variable descriptors. Forge-Sat keeps the same architecture but discards the MIP-pretrained weights and pretrains from scratch on SAT using SAT-specific features. The architectural core remains the same: a two-layer GraphSAGE encoder, vector quantization, decoding of node features and incident edges, and instance-level embeddings derived from the empirical distribution of codewords.

Evaluation is performed on the Hard split of G4SATBench, which contains 1400 unseen formulas total, with 200 from each problem type and a balanced SAT/UNSAT split, producing 14 ground-truth groups defined by $(\text{problem type}, \text{feasibility})$. Cluster quality is measured with Normalized Mutual Information (NMI) and Purity. The static baseline yields NMI $0.74 \pm 0.002$. Forge-Mip reaches $$\mathcal{L}=\mathcal{L}_{Rec}+\mathcal{L}_{Codebook}+\mathcal{L}_{Commitment},$$0, Forge-Mip-Sat reaches $$\mathcal{L}=\mathcal{L}_{Rec}+\mathcal{L}_{Codebook}+\mathcal{L}_{Commitment},$$1, and Forge-Sat performs best with NMI $$\mathcal{L}=\mathcal{L}_{Rec}+\mathcal{L}_{Codebook}+\mathcal{L}_{Commitment},$$2 and Purity $$\mathcal{L}=\mathcal{L}_{Rec}+\mathcal{L}_{Codebook}+\mathcal{L}_{Commitment},$$3. The paper reports that Forge-Sat comes closest to 14 clean clusters, with only $$\mathcal{L}=\mathcal{L}_{Rec}+\mathcal{L}_{Codebook}+\mathcal{L}_{Commitment},$$4-Clique SAT/UNSAT overlapping, and that it separates satisfiability within the same problem type, such as PS SAT vs. UNSAT (Pal et al., 16 Apr 2026).

A common misconception addressed here is that a foundational optimization representation must remain MIP-specific. The reported transfer results do not eliminate domain mismatch, but they do show that a representation pretrained on optimization data alone can transfer to SAT without supervision.

4. Derandomizing embeddings by optimizing over samplers

A different foundational strand is presented by “Optimization Can Learn Johnson Lindenstrauss Embeddings”, which asks whether JL embeddings must remain randomized objects or can instead be learned by optimization directly from data (Tsikouras et al., 2024). The paper first shows why the naïve approach fails. If one minimizes the maximum-distortion objective

$$\mathcal{L}=\mathcal{L}_{Rec}+\mathcal{L}_{Codebook}+\mathcal{L}_{Commitment},$$5

the landscape over projection matrices is non-convex and contains bad local minima. The authors construct matrices $$\mathcal{L}=\mathcal{L}_{Rec}+\mathcal{L}_{Codebook}+\mathcal{L}_{Commitment},$$6 that are strict local minima reachable from the origin, with constant distortion $$\mathcal{L}=\mathcal{L}_{Rec}+\mathcal{L}_{Codebook}+\mathcal{L}_{Commitment},$$7, even though there exist embeddings with distortion $$\mathcal{L}=\mathcal{L}_{Rec}+\mathcal{L}_{Codebook}+\mathcal{L}_{Commitment},$$8.

The workaround is to optimize over a larger space of random solution samplers. Instead of fixing $$\mathcal{L}=\mathcal{L}_{Rec}+\mathcal{L}_{Codebook}+\mathcal{L}_{Commitment},$$9, the method considers

$k$0

defines the failure-probability objective

$k$1

relaxes it to a sum over data points,

$k$2

and adds a variance penalty

$k$3

Initialization at $k$4 corresponds to i.i.d. Gaussian entries, so the JL theorem provides a good starting sampler.

The structural claim is that any point with nonzero variance is not a true local minimum: for any $k$5 and any $k$6, there exists a nearby mean $k$7 such that

$k$8

and

$k$9

This supports the paper’s reverse-diffusion interpretation: optimization moves through sampler space while progressively reducing variance. The analyzed algorithm is a deterministic second-order method, Hessian Descent, which alternates between gradient steps, negative-curvature steps, and termination at a $k$0-approximate second-order stationary point. With $k$1, the returned matrix satisfies the JL guarantee with distortion $k$2, where $k$3 (Tsikouras et al., 2024).

This paper addresses a different misconception: not that randomness is useless, but that derandomization can be achieved by directly optimizing the deterministic object of interest. The argument is that direct matrix optimization is “provably doomed for this objective,” whereas optimization over samplers avoids bad stationary points.

5. Layer selection as foundational embedding optimization

Another usage of the topic appears in foundation models whose representations evolve across depth. The single-cell study states that current benchmarks “universally extract final layer embeddings,” but its analysis of scFoundation and Tahoe-X1 shows that the optimal layer is task-dependent and context-dependent (Civale et al., 16 Apr 2026). For trajectory inference, Tahoe-X1 peaks at 60% depth, specifically layer 16 of 24, with Spearman correlation $k$4, which is 31% better than the final layer at $k$5. For perturbation response prediction, the optimum shifts across activation states: for Tahoe-X1, Rest is best at layer 1, 0% depth, $k$6; Stim8hr at layer 4, 13% depth, $k$7; and Stim48hr at layer 23, 96% depth, $k$8. The paper highlights the case in which first-layer embeddings outperform all deeper layers in quiescent cells (Civale et al., 16 Apr 2026).

The LOES paper generalizes this idea and makes it explicitly optimization-based. Layer-wise Optimal Embedding Selection (LOES) scores candidate layers by residual ridge loss, isotropy, redundancy, and simplex-like class geometry:

$k$9

Its ridge probe is

$k$0

and candidates are orthogonalized against the currently selected subspace before scoring. During downstream adaptation, GeoReg enforces isotropy and simplicial class geometry through

$k$1

Across vision, language, speech, segmentation, and multimodal settings, the paper reports that LOES consistently outperforms last-layer baselines and that gains increase with model depth (Batra et al., 21 May 2026).

These results directly contest the idea that deeper is always better or that naïve aggregation of the last few layers is sufficient. In both studies, the foundational embedding is not a fixed architectural endpoint but a selected intermediate representation whose utility depends on geometry, task, and context.

6. Other optimization-based embedding constructions

Several additional works expand the topic beyond MIP graphs and layer selection. EL Embeddings formulate the search for a vector-space interpretation of an $k$2 ontology as an optimization problem over class centers, class radii, and role translations in $k$3 (Kulmanov et al., 2019). Concepts are represented as open $k$4-balls,

$k$5

roles as translations,

$k$6

and normalized EL axioms are enforced by differentiable losses. The paper’s correctness theorem states: if $k$7 and the total normalized loss is zero, then the theory has a model. This is foundational in a model-theoretic sense: the method seeks an embedding that behaves like a model of the TBox rather than only a predictive latent representation.

OptEmbed addresses a different object, the embedding table in click-through rate prediction. It decomposes the table into field-wise tables and learns both an embedding mask and a field-wise dimension mask,

$k$8

so that redundant feature values can be pruned and embedding dimensions searched jointly in a one-shot NAS supernet (Lyu et al., 2022). Its stated requirements are no redundant embeddings, flexible dimensions across fields, and hardware compatibility. The paper argues that sparse parameter masking is hard to deploy efficiently, whereas the final OptEmbed table remains compact and dense.

jina-embeddings-v3 represents a deployment-oriented variant of foundational embedding design. It uses an XLM-RoBERTa backbone with 24 layers, ~559M base parameters and 572M with adapters, RoPE, FlashAttention 2, and support for up to 8192 input tokens (Sturua et al., 2024). Instead of a single geometry for all use cases, it offers task-specific LoRA adapters—retrieval.query, retrieval.passage, separation, classification, and text-matching—with the base model frozen during adapter training. It also uses Matryoshka Representation Learning, allowing the default 1024-dimensional output to be truncated to 768, 512, 256, 128, 64, or 32 dimensions.

EmbedGrad moves the optimization target to prompt embeddings. Starting from a human-written prompt, it extracts token embeddings from a frozen LLM, optimizes only those embeddings with gradients on labeled data, and reuses the optimized continuous prompt prefix at inference time (Hou et al., 5 Aug 2025). The model weights remain frozen, and no architectural changes or additional modules are introduced. The reported flagship result is on Math500 with Qwen2.5-Math-1.5B, where accuracy increases from 14.74% to 58.96% after prompt-embedding optimization.

7. Significance, limitations, and open directions

Across these works, foundational optimization embeddings are associated with three recurring ambitions. The first is reuse: a single representation should serve multiple tasks or domains, as in Forge’s support for clustering, warm-start prediction, and integrality-gap estimation, or SAT transfer from MIP-pretrained weights (Shafi et al., 28 Aug 2025, Pal et al., 16 Apr 2026). The second is structural alignment: embeddings are expected to preserve geometry, logic, or distortion guarantees rather than merely fit a downstream predictor, as in JL sampler optimization, EL model construction, or LOES/GeoReg layer geometry (Tsikouras et al., 2024, Kulmanov et al., 2019, Batra et al., 21 May 2026). The third is operational efficiency: several papers explicitly target reduced dependence on solver-generated labels, compact dense tables, lightweight task adapters, or dimensionality reduction without catastrophic quality loss (Shafi et al., 28 Aug 2025, Lyu et al., 2022, Sturua et al., 2024).

At the same time, the literature does not support a single universal recipe. In MIP and SAT, the foundational representation is a graph-based code distribution. In JL, it is the zero-variance limit of an optimized Gaussian sampler. In single-cell models and LOES, the optimization problem is layer selection rather than representation pretraining. In prompt optimization, the foundational object is a continuous prefix anchored to a human-written prompt. This suggests that the unifying principle is methodological rather than architectural: the embedding is treated as a primary object of optimization, and its usefulness is judged by transferability, structural fidelity, or both.

Several controversies are therefore better understood as domain-specific corrections to default practice. Final-layer extraction is not universally optimal. Randomized embeddings need not remain purely randomized, but direct deterministic optimization may still fail. Foundational representations can transfer across optimization and satisfiability, but SAT-specific features and SAT-native pretraining still improve clustering quality. Compactness and deployability can conflict with naïve sparse pruning, which motivates dense, hardware-friendly optimization of embedding tables. These are not mutually exclusive findings; they define the present research frontier of foundational optimization embeddings.