--- title: Foundational Optimization Embeddings url: https://www.emergentmind.com/topics/foundational-optimization-embeddings type: topic --- # Foundational Optimization Embeddings Searching arXiv for the cited topic and closely related papers. arXiv search query: "Foundational Optimization Embeddings Forge SAT Johnson-Lindenstrauss optimization embeddings" In the literature surveyed here, the phrase **foundational optimization embeddings** encompasses several related but non-identical practices: pre-training reusable structural representations for mixed-integer programming (MIP), transferring those representations to Boolean satisfiability (SAT), constructing deterministic Johnson–Lindenstrauss (JL) embeddings by optimizing over distributions rather than fixed matrices, selecting intermediate foundation-model layers whose frozen embeddings best support a task, and directly optimizing embeddings used for prompts, tables, or logic-grounded geometric models [2508.20330], [2604.15448], [2412.07242], [2605.23033], [2508.03533], [2208.04482], [1902.10499]. Taken together, these works suggest that the topic is not a single algorithmic recipe but a family of methods in which the embedding itself—or the space from which it is extracted—is treated as the primary optimization object. ## 1. Conceptual scope and recurrent design pattern A common thread across these works is the replacement of narrowly task-specific representation learning with a more general embedding substrate that can be reused, transferred, or derandomized. In **Forge**, the foundational object is an unsupervised, instance-level representation of MIP problems learned from static bipartite graphs without requiring optimal solutions [2508.20330]. In the SAT transfer study, the same pretrained representation is reused **without any architectural changes or supervised fine-tuning**, and the claim is that the embeddings remain meaningful even when the domain changes from optimization to decision problems [2604.15448]. In the JL setting, the foundational move is different: the paper does not pretrain a representation on many instances, but it replaces direct optimization over projection matrices with optimization over a larger space of Gaussian samplers and then shrinks variance to zero, yielding a deterministic embedding [2412.07242]. A second recurring motif is that the useful representation need not coincide with the most obvious one. The single-cell study argues that **“where” to extract features matters as much as “what” the model learns**, because intermediate layers can outperform final-layer embeddings and the optimum depends on task and cellular context [2604.14838]. The LOES work makes a parallel claim at a broader level: task-relevant information is **distributed non-monotonically across depth**, so transfer should be posed as selecting a compact task-discriminative subspace under isotropy, redundancy, and simplicial-geometry constraints rather than simply taking the last hidden state [2605.23033]. This body of work therefore rejects several default assumptions. One is that a final layer is automatically the best embedding. Another is that direct optimization over a deterministic embedding object is necessarily viable. A third is that foundational representations must remain within the domain in which they were pretrained. Each of these assumptions is explicitly challenged in at least one of the cited papers. ## 2. Foundational graph embeddings for optimization instances The clearest use of the term in combinatorial optimization is **Forge**, which introduces a **vector-quantized graph autoencoder** for MIP bipartite graphs and frames its output as a reusable instance-level representation [2508.20330]. A MIP of the form $$ f(x)=\min\{c^Tx\mid Ax\leq b,\ x\in \mathbb{R}^n,\ x_j\in \mathbb{Z}\ \forall j\in I\} $$ is encoded as a bipartite graph with variable nodes, constraint nodes, weighted edges for nonzero coefficients, and only static input-instance features. Constraint nodes receive 4 features, variable nodes receive 6 features, and these are padded into a 10-dimensional node-feature vector. Forge applies two GraphSAGE layers, vector-quantizes the node embeddings into discrete codes, replaces each code by its codeword embedding, and decodes both graph structure and node features. Its training loss is $$ \mathcal{L}=\mathcal{L}_{Rec}+\mathcal{L}_{Codebook}+\mathcal{L}_{Commitment}, $$ with reconstruction, codebook, and commitment terms defined over node embeddings and selected codewords. The paper explicitly interprets the codebook objective as **k-means-like clustering** in latent space. The central representation is not the dense readout of node states but the **distribution of discrete codes** across an instance. If the codebook has size \(k\), the instance embedding is a \(k\)-dimensional count vector; the paper likens this to a vocabulary-like representation of optimization instances. This representation is evaluated in both unsupervised and supervised settings. In the unsupervised setting, Forge is trained on **600 MIPLIB instances** plus two augmented variants of each instance formed by randomly dropping **5%** and **10%** of constraints, for **1,800 instances** total. Tested on **1,050 unseen D-MIPLIB instances** spanning **21 problem type-difficulty pairs**, the code-distribution embedding achieves **NMI = 0.843** under \(k\)-means with 21 clusters, compared with **0.790** for two-hop label propagation and **0.087** for mean readout. In supervised fine-tuning, the same pretrained encoder is reused for **integrality gap prediction** and **warm-start variable prediction**, and the resulting predictions improve Gurobi and also improve **PS-Gurobi** after dimensionality reduction of node embeddings from **1024** to **64** dimensions via PCA [2508.20330]. The practical significance is that the foundational representation is learned **without dependency on solution labels**, yet is later adapted to solver-facing tasks. This is a specific answer to a long-standing bottleneck in ML-for-optimization: collecting large numbers of hard-to-solve labeled instances. ## 3. Transfer from optimization to SAT The SAT transfer paper extends the foundational-embedding idea beyond optimization by asking whether a MIP-pretrained representation can organize decision problems in a fully unsupervised way [2604.15448]. The key technical bridge is a structural one: SAT formulas are recast into the same **bipartite constraint–variable graph** format used for MIPs. A CNF formula $$ F=C_1\land\cdots\land C_m $$ is encoded as a 0–1 MIP in which each clause becomes $$ \sum_{t=1}^k y_{jt}\ge 1, $$ with $$ y_{jt}=x_i \quad \text{if } \ell_{jt}=x_i,\qquad y_{jt}=1-x_i \quad \text{if } \ell_{jt}=\lnot x_i. $$ The paper states that this encoding preserves satisfiability exactly. Three transfer settings are studied. **Forge-Mip** applies the pretrained MIP model to SAT instances with no architectural changes and no feature redesign. **Forge-Mip-Sat** still reuses the pretrained MIP weights but substitutes SAT-specific clause and variable descriptors. **Forge-Sat** keeps the same architecture but discards the MIP-pretrained weights and pretrains from scratch on SAT using SAT-specific features. The architectural core remains the same: a two-layer GraphSAGE encoder, vector quantization, decoding of node features and incident edges, and instance-level embeddings derived from the empirical distribution of codewords. Evaluation is performed on the **Hard** split of **G4SATBench**, which contains **1400 unseen formulas total**, with **200 from each problem type** and a balanced SAT/UNSAT split, producing **14 ground-truth groups** defined by \((\text{problem type}, \text{feasibility})\). Cluster quality is measured with **Normalized Mutual Information (NMI)** and **Purity**. The static baseline yields **NMI \(0.74 \pm 0.002\)**. **Forge-Mip** reaches **\(0.76 \pm 0.001\)**, **Forge-Mip-Sat** reaches **\(0.77 \pm 0.003\)**, and **Forge-Sat** performs best with **NMI \(0.79 \pm 0.016\)** and **Purity \(0.66 \pm 0.002\)**. The paper reports that Forge-Sat comes closest to **14 clean clusters**, with only \(k\)-Clique SAT/UNSAT overlapping, and that it separates satisfiability within the same problem type, such as **PS SAT vs. UNSAT** [2604.15448]. A common misconception addressed here is that a foundational optimization representation must remain MIP-specific. The reported transfer results do not eliminate domain mismatch, but they do show that a representation pretrained on optimization data alone can transfer to SAT **without supervision**. ## 4. Derandomizing embeddings by optimizing over samplers A different foundational strand is presented by **“Optimization Can Learn Johnson Lindenstrauss Embeddings”**, which asks whether JL embeddings must remain randomized objects or can instead be learned by optimization directly from data [2412.07242]. The paper first shows why the naïve approach fails. If one minimizes the maximum-distortion objective $$ h(A)=\max_{x_1,\dots,x_n}\left|\|Ax\|_2^2-1\right|, $$ the landscape over projection matrices is non-convex and contains bad local minima. The authors construct matrices \(A\in \mathbb{R}^{k\times (k+1)}\) that are **strict local minima** reachable from the origin, with constant distortion \(\Omega(1)\), even though there exist embeddings with distortion \(O(\sqrt{\log k/k})\to 0\). The workaround is to optimize over a larger space of **random solution samplers**. Instead of fixing \(A\), the method considers $$ A\sim N(\mathbf{M},\sigma^2), $$ defines the failure-probability objective $$ f^*(\mathbf{M},\sigma^2)=\Pr_{A\sim N(\mathbf{M},\sigma^2)}[h(A)>\varepsilon], $$ relaxes it to a sum over data points, $$ f(\mathbf{M},\sigma^2)=\sum_{j=1}^n \Pr_{A\sim N(\mathbf{M},\sigma^2)} \left[ \frac{1}{k}\|Ax_j\|_2^2 \notin (1-\varepsilon,1+\varepsilon) \right], $$ and adds a variance penalty $$ g(\mathbf{M},\sigma^2)=f(\mathbf{M},\sigma^2)+\frac{\sigma^2}{2}. $$ Initialization at \((\mathbf{0},1)\) corresponds to i.i.d. Gaussian entries, so the JL theorem provides a good starting sampler. The structural claim is that any point with nonzero variance is not a true local minimum: for any \((\mathbf{M},\sigma^2)\) and any \(\gamma\in[0,\sigma]\), there exists a nearby mean \(\mathbf{M}'\) such that $$ \|\mathbf{M}-\mathbf{M}'\|_F \le 2\gamma \sqrt{kd\log\!\left(\frac{3\sqrt{kd}}{\gamma}\right)} $$ and $$ g(\mathbf{M}',\sigma^2-\gamma^2)\le g(\mathbf{M},\sigma^2)-\frac{\gamma^2}{6}. $$ This supports the paper’s reverse-diffusion interpretation: optimization moves through sampler space while progressively reducing variance. The analyzed algorithm is a deterministic second-order method, **Hessian Descent**, which alternates between gradient steps, negative-curvature steps, and termination at a \(\rho\)-approximate second-order stationary point. With \(\rho<1/\mathrm{poly}(n,k,d)\), the returned matrix satisfies the JL guarantee with distortion \(O(\varepsilon)\), where \(\varepsilon=O(\sqrt{\log n/k})\) [2412.07242]. This paper addresses a different misconception: not that randomness is useless, but that derandomization can be achieved by directly optimizing the deterministic object of interest. The argument is that direct matrix optimization is “provably doomed for this objective,” whereas optimization over samplers avoids bad stationary points. ## 5. Layer selection as foundational embedding optimization Another usage of the topic appears in foundation models whose representations evolve across depth. The single-cell study states that current benchmarks “universally extract final layer embeddings,” but its analysis of **scFoundation** and **Tahoe-X1** shows that the optimal layer is task-dependent and context-dependent [2604.14838]. For trajectory inference, **Tahoe-X1 peaks at 60% depth**, specifically **layer 16 of 24**, with **Spearman correlation \( \rho = 0.7626 \)**, which is **31% better than the final layer** at **\( \rho = 0.5843 \)**. For perturbation response prediction, the optimum shifts across activation states: for Tahoe-X1, **Rest** is best at **layer 1, 0% depth, \( \rho = 0.368 \)**; **Stim8hr** at **layer 4, 13% depth, \( \rho = 0.444 \)**; and **Stim48hr** at **layer 23, 96% depth, \( \rho = 0.536 \)**. The paper highlights the case in which **first-layer embeddings outperform all deeper layers in quiescent cells** [2604.14838]. The LOES paper generalizes this idea and makes it explicitly optimization-based. **Layer-wise Optimal Embedding Selection (LOES)** scores candidate layers by residual ridge loss, isotropy, redundancy, and simplex-like class geometry: $$ \mathrm{Score}(\ell)=\mathrm{Loss}_\ell+\alpha\bigl(1-\mathrm{Iso}(\mathbf{X}_\ell)\bigr)+\gamma\,\mathrm{Red}_\ell-\eta\,\mathrm{Tri}(\widetilde{\mathbf{X}_\ell}). $$ Its ridge probe is $$ \mathbf{W}^\star=(\mathbf{X}^\top\mathbf{X}+\lambda \mathbf{I})^{-1}\mathbf{X}^\top \mathbf{Y}, $$ and candidates are orthogonalized against the currently selected subspace before scoring. During downstream adaptation, **GeoReg** enforces isotropy and simplicial class geometry through $$ \mathcal{L}_{\mathrm{GeoReg}}=\lambda_{\mathrm{geo}}\left(\mathcal{L}_{\mathrm{iso}}-\log(A+\epsilon)\right). $$ Across vision, language, speech, segmentation, and multimodal settings, the paper reports that LOES consistently outperforms last-layer baselines and that **gains increase with model depth** [2605.23033]. These results directly contest the idea that deeper is always better or that naïve aggregation of the last few layers is sufficient. In both studies, the foundational embedding is not a fixed architectural endpoint but a selected intermediate representation whose utility depends on geometry, task, and context. ## 6. Other optimization-based embedding constructions Several additional works expand the topic beyond MIP graphs and layer selection. **EL Embeddings** formulate the search for a vector-space interpretation of an \(\mathcal{EL}^{++}\) ontology as an optimization problem over class centers, class radii, and role translations in \(\mathbb{R}^n\) [1902.10499]. Concepts are represented as open \(n\)-balls, $$ C^\mathcal{I}=\{x\in \mathbb{R}^n\mid \|f_\eta(C)-x\|