Embedding-Informed Surrogates: Methods & Applications

• Embedding-informed surrogates are models that use low- or intermediate-dimensional embeddings to approximate high-dimensional losses and system behaviors.
• They enable the construction of convex surrogates with calibrated link functions and are applied in classification, structured prediction, reinforcement learning, and scientific computing.
• By transforming complex optimization problems into tractable domains, these surrogates accelerate evaluation, enhance interpretability, and improve computational efficiency.

Embedding-informed surrogates are a broad class of surrogate models that leverage low- or intermediate-dimensional representations ("embeddings") of high-dimensional data, prediction spaces, or model parameters. These embeddings are used either to construct convex surrogates for otherwise intractable discrete losses, to accelerate the evaluation or optimization of complex systems (e.g., in scientific computing, control, or reinforcement learning), or to provide interpretable or data-efficient approximations by operating in an embedding-induced feature space. This notion arises in discriminative learning, scientific surrogate modeling, evolutionary optimization, combinatorial decision making, and interpretable modeling pipelines—each utilizing embeddings as a core algorithmic or analytic tool. The embedding-based approach is both a unifying theoretical paradigm (notably for polyhedral surrogate analysis) and a practical engineering strategy in modern ML and computational science.

1. Embedding Framework for Surrogate Loss Design

Central to the theory of embedding-informed surrogates is the embedding framework for polyhedral surrogate losses in discrete prediction settings. Given a discrete "target" loss $\ell:R \times Y \to \mathbb{R}_+$ over a finite prediction space $R$ and label set $Y$, the embedding approach begins by selecting a representative subset $S \subset R$ and defining an injective map $\varphi: S \to \mathbb{R}^d$. The surrogate $L: \mathbb{R}^d \times Y \to \mathbb{R}_+$ must then satisfy two crucial properties:

• For all $r \in S$, $y \in Y$: $L(\varphi(r), y) = \ell(r, y)$.
• For all conditional distributions $p$ over $Y$ and all $r \in S$:

$$r \in \arg\min_{r'} \mathbb{E}_p[\ell(r',Y)] \iff \varphi(r) \in \arg\min_{u} \mathbb{E}_p[L(u,Y)].$$

This embedding establishes a direct correspondence between discrete predictions and their embedded surrogates. The final surrogate $L$ is obtained by convexification—most commonly as a pointwise max over finitely many affine functions—yielding a polyhedral, piecewise-linear surrogate. The resulting surrogate admits calibrated link functions $\psi: \mathbb{R}^d \to R$ that guarantee statistical consistency via separation arguments; that is, if a prediction $u$ is not close (in the $\infty$-norm) to the optimal set for some $p$, the link $\psi(u)$ does not match the Bayes-optimal discrete decision, incurring positive excess risk (Finocchiaro et al., 2019, Finocchiaro et al., 2022, Finocchiaro et al., 2022).

2. Applications in Structured Prediction and Polyhedral Surrogates

Embedding-informed surrogates have been extensively developed for multiclass, ranking, and structured losses:

• Top-$k$ Classification: Piecewise-linear (polyhedral) surrogates, such as those of Lapin et al. and Yang & Koyejo, can be analyzed via embedding: for each, the embedded discrete problem is identified, and the region of conditional distributions under which consistency with the intended top-$k$ loss holds is characterized. For example, surrogates $\psi_2$, $\psi_3$, and $\psi_4$ embed finite losses that only align with the canonical top-$k$ loss for special subregions of the label simplex. The first truly consistent polyhedral surrogate for top-$k$ is constructed by embedding the top-$k$ loss and taking the convex envelope via Bayes risk conjugation, resulting in a closed-form piecewise-linear loss with a canonical argmax link (Finocchiaro et al., 2022).
• Multiclass and Structured Abstain: Embedding allows construction of convex surrogates for abstaining classifiers, matching original loss values at embedded points, with separation guarantees for the abstain link (Finocchiaro et al., 2022).
• General Embedding–Polyhedral Duality: Every discrete loss can be embedded into a suitable polyhedral surrogate, and every polyhedral surrogate corresponds to some finite loss it embeds. Matching of Bayes risks is both necessary and sufficient for consistency, and the embedding construction is constructive in both directions (Finocchiaro et al., 2022, Finocchiaro et al., 2019).

Embedding-informed analysis not only yields new consistent surrogates, but also serves as a diagnostic: any polyhedral surrogate not embedding the true discrete loss cannot be consistent outside the alignment region of their Bayes risks. Thus, embedding provides both the tools for constructing new surrogates and for diagnosing the fundamental limitations of existing ones.

3. Embedding-Based Surrogates in Scientific Computing and Control

Embedding-informed surrogates are crucial for constructing tractable, high-fidelity surrogate models when the original function or system is computationally expensive or high-dimensional:

• Evolutionary Reinforcement Learning: In high-dimensional policy spaces, performance surrogates built directly on parameter vectors are unscalable. The PE-SAERL framework uses a random projection $e: \mathbb{R}^D \to \mathbb{R}^d$, where $d \ll D$, to embed DNN policies. A relative surrogate is then trained in the embedding space to classify candidates as "promising" or not. Decoding back to parameter space is achieved via the left-inverse of the embedding matrix, and the framework yields up to $7\times$ acceleration on Atari benchmarks compared to non-surrogate evolutionary RL (Tang et al., 2023).
• Hybrid Physics-Informed Surrogates: In metabolic cybergenetics, neural surrogates are trained to map enzyme levels (embedding of gene expression programs) into steady-state metabolic exchange fluxes (outputs of flux balance analysis, FBA). Embedding the FBA physics into a low-dimensional NN enables replacement of bilevel dynamic optimization by single-level control. Dramatic speed-ups ($10^3$–$10^4\times$ over FBA calls) and exact recovery of the known trade-offs are achieved in the optogenetic itaconate production in E. coli (Espinel-Ríos et al., 2024).
• Embedding Physics-Informed NNs in NMPC: In nonlinear model predictive control (NMPC) with physics-informed neural network (PINN) surrogates, two embedding strategies are benchmarked. Explicit algebraic embedding introduces one auxiliary variable per neuron, yielding a large NLP; external-function embedding treats the entire surrogate as a black-box with optimized autodiff—leading to superior performance and scalability in direct-transcription NMPC problems (Casas et al., 10 Jan 2025).
• Multi-Step Embedding in Surrogate Dynamics: Multi-step Embed-to-Control (MS-E2C) for reservoir simulation replaces one-step, locally-linear E2C transitions with a global Koopman operator acting in a learned embedding of system state. This multi-step embedding drastically reduces error accumulation in long-horizon surrogate rollouts, achieving $50\%$ reduction in mean absolute error in complex waterflooding scenarios (Chen et al., 2024).

4. Embedding-Informed Surrogates for Enhanced Interpretability and Efficiency

Embedding-based surrogates facilitate interpretable and efficient modeling in both classical ML and deep learning contexts:

• Symbolic Surrogates for Transformer Embeddings: The "From Embeddings to Equations" pipeline partitions fixed pretrained Transformer embeddings into disjoint, information-preserving views and applies genetic programming to learn closed-form, additive symbolic logit programs. The result is parsimonious, calibrated surrogates retaining strong discrimination (F1 up to $0.99$) on vision and text datasets while using only a handful of embedding dimensions per class and offering explicit global explanations (e.g., partial dependence/ALE profiles, dimension overlap, term importance) (Khorshidi et al., 16 Sep 2025).
• Programming Frameworks for Embedded ML Surrogates: HPAC-ML enables embedding-informed surrogate deployment in scientific code via directive-based syntax. Its Data-Bridge maps application memory to ML tensor embeddings, and Execution-Control dynamically switches between accurate code and ML inference. Edge cases include autoregressive forecast surrogates (with error accumulation mitigated by interleaving true steps), yielding up to $83.6\times$ acceleration at low error (RMSE $\leq 0.01$) on financial, molecular, and physical benchmarks (Fink et al., 2024).
• Embedding-Guided Surrogates in Network Science: Surrogate null models for spatially embedded complex networks impose constraints on topological and spatial embedding statistics to disentangle purely structural effects from geometric ones. Four nested surrogates (random rewiring, degree-preserving, global link-length, and local-plus-global link-length) are implemented via embedding-induced constraints, with explicit pseudocode, quantitative metrics (e.g., clustering coefficient, mean shortest-path), and systematic model-selection for attribution of observed properties (Wiedermann et al., 2015).

5. Embedding-Informed Surrogates for Direct Learning of Evaluation Metrics

Deep embedding frameworks are leveraged to learn surrogates that directly approximate complex or non-differentiable target metrics:

• Deep Embedding Surrogates for Non-differentiable Losses: "Learning Surrogates via Deep Embedding" trains an embedding network $E_\phi: \mathcal{Y} \to \mathbb{R}^D$ such that $\|E_\phi(\hat y)-E_\phi(y)\|_2$ closely approximates the desired evaluation metric $d(\hat y, y)$, where $\hat y$ and $y$ are (possibly structured) predictions and targets, respectively. The differentiable surrogate $$L_\mathrm{sur}(\hat y, y) = \|E_\phi(\hat y) - E_\phi(y)\|_2^2$$ is then used to post-tune base models, yielding up to $39\%$ reduction in total edit distance and $4.25\%$ F1 improvement over standard objectives in text and detection tasks. Local-global batch mixing and careful training of embeddings are required to ensure surrogate gradients meaningfully track the target metric (Patel et al., 2020).

6. Limits, Diagnostic Power, and Extensions

While embedding-informed surrogates provide a general, constructive route to consistent convex approximation and enable acceleration across domains, they exhibit some limitations:

• Calibration Is Tied to Embedding: For polyhedral surrogates, only those whose embedding matches the Bayes risk of the intended discrete loss are consistent across the whole label simplex; hinge-like surrogates often fail this criterion for top-$k$ and structured losses, being consistent only in special subregions (Finocchiaro et al., 2022, Finocchiaro et al., 2022).
• Expressiveness and Generalization: Embedding-based NNs may fail outside the training distribution or embedding range; high-dimensional shuffle or overfitting can occur if embedding selection and regularization are not carefully managed (Espinel-Ríos et al., 2024, Chen et al., 2024, Fink et al., 2024).
• Auto-Regressive Error Accumulation: In time-iterated or multi-step surrogate applications, accumulation of errors in the embedding space may limit performance unless specifically regularized (as in MS-E2C) or mitigated by interleaving steps (Chen et al., 2024, Fink et al., 2024).

Potential extensions include uncertainty quantification (e.g., Bayesian surrogates over embeddings), online adaptation or co-training of embeddings, hybrid physical/symbolic/ML embedding recipes, and information-guided partitioning for model explanation (Khorshidi et al., 16 Sep 2025, Espinel-Ríos et al., 2024, Fink et al., 2024).

7. Summary Table: Representative Approaches

Approach/Paper	Embedding Strategy	Domain/Application
Polyhedral surrogate design (Finocchiaro et al., 2022, Finocchiaro et al., 2019, Finocchiaro et al., 2022)	Discrete report embedding + convexification	Consistent surrogates for classification, ranking, top-$k$
Deep RL surrogate (Tang et al., 2023)	Random projection policy embedding	Accelerated evolutionary RL in Atari
Physics-informed ML + FBA (Espinel-Ríos et al., 2024)	Enzyme-level to flux embedding	Metabolic dynamic control, cybergenetics
PINN/NMPC embedding (Casas et al., 10 Jan 2025)	NN state embedding as external function	Surrogate-based control for PDEs
Surrogate for metric learning (Patel et al., 2020)	Embedding output predictions	Non-differentiable metric optimization
Symbolic surrogates for Transformer embeddings (Khorshidi et al., 16 Sep 2025)	SPFP partitioning of embedding	Interpretable, calibrated text/vision models
Surrogatized spatial networks (Wiedermann et al., 2015)	Node coordinate/metric embedding constraints	Attribution of network statistics to spatial embedding
Surrogate code deployment (Fink et al., 2024)	Data-bridge to tensor embedding	Scientific application acceleration

In conclusion, embedding-informed surrogates encompass a principled and multifaceted set of methodologies for constructing, analyzing, and deploying surrogates by leveraging projections, latent feature representations, or symbolic mappings, enabling breakthroughs in theoretical consistency, computational tractability, interpretability, and cross-domain applicability.