Surrogate and Reinforcement Learning

• Surrogate + RL is an approach that integrates data-driven approximations to replace expensive simulations and sparse rewards with efficient, proxy signals.
• It leverages techniques like surrogate rewards, environment modeling, and meta-level optimization to improve training stability and sim-to-real transfer.
• Applications include evolutionary policy search, robust testing, and interpretability, all aimed at enhancing RL performance in complex, high-dimensional domains.

Surrogate + Reinforcement Learning

Surrogate models have become a cornerstone for accelerating and enhancing reinforcement learning (RL), especially in domains where direct evaluation or simulation is costly, high-dimensional, or lacks dense supervision. The integration of surrogates and RL spans a wide range of methodologies: from optimizing policy learning via surrogate rewards, to replacing environment models, to meta-level algorithmic scheduling, and to supporting interpretability, testing, and sim-to-real transfer.

1. Surrogate Signals and Rewards in Reinforcement Learning

A central approach uses surrogate signals as alternatives to standard, often unavailable or sparse, ground-truth rewards. Notably, "Surrogate Signals from Format and Length: Reinforcement Learning for Solving Mathematical Problems without Ground Truth Answers" introduces two such surrogates for RL fine-tuning of LLMs on mathematical problems (Xin et al., 26 May 2025):

• Format correctness ($R_{\mathrm{format}}$): A binary function detecting adherence to prescribed solution formats (e.g., proper math boxing), essentially acting as a structural proxy for correctness.
• Length reward ($R_{\mathrm{length}}$): A concave, smooth function, peaking at a calibrated solution length, penalizing answers that are too short (incomplete) or too long (verbose).

The composite surrogate reward

$$R_{\mathrm{fl}}(y) = \begin{cases} R_{\mathrm{format}}(y) + R_{\mathrm{length}}(y) & \text{if } R_{\mathrm{format}}(y) = 1 \ \min(0, R_{\mathrm{format}}(y)+R_{\mathrm{length}}(y)) & \text{otherwise} \end{cases}$$

substitutes for answer-level correctness. Early RL mainly instills answer-formatting habits; the length component further encourages logically rich responses, yielding performance that can match or exceed traditional exact-match RL, as demonstrated by a 40.0% AIME2024 accuracy with a 7B LLM (Xin et al., 26 May 2025).

In broader RL, surrogate rewards address variance reduction (e.g., long $N$-step surrogate rewards, which exponentially tighten the bound on Q-function variance (Zhong et al., 2022)), agent selection in multi-armed bandits (dense information-gain signals for reward shaping (Merentitis et al., 2019)), and improved policy-gradient stability (families of provably safe surrogate functions derived via functional mirror ascent (Vaswani et al., 2021)).

2. Surrogate Models for Environment Dynamics and Costly Simulations

Surrogate modeling replaces expensive or unavailable environment simulators with fast, data-driven approximations for RL agents. This is prevalent in model-based RL and optimization:

• Physics-informed neural networks (PINNs):

Used as surrogates to encode hard physical constraints (e.g., KKT conditions, AC power-flow equations) in smart grid energy management (Cestero et al., 20 Oct 2025). PINN surrogates drastically reduce inference costs (2.1 ms vs. 19.4 ms/original), accelerate PPO training by up to 50%, and preserve physical fidelity.

• Sparse identification of nonlinear dynamics (SINDy):

SINDy surrogates represent system dynamics in explicit, sparse forms, yielding interpretable, fast models for environments such as Mountain Car and Lunar Lander (Dixit et al., 25 Apr 2025). With as few as 75 data points, SINDy surrogates enable RL agents to reach final performance at 20–35% fewer training steps, without significant loss in policy quality.

• Surrogates for state transition/model learning without access to clean ground truth:

For data assimilation when the system is only partially and noisily observed, RL (PPO) is used for maximum-likelihood estimation of surrogate state-transition models, outperforming expectation-maximization approaches in high-noise and high-dimensional settings (Wang et al., 4 Nov 2025).

• Surrogates for sim-to-real in robotics/control:

Replacing soft-body physics with surrogate variables (limb scaling, center-of-mass shift) inside a rigid-body simulator, followed by RL with heavy domain randomization, yields robust hardware-polices in soft robotics (Wang et al., 8 Dec 2025). For industrial control (ORC superheat), LSTM surrogates enable safe, rapid offline PPO pre-training and highly sample-efficient sim-to-real policy transfer (Lin et al., 2023).

3. Surrogate-Assisted Evolutionary RL and Policy Search

Evaluating deep policy networks with evolutionary algorithms is computationally demanding. Multiple frameworks introduce surrogate models to preselect or filter candidate policies before expending expensive simulation rollouts:

Method (Editor’s term)	Policy Embedding	Surrogate Architecture	Pre-selection Mechanism	Benefit
PE-SAERL (Tang et al., 2023)	Random projection	Fuzzy-KNN classifier in embedding space	Selects best among offspring	Up to 7× speedup, comparable/better Atari performance
AE-HNN-NCS (2505.19423)	Autoencoder (learned)	Hyperbolic neural network on Poincaré ball	Picks most promising per parent	SOTA on 10 Atari/4 MuJoCo, 38% runtime reduction
SMB-NE (Stork et al., 2019)	Phenotypic (behavioral)	Gaussian process with phenotypic distance kernel	Optimize Expected Improvement	70–80% reduction in evals on CartPole/MountainCar
Surrogate Controller (Wang et al., 2022)	None	RL critic as fitness predictor	Generation- and individual-level use	Up to 80% reduction in env interactions in hybrid ERL**

Surrogate-assisted selection depends critically on representation: random projections for high-dim DNNs suffice in some domains (PE-SAERL), but data-aware autoencoders or behavioral distances preserve fitness relationships more robustly (AE-HNN-NCS, SMB-NE). Robustness is achieved with management strategies such as elite protection and alternating real/surrogate evaluation (Wang et al., 2022, 2505.19423).

4. Surrogates in Meta-level Optimization and RL-Driven Scheduling

Surrogates are increasingly embedded in meta-RL loops, driving not just inner-loop policy evaluation but also adaptation and dynamic algorithmic scheduling:

• Surrogate ensemble for multi-objective optimization:

The SEEMOO framework maintains a pool of diverse surrogates (GP, MLPs, KNN, RBFN) and applies deep Q-learning (DQN) to dynamically select which model to use at each stage of NSGA-II search, based on attention-processed features of the optimization process (Wu et al., 31 Jan 2026). This policy, trained on problem distributions, yields lower IGD and better Pareto convergence than any static model.

• Surrogate-driven meta-black-box optimization:

Surr-RLDE leverages a KAN surrogate (with a rank-order-aware loss) for high-fidelity function approximation and couples it with meta-RL (Double-DQN) to configure Differential Evolution mutation dynamics efficiently. With careful ablations, Surr-RLDE achieves in-distribution and out-of-distribution generalization rivaling the best meta-BBO techniques while using an order-of-magnitude fewer function calls (Ma et al., 23 Mar 2025).

• ESP (Evolutionary Surrogate-Assisted Prescription):

Surrogate models trained on historical outcomes provide fitness landscapes for neuroevolutionary policy search, dramatically increasing sample efficiency and regularizing policy search. ESP regularization outperforms both classic direct evolution and advanced policy-gradient RL in regret and convergence (Francon et al., 2020).

5. Surrogates in Testing, Interpretability, and Robustness

Surrogate models extend beyond learning to testing and interpretation of RL agents:

• Surrogate-based testing for failure discovery:

Train a failure-prediction classifier on agent/environment pairs collected during RL training, then use it as a fast fitness/scoring function in search processes (genetic algorithms, hill climbing), yielding 50% more discovered failures and ~75% higher behavior/input diversity than random baselines (Biagiola et al., 2023).

• Surrogate fitness functions for demonstration selection and interpretability:

Optimization frameworks (e.g., REACT) employ composite surrogate metrics that combine global diversity, local diversity, and behavioral certainty for selecting informative agent demonstration trajectories. These surrogate designs enable discovery of rare and representative policy behaviors, improving RL model explainability in safety-critical contexts (Altmann et al., 20 Apr 2025).

6. Theoretical and Practical Considerations

The surrogate + RL paradigm is underpinned by rigorous theoretical and empirical analyses:

• Policy improvement and monotonicity:

Parameterized surrogate functions (as in TRPO/PPO/MPO) guarantee policy improvement—under smoothness and convexity conditions—by providing lower bounds on expected return and unifying major policy-gradient classes (Vaswani et al., 2021).

• Variance reduction and stability analysis:

Multi-step surrogate rewards (e.g., the LNSS estimator) exponentially reduce Q-value variance without biasing estimates, yielding provable improvements in convergence speed, reward maximization, and coefficient of variation on complex continuous control benchmarks (Zhong et al., 2022).

• Regularization and generalization:

Surrogate-level and policy-level regularizations (as in ESP) smooth the search landscape, reduce susceptibility to local optima, and implicitly ensemble across prediction models, resulting in improved reliability even with limited true environment evaluations (Francon et al., 2020).

7. Limitations and Open Directions

Despite broad success, surrogate-driven RL is subject to key caveats:

• Domain- and model-dependence:

Surrogate reward or model transferability across domains (e.g., from mathematics to code, or between simulators with different physics) is unproven and may necessitate new proxy designs (Xin et al., 26 May 2025, Wang et al., 8 Dec 2025).

• Surrogate-induced bias or false minima:

Surrogate fidelity is vital. Low-quality surrogates or poorly managed surrogate/real evaluation cycles can mislead search, especially early in training (Wang et al., 2022, 2505.19423).

• Active learning and dynamic update scheduling:

Input-set selection (as in phenotypic surrogates), policy-embedding alignment, and model re-training are unresolved engineering challenges, particularly in high-noise, nonstationary, or partially observable regimes (Stork et al., 2019, Wang et al., 4 Nov 2025).

• Reward hacking and adversarial optimization:

Improperly calibrated surrogate signals are vulnerable to reward hacking—trivial solutions that maximize the surrogate without actually solving the true task (Xin et al., 26 May 2025).

A plausible implication is that future research will focus on principled surrogate construction—integrating uncertainty quantification, adversarial validation, and real-time adaptation—to ensure robust and trustworthy surrogate-augmented RL across ever broader scientific and engineering domains.