Surrogate Representation Inference (SRI)

• SRI is a methodological family that substitutes complex or unavailable objects with learned surrogates to facilitate efficient and precise inference.
• It encompasses strategies like conditional density estimators, Bayesian surrogate models, and low-dimensional encoders while ensuring identifiability and calibration.
• Applications range from simulation-based inference to long-term causal effect estimation, with evaluations based on surrogate fidelity metrics such as r².

Searching arXiv for papers that use or define “Surrogate Representation Inference” and closely related surrogate-inference formulations. arxiv_search query: "Surrogate Representation Inference" max_results: 10

Surrogate Representation Inference (SRI) denotes a class of inference strategies that replace an unavailable, delayed, noisy, or computationally expensive object with a learned or constructed surrogate and then perform downstream estimation through that surrogate. In current usage, the label covers several distinct formulations: conditional neural density surrogates for likelihood-free simulation-based inference, fully Bayesian surrogate models with explicit uncertainty propagation, sequential latent representations for long-term causal effect estimation, low-dimensional encoders for valid inference with noisy text and image annotations, and open-model surrogates used to study closed-model representations (Refaeli et al., 16 Apr 2025, Reiser et al., 2023, Cheng et al., 2020, Nakamura, 15 Sep 2025, Chlenski et al., 30 Jun 2026). This suggests that SRI is best understood as a methodological family rather than a single algorithm, unified by the question of how much inferential content can be carried by a surrogate object without sacrificing identifiability, calibration, or efficiency.

1. Core concept and terminological scope

Across the literature, SRI couples two operations. First, it constructs a surrogate object that is cheaper, more accessible, or more statistically manageable than the original target. Second, it performs inference on the parameter, outcome, or representation of interest using that surrogate in place of direct access to the original object. The surrogate may be a conditional density $\hat p_\phi(x\mid\theta)$, a parametric simulator approximation $\tilde M(\cdot)$, a latent sequential representation $R_t$, a low-dimensional embedding $W_i=f(Y_i)$, or an open model $M_S$ used to reason about a closed model $M_T$ (Refaeli et al., 16 Apr 2025, Reiser et al., 2023, Cheng et al., 2020, Nakamura, 15 Sep 2025, Chlenski et al., 30 Jun 2026).

Setting	Surrogate object	Inferential target
Simulation-based inference	$\hat p_\phi(x\mid\theta)$	posterior $q_\psi(\theta\mid x_{\mathrm{obs}})$
Surrogate-based Bayesian inference	$\tilde M(\omega,\phi,e_S)$ and $p(\phi\mid D_t)$	$\tilde M(\cdot)$0
Long-term causal inference	latent confounders $\tilde M(\cdot)$1 and representation $\tilde M(\cdot)$2	long-term ATE $\tilde M(\cdot)$3
Noisy annotation inference	representation $\tilde M(\cdot)$4	covariate-adjusted mean $\tilde M(\cdot)$5
Mechanistic interpretability	open surrogate $\tilde M(\cdot)$6	fidelity of measurements on $\tilde M(\cdot)$7

The common mathematical pattern is a bridge from an original data-generating object to an estimand through an intermediate representation. In some papers the bridge is probabilistic and explicit, as in

$\tilde M(\cdot)$8

while in others it is encoded by conditional independence, such as

$\tilde M(\cdot)$9

or by approximate representational alignment,

$R_t$0

This suggests that the defining issue in SRI is not the form of the surrogate itself, but whether the surrogate preserves the inferentially relevant structure of the original problem.

2. Surrogate likelihoods and Bayesian replacement of expensive simulators

In simulation-based inference, SRI replaces expensive calls to a stochastic simulator by samples from a learned conditional neural density estimator. Let $R_t$1 be the parameter and $R_t$2 the data. The true simulator induces an intractable likelihood $R_t$3, from which one can sample but not evaluate or differentiate directly. The SRI construction introduces a tractable surrogate likelihood

$R_t$4

parameterized by neural-network weights $R_t$5, and uses surrogate draws $R_t$6 in place of real simulator outputs (Refaeli et al., 16 Apr 2025). In the first round of Sequential Neural Posterior Estimation (SNPE), one samples $R_t$7, obtains $R_t$8, collects $R_t$9, and trains the surrogate by maximum conditional likelihood,

$W_i=f(Y_i)$0

with an importance-weighted variant when parameters are drawn from a proposal $W_i=f(Y_i)$1. A posterior NDE $W_i=f(Y_i)$2 is trained on the same data. In subsequent rounds, standard SNPE is modified by drawing $W_i=f(Y_i)$3, replacing simulator calls with

$W_i=f(Y_i)$4

and updating $W_i=f(Y_i)$5 with the importance-weighted negative log-likelihood

$W_i=f(Y_i)$6

Because surrogate sampling is ultrafast, one can take $W_i=f(Y_i)$7 times as many surrogate-based samples in later rounds without further real simulator calls. The same work optionally combines this with support points, where an $W_i=f(Y_i)$8-point design $W_i=f(Y_i)$9 is chosen to represent a larger $M_S$0-sample from $M_S$1 while remaining space-filling, via the difference-of-convex optimization in equation (5) of the paper.

A broader Bayesian formulation treats surrogate uncertainty itself as part of the inferential problem. In surrogate-based Bayesian inference, a computationally expensive simulator $M_S$2 is replaced by a faster surrogate $M_S$3, with $M_S$4 containing both functional parameters and approximation-error hyperparameters (Reiser et al., 2023). The exact simulator likelihood $M_S$5 is replaced by the surrogate likelihood

$M_S$6

and the full Bayesian target becomes

$M_S$7

The computational workflow separates a training step that infers the T-posterior $M_S$8 from an inference step that propagates this uncertainty into the I-posterior $M_S$9. The paper compares four propagation schemes: Point Estimate, Expected-Posterior (E-Post), Expected-Likelihood (E-Lik), and Expected-Log-Likelihood (E-Log-Lik). The distinction is substantive rather than cosmetic: E-Post marginalizes $M_T$0 in the posterior of $M_T$1, E-Lik marginalizes $M_T$2 in the likelihood, and E-Log-Lik marginalizes $M_T$3 inside the log-likelihood. In the reported case studies, E-Lik and E-Post preserve calibration under Simulation-Based Calibration when the number of simulator training points is limited, whereas Point and E-Log-Lik can be overconfident for small $M_T$4. The paper also reports that E-Log-Lik can exhibit counter-intuitive shrinkage with increasing T-posterior uncertainty in the linear slope–intercept model.

Empirically, the simulator-replacement version of SRI was evaluated on six standard SBI test problems—1D GMM, 2D Two Moons, 3D GMM, 5D SLCP, 6D Bayesian LR, 10D Bernoulli-GLM—plus the 8D Hodgkin–Huxley model, with accuracy measured by $M_T$5, C2ST, and $M_T$6 (Refaeli et al., 16 Apr 2025). The surrogate method yielded consistent, if modest, improvements on most tasks and budgets, with mean C2ST gains of order $M_T$7–$M_T$8 percentage points and up to $M_T$9 percentage points in a single instance. Support points were more mixed: helpful on some problems, including 1D-GMM and 6D-LinReg, but harmful on others, including Two Moons and Bernoulli-GLM.

3. Learned surrogate representations in causal and observational inference

One major use of SRI arises when the primary outcome is delayed and available short-term outcomes are informative but not fully sufficient. In long-term effect estimation, the central obstacle is the classical strong surrogacy assumption

$\hat p_\phi(x\mid\theta)$0

which posits that treatment carries no additional information about the long-term outcome once the final short-term surrogate is observed (Cheng et al., 2020). The framework termed SRI, and also referred to as LTEE, avoids this assumption by introducing time-varying latent confounders $\hat p_\phi(x\mid\theta)$1 and a learned surrogate representation $\hat p_\phi(x\mid\theta)$2. Its generative process is

$\hat p_\phi(x\mid\theta)$3

The key assumption is temporal unconfoundedness: $\hat p_\phi(x\mid\theta)$4 Inference is variational and sequential. An approximate posterior

$\hat p_\phi(x\mid\theta)$5

is parameterized by an RNN, and the training objective is a regularized ELBO with a balancing penalty,

$\hat p_\phi(x\mid\theta)$6

After training, the counterfactual prediction under a static policy $\hat p_\phi(x\mid\theta)$7 is

$\hat p_\phi(x\mid\theta)$8

and the estimated long-term average treatment effect is

$\hat p_\phi(x\mid\theta)$9

Under temporal unconfoundedness, positivity, comparability, and model well-specification, the paper states that $q_\psi(\theta\mid x_{\mathrm{obs}})$0. On IHDP and News, SRI/LTEE is reported to achieve the lowest $q_\psi(\theta\mid x_{\mathrm{obs}})$1 across all combinations of $q_\psi(\theta\mid x_{\mathrm{obs}})$2, often cutting errors in half relative to the next best baseline.

A second causal-statistical formulation treats unstructured data themselves as the mediator from which a valid surrogate representation should be learned. In noisy text and image annotation settings, SRI observes a high-dimensional datum $q_\psi(\theta\mid x_{\mathrm{obs}})$3, a structured predictor $q_\psi(\theta\mid x_{\mathrm{obs}})$4, covariates $q_\psi(\theta\mid x_{\mathrm{obs}})$5, and a possibly noisy human annotation $q_\psi(\theta\mid x_{\mathrm{obs}})$6 of a latent concept $q_\psi(\theta\mid x_{\mathrm{obs}})$7. It introduces a low-dimensional representation

$q_\psi(\theta\mid x_{\mathrm{obs}})$8

and posits Assumption 1 (Surrogate Sufficiency),

$q_\psi(\theta\mid x_{\mathrm{obs}})$9

with the annotator version $\tilde M(\omega,\phi,e_S)$0 under perfect annotation (Nakamura, 15 Sep 2025). For the covariate-adjusted mean under $\tilde M(\omega,\phi,e_S)$1,

$\tilde M(\omega,\phi,e_S)$2

identification proceeds through the conditional expectation of $\tilde M(\omega,\phi,e_S)$3 given $\tilde M(\omega,\phi,e_S)$4 and $\tilde M(\omega,\phi,e_S)$5. The paper then derives a semiparametric efficient influence function involving the nuisance functions

$\tilde M(\omega,\phi,e_S)$6

and shows that cross-fitted estimating equations yield a doubly robust, asymptotically normal, efficient estimator of $\tilde M(\omega,\phi,e_S)$7. The representation and nuisances are learned jointly with a DragonNet-style objective,

$\tilde M(\omega,\phi,e_S)$8

When multiple independent annotations are available, the framework corrects non-differential measurement error by constructing a surrogate outcome $\tilde M(\omega,\phi,e_S)$9 that satisfies $p(\phi\mid D_t)$0 and $p(\phi\mid D_t)$1. The paper reports that, when machine learning prediction accuracy is moderate, SRI reduces standard errors by over $p(\phi\mid D_t)$2, and in the congressional tone application yields $p(\phi\mid D_t)$3 with $p(\phi\mid D_t)$4 under the perfect-annotation assumption and $p(\phi\mid D_t)$5 with noisy-annotation correction, compared with $p(\phi\mid D_t)$6 for existing bias-correction methods.

A related surrogate-bridge construction appears in delayed-outcome estimation under administrative censoring. There, identification proceeds through

$p(\phi\mid D_t)$7

which rewrites the causal parameter as an integral of the observed-outcome regression over the conditional surrogate distribution (Li, 11 Mar 2026). The resulting surrogate-assisted TMLE is asymptotically linear and doubly robust, and its two-stage targeting step removes a second-order cross-product remainder without requiring direct estimation of the surrogate density.

4. Surrogate fidelity in representation-level analysis of LLMs

In mechanistic interpretability, SRI refers to the use of an open-model surrogate to make claims about a closed model’s internal representations. The target model $p(\phi\mid D_t)$8 is accessible only through top-$p(\phi\mid D_t)$9 log-probabilities, while the surrogate $\tilde M(\cdot)$00 exposes full white-box internals. The central hypothesis is that there exists a linear map $\tilde M(\cdot)$01 such that

$\tilde M(\cdot)$02

for every input $\tilde M(\cdot)$03, where $\tilde M(\cdot)$04 are internal representations (Chlenski et al., 30 Jun 2026). For binary classification, if $\tilde M(\cdot)$05 is the residual stream and $\tilde M(\cdot)$06 is the readout direction derived from the positive and negative unembedding columns, then the API-compatible scalar readout is the log-odds

$\tilde M(\cdot)$07

For an intervention $\tilde M(\cdot)$08, attribution is

$\tilde M(\cdot)$09

Surrogate fidelity is defined at multiple levels by squared Pearson correlation $\tilde M(\cdot)$10 over a shared corpus $\tilde M(\cdot)$11. Prediction-level fidelity is

$\tilde M(\cdot)$12

attribution-level fidelity is

$\tilde M(\cdot)$13

and representation-level fidelities for open–open pairs include

$\tilde M(\cdot)$14

together with attention-based $\tilde M(\cdot)$15. For open-to-closed comparisons, the paper defines

$\tilde M(\cdot)$16

Multi-class tasks replace $\tilde M(\cdot)$17 with the RV coefficient on stacked pairwise log-odds.

The empirical findings sharply distinguish levels of agreement. On BoolQ, sentence-level ablation across eleven models yields median $\tilde M(\cdot)$18 for open→open pairs and $\tilde M(\cdot)$19 for open→closed pairs, but median $\tilde M(\cdot)$20 and $\tilde M(\cdot)$21, respectively. By contrast, white-box representation signals are highly stable across open models: median $\tilde M(\cdot)$22, median $\tilde M(\cdot)$23, and median $\tilde M(\cdot)$24. Yet cross-level fidelity remains low, with median $\tilde M(\cdot)$25–$\tilde M(\cdot)$26 and median $\tilde M(\cdot)$27. The paper terms this an access-validity inversion: attention patterns and perturbation magnitudes are stable across models but weakly predictive of causal attributions. It also reports a prediction-vs-attribution gap, confidence stratification in which fidelity improves on high-confidence examples, and a per-layer pattern in which functional agreement stays near zero for the first $\tilde M(\cdot)$28 of layers and rises sharply only near the readout. The main conclusion is precise: prediction-level agreement is insufficient to warrant transfer of mechanistic claims from an open surrogate to a closed target.

5. Representation design and low-rank surrogate construction

A distinct line of work concerns how surrogate representations should be encoded before any downstream inference is attempted. In surrogate modeling of High Entropy Alloys, the data are alloy compositions represented by atomic fractions of up to $\tilde M(\cdot)$29 elements, and the paper compares unstructured vectors, chemically ordered one-dimensional layouts, and a two-dimensional Periodic Table Representation (PTR) (Debnath et al., 2022). The unstructured input is

$\tilde M(\cdot)$30

where ordering is immaterial for dense networks. Structured one-dimensional representations instead assign the index according to periodic order, the Pettifor scale, a modified Pettifor scale, an alphabetic control, or a random permutation; these are processed by 1D CNNs. The PTR embeds elements into a $\tilde M(\cdot)$31 grid,

$\tilde M(\cdot)$32

and is processed by a 2D CNN. In single-task deep learning on glass-forming ability, the unstructured DNN achieves the highest F1, $\tilde M(\cdot)$33, while CNN1-atom, CNN1-pet, CNN1-modpet, and CNN2-PTR are all within $\tilde M(\cdot)$34 and not significantly different. By contrast, CNN1-alphabetic and CNN1-random perform significantly worse, around $\tilde M(\cdot)$35. In transfer learning, Random Forests on raw atomic fractions often outperform transfer features extracted from deep representations: for hardness regression, RF on raw fractions reduces RMSE from $\tilde M(\cdot)$36 HV to $\tilde M(\cdot)$37 HV, and for yield strength regression it reduces RMSE from $\tilde M(\cdot)$38 MPa to $\tilde M(\cdot)$39 MPa. The paper’s practical recommendation is explicit: chemically meaningful adjacency helps when the architecture has local spatial inductive bias, but raw-feature Random Forests remain extremely strong baselines in low-data regimes.

An earlier numerical-analysis formulation builds surrogates through low-rank separated representations of high-dimensional stochastic functions. For a vector-valued response $\tilde M(\cdot)$40 with $\tilde M(\cdot)$41, the surrogate is

$\tilde M(\cdot)$42

or, when physical dependence $\tilde M(\cdot)$43 is explicit,

$\tilde M(\cdot)$44

(Validi, 2013). Construction proceeds by Alternating Least Squares with Tikhonov regularization and a roughening matrix chosen so that $\tilde M(\cdot)$45. Model complexity is selected by a perturbation-based error indicator,

$\tilde M(\cdot)$46

and the optimal $\tilde M(\cdot)$47 minimizes this quantity. The reported computational scaling is notable: the number of required realizations $\tilde M(\cdot)$48 grows linearly in the dimension $\tilde M(\cdot)$49, and, under $\tilde M(\cdot)$50, construction cost behaves as $\tilde M(\cdot)$51 rather than exponentially. On a 41-dimensional elliptic PDE with $\tilde M(\cdot)$52 outputs, the surrogate built with $\tilde M(\cdot)$53 samples and optimal $\tilde M(\cdot)$54 attains average relative error in the mean of $\tilde M(\cdot)$55 and in the standard deviation of $\tilde M(\cdot)$56, while Monte Carlo requires $\tilde M(\cdot)$57 for comparable accuracy. In DRAM MCMC with $\tilde M(\cdot)$58 steps, the surrogate evaluation is negligible compared with $\tilde M(\cdot)$59 second per PDE solve, producing a reported $\tilde M(\cdot)$60 speed-up.

These two lines of work make different but compatible points. One emphasizes domain-structured encodings that align model inductive bias with chemically meaningful adjacency; the other emphasizes algebraic factorization and regularized low-rank approximation for high-dimensional stochastic maps. Taken together, they indicate that surrogate quality depends not only on function class and optimization, but also on whether the representation exposes the structure that downstream inference actually uses.

6. Validation, calibration, and failure modes

The principal limitation of SRI is that replacing the original object with a surrogate introduces a second inferential problem: validating the surrogate itself. In simulation-based inference, the paper identifies surrogate bias, NDE training instability, high output dimension, underdispersed posterior behavior under an under-trained first-round budget, and support-point non-robustness as the main failure modes (Refaeli et al., 16 Apr 2025). If the likelihood NDE misses multi-modality or high-frequency structure in $\tilde M(\cdot)$61, downstream posterior updates can be biased and may fail to concentrate around the true $\tilde M(\cdot)$62. If support-point optimization concentrates too strongly in sub-regions, exploration is harmed. The recommended response is active monitoring of surrogate quality through held-out simulator tests or simulation-based calibration, together with fallback to real-simulator sampling or hybrid schedules.

A closely related warning appears in Bayesian surrogate modeling: failing to propagate surrogate uncertainty can lead to biased and/or overconfident estimates of the parameters of interest (Reiser et al., 2023). This is borne out in the nonlinear logistic case study, where E-Lik and E-Post attain near-uniform SBC rank histograms for $\tilde M(\cdot)$63, whereas Point and E-Log-Lik do not until $\tilde M(\cdot)$64. The same paper notes that E-Post is trivially parallelizable but requires $\tilde M(\cdot)$65 independent samplings, while E-Lik uses one more complex model and can exploit within-chain threading. The practical issue is therefore not only correctness, but also how uncertainty propagation trades off with computational cost.

In mechanistic interpretability, validation must be claim-specific rather than representation-generic. High prediction fidelity, high attention similarity, or high perturbation-magnitude stability does not imply faithful transfer of causal or mechanistic attributions (Chlenski et al., 30 Jun 2026). The paper’s recommendation is correspondingly strict: black-box API claims should report $\tilde M(\cdot)$66 and $\tilde M(\cdot)$67, and representation-level claims should be grounded in $\tilde M(\cdot)$68, $\tilde M(\cdot)$69, $\tilde M(\cdot)$70, and $\tilde M(\cdot)$71 wherever open-weight comparisons are possible. The central misconception addressed in that literature is that agreement on answers implies agreement on reasons; the reported results show otherwise.

In annotation-based SRI, the main concern is not simulator bias but violation of the surrogate sufficiency and measurement-error assumptions. The paper’s contribution is that, when multiple human annotations are available and errors are non-differential, one can construct a surrogate outcome that restores unbiasedness without gold labels (Nakamura, 15 Sep 2025). The empirical result is not merely lower variance: existing methods can become severely biased and have invalid coverage under annotator error, whereas SRI with multiple annotators recovers unbiased estimates with correct coverage.

These validation regimes reveal a general pattern. SRI succeeds when the surrogate preserves the relevant conditional laws, latent adjustment structure, or causal readouts of the original problem, and it fails when surrogate adequacy is assessed only indirectly. This suggests that the central methodological question in SRI is not whether a surrogate is accurate in some global sense, but whether it is accurate at the exact inferential level—likelihood, posterior, causal bridge, attribution, or efficient score—on which the substantive claim depends.