Benchmarks to quantify cost–accuracy frontiers for surrogate training

Develop open benchmarks that pair instrumented corpora with surrogate models to quantify when training surrogates amortizes the generation cost of instrumented data and to characterize the resulting cost–accuracy trade-offs.

Background

Generating instrumented data can be expensive for complex simulations; training surrogates is a common approach to amortize this cost. However, the break-even points and trade-offs are not quantified systematically across domains.

Open benchmarks would enable consistent measurement of when and how surrogates achieve acceptable accuracy at reduced cost, guiding corpus design and deployment decisions.

References

Nine open questions will determine whether instrumented data matures into a recognised substrate for scientific machine learning. Cost--accuracy frontiers. Open benchmarks pairing instrumented corpora with surrogates are needed to quantify when surrogate training amortises the substrate's cost.

— Instrumented data for causal scientific machine learning  (2606.07865 - Wilke, 5 Jun 2026) in Section 7, Methodological questions for the community, Item 5

We have not adopted it for the present release because it requires retraining and its cost in leave-one-out error has not yet been measured.

— NRHJSur3dq8: a spectral numerical-relativity surrogate family for non-eccentric aligned-spin binary-black-hole waveforms  (2609.09088 - Prasad, 8 Sep 2026) in Section 9, subsection “Exchange symmetry and the accuracy floor set by the training data” (Section \ref{sec:exchange})

The added cost, dominated by training a network at each of the O(500) design points, remains to be quantified against the Bayesian analysis itself, and it is not yet established whether a few hundred events suffice across the full parameter space, where the structure of the residual may change with the model parameters, for instance the nucleon width.

— Final-state centrality in relativistic heavy-ion simulations from neural networks  (2610.13828 - Bezerra et al., 9 Oct 2026) in Section VI, concluding discussion

Finally, for cost reporting, studies use different model scales, token budgets, and hardware. Which accounting protocol can compare them while separating amortisable setup from computation repeated for every target run?

— Data Strategies for Large Language Models: A Unified Perspective on the Training Lifecycle  (2610.21681 - Xu et al., 15 Oct 2026) in Section 3, subsection “Summary and Discussion,” paragraph “Open questions”