Transfer Learning Across Fast- and Full-Simulation Domains in High-Energy Physics

Abstract: Machine-learning models in high-energy physics are often trained on simulated data, where fully simulated samples are computationally expensive while fast simulation provides large statistics at reduced realism. In this work, we systematically study transfer learning between fast-simulated and fully simulated datasets in a realistic LHC environment. We consider three representative tasks, signal-background classification, quark-gluon jet tagging, and missing transverse energy reconstruction, using dense neural networks, graph neural networks, and transformer-based architectures. Models are pretrained on ATLAS-like fast simulation and adapted to CMS-like fast simulation and to fully simulated ATLAS Open Data. Across all tasks, pretrained models consistently outperform independently trained baselines and require significantly less target-domain training data, typically reducing the needed statistics by about a factor of two. These results demonstrate that fast simulation can be used to learn robust, reusable representations and motivate publishing trained models as reusable scientific assets beyond large foundation models.

• The paper demonstrates that pretraining on FastSim data significantly reduces the required FullSim samples while preserving model performance.
• It leverages architectures like DNNs, GNNs, and transformers to address domain shifts in signal-background classification, jet tagging, and Eₘiss regression.
• Results confirm that full fine-tuning outperforms partial freezing, enabling robust cross-domain adaptation and improved physics fidelity.

Transfer Learning Between Fast and Full Simulation Domains for Neural Networks in High-Energy Physics

Introduction

This paper addresses a critical challenge in high-energy physics (HEP): leveraging simulated data for ML model training where fully detailed detector simulations ("FullSim", e.g., GEANT4-based) are computationally prohibitive, and faster alternatives ("FastSim", e.g., DELPHES) enable large-scale data generation but introduce realism gaps. The authors systematically evaluate transfer learning strategies that bridge these domains, demonstrating efficiency and robustness gains in ML models across major LHC analysis tasks. The selected benchmarks—signal-background classification, quark-gluon jet tagging, and missing transverse energy ($E_\text{miss}$) reconstruction—employ architectures reflecting standard practice: dense NNs, GNNs, and transformers.

Experimental Design and Dataset Construction

The experimental setting specifically targets domain shifts relevant to LHC physics. The studies utilize three simulation domains: ATLAS-like FastSim (DELHPES, no pile-up), CMS-like FastSim (DELHPES, pile-up), and FullSim ATLAS Open Data (GEANT4-based). The domains are differentiated not just by detector response and pile-up but also by variations in physics modeling (e.g., parton shower, matrix elements, underlying-event tunes), intensifying the domain adaptation problem.

Datasets span four representative processes: $t\bar{t}$ (semi-leptonic), $W$/Z+jets (leptonic), and $WW$ (semi-leptonic), with analysis-level object selection consistent with LHC standards. Sample sizes ($\mathcal{O}(10^5)$ events per domain and process) ensure statistical precision. This design allows a controlled investigation of both moderate (ATLAS→CMS FastSim) and severe (FastSim→FullSim) domain shifts.

Model Architectures and Pretraining Hypothesis

Each ML task targets distinct input and architectural regimes:

• Signal-background classification: event-level DNN on 12 kinematic features; a multi-layer perceptron with dropout regularization.
• Quark-gluon tagging: jet-level GNN (EdgeConv-based) on up to 50 charged-particle tracks, dynamically constructing a $k$NN graph per jet.
• $E_\text{miss}$ regression: event-level transformer on trackwise four-vector and impact parameter inputs, using self-attention to capture global event structure.

The central hypothesis asserts that pretraining on FastSim enables effective transfer to new domains, reducing the amount of target-domain data necessary to match or exceed the performance of from-scratch training, and that task-specific architectures (not only large foundation models) can benefit from this paradigm.

Transfer Learning Protocol and Strategies

The models are first pretrained on comprehensive ATLAS FastSim data. Transfer to a new domain is evaluated via two strategies:

1. Full fine-tuning: All parameters are retrained in the target domain.
2. Partial retraining: Early network layers are frozen, only later layers (or output heads) adapt.

Performance is systematically benchmarked as a function of available target-domain training statistics. Multiple random seeds for train/validation/test splits and weight initialization control for statistical uncertainty.

Key Results

Signal-Background Classification

Pretrained DNNs consistently and significantly outperform independently trained baselines in both FastSim→FastSim and FastSim→FullSim adaptation. For ATLAS→CMS transfer, strong reuse of latent features enables superior performance with only a small fraction of target-domain data. For ATLAS FastSim→FullSim, the pretrained network with full retraining achieves the same area-under-ROC (AROC) performance as scratch-trained models with approximately half the training data. Freezing early layers is sub-optimal for large domain shifts, indicating the necessity for representational flexibility under severe domain discrepancy.

Quark-Gluon Jet Tagging

Quark-gluon GNNs exhibit parallel trends. For moderate shifts (ATLAS→CMS FastSim), partial freezing is effective in the low-statistics regime, indicating robust generalization of jet substructure encodings. For FastSim→FullSim, only full fine-tuning allows the model to accommodate the combined shifts in detector and physics simulation; freezing GNN layers degrades performance. Transfer learning achieves comparable AROC with approximately half the events.

$E_\text{miss}$ Regression

This regression task, being fundamentally sensitive to subtle detector effects, displays more nuanced behavior. Transfer between fast simulation domains yields only modest gain, as the models can learn necessary corrections de novo with moderate data. However, for FastSim→FullSim transfer, pretraining provides substantial data efficiency (halving the required training set), and the resolution and bias of the reconstructed $E_\text{miss}$ show improved physical fidelity. Notably, in all cases, full retraining is necessary; freezing transformer encoder layers is consistently detrimental.

Cross-Experimental Training

A proof-of-principle study mixing ATLAS and CMS FullSim data demonstrates that combined training does not degrade detector-specific performance and can enhance model generalization in low-statistics settings. This approach offers a route to alleviate the extreme data requirements posited by scaling law studies for next-generation flavor tagging.

Implications

This work establishes the following:

• Robustness of FastSim representations: Pretrained models learn reusable, physics-rich encodings effective even under significant simulation and detector shifts, underpinning the legitimacy of ML-based studies relying on FastSim given appropriate fine-tuning.
• Architectural generality: The benefit of transfer learning extends to common, resource-efficient network designs and is not limited to large foundation models.
• Model sharing: The practice of publishing pretrained weights, in addition to code and datasets, is highlighted—enabling reproducibility, cross-experiment synergy, and more efficient downstream analysis.
• Data efficiency: Systematic application of transfer learning can halve the demand for costly FullSim data across core HEP tasks.
• Simulation strategy: Combining data across experimental collaborations (e.g., ATLAS+CMS) is viable for scaling training, offering a practical path to approach ultimate tagger performance.

Future Directions

Several avenues for further research are identified:

• Application to real collision data, encompassing full sim-to-real transfer challenges;
• Extension to additional tasks, detector configurations, and more complex physics processes;
• Large-scale cross-experimental pretraining studies, exploiting public LHC Open Data to approach the scaling regimes relevant for ultimate ML performance in flavor tagging and object identification.

Conclusion

This paper rigorously demonstrates that transfer learning across FastSim and FullSim domains is a practical, broadly applicable mechanism for maximizing the utility of limited FullSim data, accelerating ML-driven scientific discovery at the LHC. The clarified conditions for optimal transfer (necessity of full retraining for large domain shifts), architectural independence of the approach, and the viability of cross-experiment training collectively establish a foundation for more sustainable, efficient, and reproducible ML practice in high-energy physics.

Reference: "Transfer Learning Across Fast- and Full-Simulation Domains in High-Energy Physics" (2605.07471)