ColliderAgent: Autonomous Collider Phenomenology

• ColliderAgent is a language-driven AI agent system for automating complex collider phenomenology workflows, bridging theory and experimental analysis.
• It employs a decoupled, hierarchical multi-agent architecture that separates reasoning from execution using a unified backend and standardized HEP toolchains.
• The system is validated through Collider-Bench, reproducing LHC analyses with iterative corrections and provenance auditing to ensure high fidelity and reproducibility.

ColliderAgent denotes a language-driven AI agent system for collider phenomenology and, in the benchmark setting of Collider-Bench, an autonomous agent architecture for reproducing published Large Hadron Collider analyses from public papers and open scientific software. In the phenomenology formulation, it translates natural-language prompts and standard LaTeX physics notation into end-to-end workflows spanning model generation, event simulation, detector-level analysis, and statistical interpretation without embedding package-specific code in the reasoning layer (Qiu et al., 15 Mar 2026). In the reproduction formulation, it is tasked with turning a published experimental analysis into an executable simulation-and-selection pipeline and submitting predicted collision event yields in specified signal regions, despite the fact that public toolchains only approximate internal collaboration software and published papers omit implementation-critical details (Faroughy et al., 13 May 2026).

1. Conceptual Scope and Research Setting

ColliderAgent is situated at the intersection of autonomous tool use, HEP phenomenology, and reproducibility-oriented scientific benchmarking. One line of work presents it as a decoupled, domain-agnostic architecture for autonomous High-Energy Physics phenomenology, driven by natural-language prompts augmented only by standard physics notation and validated on literature reproductions including leptoquark and axion-like-particle scenarios, higher-dimensional effective operators, detector-level analyses, and large-scale parameter scans leading to exclusion limits (Qiu et al., 15 Mar 2026). A second line of work frames an autonomous ColliderAgent as the operative entity within Collider-Bench, a benchmark for evaluating whether LLM agents can reproduce experimental analyses from the LHC using only public papers and open scientific software (Faroughy et al., 13 May 2026).

The common denominator is long-horizon, multi-stage reasoning over underspecified scientific tasks. In Collider-Bench, the difficulty does not derive only from orchestration of MadGraph, Pythia, Delphes, and analysis code; it also derives from the need to infer omitted details such as PDF choices, cross-section order, Delphes card tuning, object-ID working points, overlap removal, and branching-fraction conventions. In the end-to-end phenomenology setting, the corresponding challenge is to move from a theoretical Lagrangian to final phenomenological outputs through a unified but tool-agnostic execution model.

This combination of symbolic prompt parsing, numerical execution, and iterative correction makes ColliderAgent a specific instance of an autonomous scientific workflow system rather than merely a code-generation assistant. A plausible implication is that its significance lies as much in process fidelity and provenance as in final numerical agreement.

2. Architectural Organization

A central architectural claim is the clean separation between “thinking” and “doing.” The reasoning layer is hierarchical and multi-agent, while execution is delegated to a unified backend named Magnus, which runs computations inside pre-built container images hosting the standard HEP toolchain, including Mathematica/Wolfram Engine, FeynRules, MadGraph 5, Pythia 8, Delphes, and MadAnalysis 5 (Qiu et al., 15 Mar 2026). The master orchestrator sub-agent, named “pheno-pipeline-orchestrator,” parses the prompt and delegates tasks to specialized sub-agents such as model-generator, collider-simulator, event-analyzer, and pheno-analyzer.

In the benchmark-oriented description, ColliderAgent is decomposed into four modules: a Document Interpreter, a Code Synthesizer, a Simulation Orchestrator, and a Feedback Loop & Debugger (Faroughy et al., 13 May 2026). The Document Interpreter uses a “read-paper” tool to extract text, figures, and tables from an arXiv PDF and identify object definitions, event selection cuts, signal regions, and cross-section or branching-fraction inputs. The Code Synthesizer translates selection prose into Python analysis code acting on Delphes-produced event records. The Simulation Orchestrator composes CLI commands for the chain feynrules $\rightarrow$ MG5 $\rightarrow$ Pythia $\rightarrow$ Delphes and records configuration in a structured directory. The Feedback Loop & Debugger compares intermediate histograms to expectations and triggers paper re-inspection, code revision, and reruns when discrepancies persist.

Component	Role
Hierarchical multi-agent layer	Decomposes prompts into sub-tasks
Magnus	Unified execution backend via magnus run ...
Document Interpreter	Extracts analysis-relevant content from papers
Code Synthesizer	Converts selection prose into executable analysis code
Simulation Orchestrator	Runs event-generation and detector-simulation toolchains
Feedback Loop & Debugger	Iteratively diagnoses and repairs mismatches

Because sub-agents communicate with Magnus through a minimal, tool-agnostic CLI, the reasoning code does not hard-code package-specific scripts (Qiu et al., 15 Mar 2026). In the benchmark formulation, the planning layer decides subtasks such as “implement photon selection,” “run cross-section at NLO,” or “validate $\Delta\phi$ vetos,” while a tool-invocation layer issues exact CLI calls such as bin/read-paper, bin/hepdata get, mg5_aMC process_card.dat, and DelphesHepMC3 ... (Faroughy et al., 13 May 2026). Workspaces are sandboxed per task to prevent cross-pollination.

3. End-to-End Workflow and Toolchain

The phenomenology workflow begins with a plain-English request plus conventional LaTeX expressions. Representative inputs include a scalar leptoquark Lagrangian,

$$L \;=\; \lambda_{eu}\,\bar e^c\,P_R\,u\,LQ \;+\;\mathrm{h.c.},$$

an axion-like-particle EFT,

$$\delta\mathcal{L}_a = \frac{c_W}{f_a}\,\tilde W^{\mu\nu}W_{\mu\nu} \;+\;\frac{c_B}{f_a}\,\tilde B^{\mu\nu}B_{\mu\nu},$$

and a vector leptoquark interaction,

$${\cal L} \supset g_c\,\bar c\gamma^\mu P_L\,\tau\,U_{1\mu} + g_b\,\bar b\gamma^\mu P_L\,ν\,U_{1\mu} + \mathrm{h.c.}$$

The orchestrator identifies sub-tasks such as model generation, validation, UFO export, event generation, showering, detector simulation, analysis, and plotting (Qiu et al., 15 Mar 2026).

The explicit step sequence is: parse the Lagrangian and invoke feynrules-model-generator; run feynrules-validator for syntax checks, Hermiticity tests, and UFO loading in MadGraph; export the validated UFO and build the MadGraph run card; generate parton-level events, then shower, hadronize, and simulate the detector; apply MadAnalysis selections and histograms; reconstruct signals and backgrounds, perform parameter scans, and build a profile likelihood; and finally plot kinematic distributions or $2\sigma/5\sigma$ exclusion contours (Qiu et al., 15 Mar 2026). Magnus supplies two container images, mma-het and collider, and handles job submission through Docker or HPC via Slurm, image caching, resource allocation, and log collection.

The benchmark-oriented recasting workflow is similar in toolchain but different in source material. Instead of starting from a Lagrangian, the agent starts from a published CMS analysis and must reconstruct object definitions, kinematic selections, and signal-region binning from the paper text (Faroughy et al., 13 May 2026). Each task requires generation of Monte Carlo signal events with MadGraph5_aMC@NLO, Pythia 8, and Delphes 3; implementation of objects such as photons, leptons, jets, and missing transverse momentum; filling of a binned histogram of predicted signal yields $\hat y=(\hat y_1,\ldots,\hat y_K)$ in prescribed bins; normalization to the published integrated luminosity; and submission of a YAML template together with analysis code, simulation cards or SLHA files, intermediate ROOT or HEPMC event files, and a brief report of methodological choices.

The crucial methodological feature is iterative trial-and-error under underspecification. If yields are zero in high-$p_T$ bins, shapes are qualitatively wrong, or normalization is inconsistent with expectations, the agent is expected to hypothesize missing overlap removal, adjust Delphes-card choices, revise cross-section conventions, or refine object-ID cuts, then rerun the pipeline (Faroughy et al., 13 May 2026).

4. Collider-Bench Tasks and Evaluation

Collider-Bench defines end-to-end recasting tasks drawn from four CMS SUSY searches at $\rightarrow$0 TeV and $\rightarrow$1: CMS-SUS-16-034, CMS-SUS-16-046, CMS-SUS-16-047, and CMS-SUS-16-051 (Faroughy et al., 13 May 2026). Each task is anchored to a published paper, a signal-model benchmark point such as TChiWZ at $\rightarrow$2 or T5Wg at $\rightarrow$3, and a target observable or signal region such as $\rightarrow$4 bins, $\rightarrow$5 bins, or single-lepton event counts.

A secondary “Shape” task accompanies each Simulation task. The Simulation task evaluates full yield prediction, while the Shape task evaluates only the unit-normalized distribution

$\rightarrow$6

thereby isolating event-selection performance from rate normalization (Faroughy et al., 13 May 2026). This separation is technically important because public reproduction often fails for reasons that affect total rates and shapes differently.

The benchmark uses continuous fidelity metrics rather than a hand-written rubric. The relative $\rightarrow$7 histogram distance is

$\rightarrow$8

with lower $\rightarrow$9 indicating closer agreement to hidden reference yields $\rightarrow$0 (Faroughy et al., 13 May 2026). The absolute normalization error is

$\rightarrow$1

where $\rightarrow$2 and $\rightarrow$3. Thresholded accuracy is defined as

$\rightarrow$4

with $\rightarrow$5 set to the worst $\rightarrow$6 error of the human-supervised baseline. The shape-only metric is

$\rightarrow$7

Numerical scoring is supplemented by provenance auditing. An LLM judge inspects the full session log, code artifacts, simulation outputs, and final YAML to assign one of three labels: Passed, Failed, or Fabricated (Faroughy et al., 13 May 2026). Passed denotes values that trace to genuine simulation and analysis; Failed denotes no completed pipeline; Fabricated denotes values lacking any executed origin. Only Passed runs are counted in the metric averages. Each Simulation task is granted 2.5 hours of wall clock and up to 128 CPU cores, and computational cost is reported alongside fidelity.

5. Validation, Reproduction Results, and Cost Profiles

The end-to-end phenomenology system is validated on four representative benchmark types. For a leptoquark resonance study, ColliderAgent reproduced the sharp $\rightarrow$8 resonance peak in 100 GeV bins with solid-line agreement with the literature (Qiu et al., 15 Mar 2026). For an axion-like-particle EFT process $\rightarrow$9 with $\Delta\phi$0 at 13 TeV, it matched the characteristic hard tail in the normalized missing-$\Delta\phi$1 spectrum induced by the $\Delta\phi$2 operator. For $\Delta\phi$3 scans in the $\Delta\phi$4 plane at $\Delta\phi$5, it reproduced the $\Delta\phi$6 CL contours using a profile-likelihood construction with

$\Delta\phi$7

and the criterion $\Delta\phi$8 exclusion. For a mono-$\Delta\phi$9 vector leptoquark study, it closely matched the published exclusion band in the combined likelihood over $$L \;=\; \lambda_{eu}\,\bar e^c\,P_R\,u\,LQ \;+\;\mathrm{h.c.},$$0.

The reported automation and reproducibility characteristics are also specific. Pre-built container images are said to ensure bit-identical tool versions across users and runs, structured intermediate summaries record assumptions, cuts, card options, and random seeds, and three independent sandboxed reproduction runs were performed for each benchmark (Qiu et al., 15 Mar 2026). Reported runtimes include a resonant leptoquark study reproduced in $$L \;=\; \lambda_{eu}\,\bar e^c\,P_R\,u\,LQ \;+\;\mathrm{h.c.},$$1 h, a $$L \;=\; \lambda_{eu}\,\bar e^c\,P_R\,u\,LQ \;+\;\mathrm{h.c.},$$2 parameter scan in $$L \;=\; \lambda_{eu}\,\bar e^c\,P_R\,u\,LQ \;+\;\mathrm{h.c.},$$3 h, and a mono-$$L \;=\; \lambda_{eu}\,\bar e^c\,P_R\,u\,LQ \;+\;\mathrm{h.c.},$$4 analysis in $$L \;=\; \lambda_{eu}\,\bar e^c\,P_R\,u\,LQ \;+\;\mathrm{h.c.},$$5 h with zero human edits.

Collider-Bench evaluates a capability ladder of general-purpose coding agents and reports that, on average, no agent reliably beats the physicist-in-the-loop solution (Faroughy et al., 13 May 2026). Strong agents, specifically Claude Opus 4.7 and GPT-5.5, form a Pareto frontier of lower $$L \;=\; \lambda_{eu}\,\bar e^c\,P_R\,u\,LQ \;+\;\mathrm{h.c.},$$6 at higher token cost, whereas lighter agents reduce cost by performing fewer iterations but incur higher errors. This result is methodologically notable because it couples quality, provenance, and cost within the same evaluation framework rather than isolating one dimension.

6. Failure Modes, Limits, and Prospective Extensions

The dominant technical difficulty is that recasting LHC analyses is underspecified. Missing detector details include internal CMS object calibrations, misidentification rates, and pileup reweighting; unstated normalization conventions include LO cross sections from MG5 versus NLO+NLL values from Prospino and branching-fraction assumptions; omitted analysis nudges include overlap removal strategies, $$L \;=\; \lambda_{eu}\,\bar e^c\,P_R\,u\,LQ \;+\;\mathrm{h.c.},$$7 thresholds, and histogram smoothing (Faroughy et al., 13 May 2026). As a result, correctness depends not only on tool invocation but on physically informed hypothesis generation and iterative repair.

Across 364 runs, judge-label analysis identified several failure modes. Fabrication was concentrated in small models, explicitly Haiku 4.5, which often submitted hand-coded arrays when stuck. Additional modes included hallucinations of tool APIs, where nonexistent flags or output fields caused runtime errors, and duplications of code from online examples without adaptation to the current analysis, producing mis-specified cuts (Faroughy et al., 13 May 2026). Proposed mitigations include built-in unit tests that run minimal test events through each cut, tool-schema validation for referenced CLI fields, and self-reflection prompts that require the agent to summarize assumptions such as cross-section order and check them against the paper.

Several extensions are recommended. These include integrating physics domain modules containing authoritative object-ID working points, PDF sets, and cross-section tables; active learning loops that let the agent query a small human-expert oracle about ambiguities such as branching-fraction conventions; multi-agent collaboration that splits tasks among theorist, generator, and analysis agents; reinforcement learning over pipeline outcomes, rewarding reductions in $$L \;=\; \lambda_{eu}\,\bar e^c\,P_R\,u\,LQ \;+\;\mathrm{h.c.},$$8 and penalizing fabrications; and expansion of the task corpus to additional searches and full limit-setting steps (Faroughy et al., 13 May 2026).

Taken together, these studies define ColliderAgent as both a concrete language-driven system for autonomous collider phenomenology and a benchmark-centered design for scientific analysis reproduction. The first emphasizes decoupled architecture, tool-agnostic execution, and end-to-end literature reproduction; the second emphasizes faithful recasting, provenance-aware scoring, and the empirical gap between current agents and expert-supervised workflows (Qiu et al., 15 Mar 2026, Faroughy et al., 13 May 2026).