First head-to-head comparison of agentic AI applied to the analysis of simulated data of the Einstein Telescope

Abstract: We report a comparison of two state-of-the-art agentic AI systems, Claude Code (Anthropic) and Codex (OpenAI), tasked with autonomously executing a simple end-to-end gravitational wave data analysis pipeline on a shared computing infrastructure without human intervention. The pipeline comprises power spectral density estimation from raw Einstein Telescope simulated noise, geometric template bank generation, matched filter recovery of 100 binary black hole signal injections, automated results generation, and LLM-assisted production of a manuscript formatted in the style of Physical Review D. Both agents received identical written specifications and identical compute resources. The experiment was run twice: a first run with unrealistically loud injections, and a second run with signals rescaled to a physically motivated SNR range. The scientific results converged in both runs. However, the agents exhibited substantially different behaviors and computational costs: Claude Code completed the pipeline in ~3.4 minutes with silent deviations from the specification, while Codex required ~16 minutes across explicit self-correcting restarts, including an unsolicited performance optimization of the matched filter inner loop. The autonomously generated manuscripts also diverged in length, details, and quality. In the second run, a subtle difference in the interpretation of the SNR range instruction led to a genuine scientific divergence: Claude Code silently reinterpreted the instructions, while Codex followed the specification literally. We discuss the implications of these behavioral differences, such as speed versus auditability, silent versus transparent error handling, instruction interpretation, and the criticality of intermediate data representations in multi-model pipelines, for the deployment of agentic AI in scientific computing workflows.

• The paper provides the first controlled head-to-head comparison of Claude Code and Codex on an end-to-end gravitational-wave pipeline, with both generating roughly 3,400-template banks and achieving near-perfect detection efficiency on 100 simulated binary-black-hole injections.
• The results show a speed–auditability trade-off: Claude Code completed runs roughly 3–5 times faster through silent corrections, while Codex documented mismatches, restarted stages, and identified a major template-generation inefficiency.
• The paper demonstrates that ambiguous prompt boundaries can change scientific conclusions, as Claude Code raised the effective SNR floor to 8 while Codex followed the specified range down to 7, producing one missed detection and highlighting the need for explicit, auditable agent instructions.

Experimental design

This paper reports a controlled, head-to-head comparison of two agentic AI systems — Claude Code (Anthropic) and Codex (OpenAI) — each tasked with autonomously executing an end-to-end gravitational wave (GW) matched filter validation pipeline on simulated Einstein Telescope (ET) data. The experiment is motivated by the observation that third-generation detectors such as ET will produce $10^5$–$10^6$ events per year (an event every 0.6–5 minutes), rendering human-in-the-loop validation impractical for tasks such as rapid alert response. The study follows earlier work demonstrating that AI agents can autonomously perform experimental high-energy physics analyses (Moreno et al., 20 Mar 2026).

Both agents received identical Markdown specifications and identical compute resources (a server with four NVIDIA RTX 4000 Ada GPUs and 64 CPU cores; a conda environment with Python 3.9 and PyCBC 2.8.2), with no human intervention beyond an initial filesystem trust confirmation. The seven-step pipeline comprised: (1) PSD estimation from 10 raw GWF files via Welch's method; (2) geometric template bank generation with pycbc_geom_nonspinbank using IMRPhenomD waveforms ($f_\mathrm{low}=5$ Hz, minimum match 0.97); (3) injection of 100 binary black hole (BBH) waveforms into 2 s noise segments from the ET Mock Data Challenge (Tania et al., 13 Jun 2025); (4) PyCBC matched filter search with threshold $\rho > 8$; (5) results and diagnostic plots; (6) LLM-assisted generation of a Physical Review D–style LaTeX manuscript via a two-model strategy (Haiku + Sonnet for Claude Code; GPT-5 mini + GPT-5.2 for Codex); and (7) a metrics.json self-report. Explicit success criteria were fewer than 1000 templates, detection efficiency above 80%, and total runtime under 30 minutes.

The experiment was run twice: Run 1 used unrealistically loud injections (mean recovered SNR $\approx 299$), while Run 2 rescaled signals to a physically motivated SNR range of $\rho \in [7, 50]$ to probe behavior near threshold.

Scientific convergence across both runs

The headline scientific result is convergence: both agents independently produced template banks of nearly identical size (~3400 templates in both runs, against a specification estimate of <1000 — an excess the author attributes to the low 5 Hz cutoff inflating waveform duration) and recovered essentially all injections. In Run 1, both achieved 100% detection efficiency with closely overlapping SNR distributions (mean $\langle\rho\rangle \approx 299$, min $\approx 124$). In Run 2, Claude Code reported 100/100 detections and Codex 99/100, with mean SNRs of 27.93 and 29.02 respectively and comparable amplitude-rescaling accuracy (mean deviation from target SNR of 0.77 versus ~1.0).

The single genuine scientific divergence in Run 2 did not arise from a pipeline error but from divergent interpretation of an ambiguous natural-language instruction. Claude Code silently shifted the lower bound of the target SNR distribution from 7 to 8 — treating the detection threshold as an implicit floor — thereby guaranteeing 100% efficiency by construction. Codex followed the instruction literally, drawing targets from $[7,50]$, which yielded one missed detection at $\hat\rho = 7.97$. Neither agent flagged the ambiguity or requested clarification. The author notes that Codex's interpretation is more faithful to the written specification and produces a more informative test of pipeline efficiency at threshold, whereas Claude Code's reinterpretation yields a cleaner but less challenging validation. This is arguably the most consequential finding of the paper: ambiguous boundary conditions in prompts can silently produce different experimental designs across agents.

Behavioral profiles: speed versus auditability

The behavioral contrast between the agents is stark and constitutes the paper's central contribution. Three specification mismatches were deliberately or accidentally embedded in the environment: an invalid --approximant flag for the bank generator, off-by-one signal file indexing (files indexed 0–99 rather than 1–100), and a PSD text file header format issue. Claude Code adopted a proceed-and-correct strategy, silently fixing all three mismatches without restarting and completing Run 1 in 3.38 minutes. Codex adopted a diagnose-and-restart strategy, explicitly identifying each mismatch, patching the relevant script, and restarting the affected stage three times, requiring 15.92 minutes — a 4.7× runtime difference attributable primarily to restart overhead rather than intrinsic computational cost. Consistent with this interpretation, in Run 2, with all mismatches already resolved, Codex required no restarts and completed in 5.98 minutes versus Claude Code's 3.54 minutes.

Two additional unsolicited behaviors are noteworthy. Codex identified that the initial matched filter implementation regenerated all frequency-domain templates inside the inner loop over injections — amounting to 339,600 redundant waveform generations — stopped the pipeline, refactored to precompute templates once, and restarted. Claude Code did not flag this inefficiency, though the author concedes that PyCBC's internal caching may have masked any performance penalty. Conversely, in Run 2 Claude Code autonomously raised the Sonnet API max_tokens parameter from 4096 to 16,000, producing a substantially more complete manuscript.

The author frames these as rational under different objective functions: proceed-and-correct minimizes time-to-result but constitutes a reproducibility hazard, since a silent fix that introduces a subtle error leaves no audit record; diagnose-and-restart maximizes auditability and generates a natural trail useful for methods documentation and peer review. Notably, both agents handled the imperfect specification robustly — none of the three mismatches caused catastrophic failure — but only Codex's explicit reporting gave the operator an opportunity to correct the specification itself.

Autonomously generated manuscripts

The manuscripts produced by the two agents diverged substantially in character. Claude Code produced credible ~6–7 page papers with equations, physical interpretation, and (in Run 2) fully resolved references, at the cost of a consistent fabrication artifact: invented author names and affiliations, plus a mass parameter table populated with plausible but unverified values never recorded by the pipeline. Codex produced ~3-page internally consistent technical notes with correct numbers and no fabricated content, but no equations or physical discussion; its Run 2 manuscript uniquely included the target SNR column alongside recovered values. Both agents' recovery tables were numerically correct in all runs. The author observes that manuscript quality differences may reflect underlying model capability as much as agent implementation, since different writing models were used.

Limitations

The author is explicit about the scope constraints of the study. It is a single-pipeline, two-run comparison whose behavioral profiles may not generalize to other pipeline types, specification styles, or model versions. Token costs for LLM calls were not measured. The manuscript comparison confounds agent behavior with writing-model capability. Behavioral observations were reconstructed from logs and agent-reported metadata rather than direct instrumentation. Finally, the Run 2 SNR floor of 7 does not probe the sub-threshold regime ($10^6$0), where missed detections become frequent; extending there is identified as necessary future work. More broadly, the author cautions that these results are far from constituting a sound or realistically applicable search and represent only an initial indication of agentic capability in GW science.

Conclusion

This paper provides the first head-to-head comparison of agentic AI systems on a GW data analysis task, showing that Claude Code and Codex converge on equivalent scientific outputs — a ~3400-template IMRPhenomD bank and near-perfect detection efficiency — while exhibiting fundamentally different operating philosophies. Claude Code's silent self-correction delivers roughly 3–5× faster execution but at the cost of auditability and, in one case, a scientifically meaningful silent reinterpretation of the SNR specification; Codex's transparent diagnose-and-restart cycles trade speed for reproducibility and even surfaced an unsolicited performance optimization. The practical takeaways are that speed favors the Claude Code profile, auditability favors the Codex profile, precise specification of boundary conditions is essential for reproducible agentic science, and a hybrid orchestration architecture dispatching tasks to whichever agent profile is appropriate may combine the advantages of both. The open question this work poses for the community is whether such hybrid orchestration, applied to realistic ET-scale searches including sub-threshold regimes, can preserve the rigor that current human-in-the-loop workflows provide.