The Agentic Garden of Forking Paths

Abstract: Empirical research rarely admits a unique analysis. Different analytical choices can lead to different conclusions from the same data, yet these hidden forking paths are difficult to observe. We show that AI agents capture much of the analytical variation among human researchers while making these paths explicit. Across four high-stakes domains, assigning different personas is sufficient for AI agents to report divergent, often opposing, conclusions from the same data and question, with findings systematically aligned with those beliefs. In a study in which 42 human research teams analyzed the same immigration dataset, AI agents reproduced 72% of the human ideological gap in reported effect estimates. Despite reaching opposing conclusions, it is difficult to identify clear issues in each analysis based on the final AI reports: 86% passed independent AI review and 78% passed majority human expert review. These findings suggest that the central challenge is often not flawed analyses, but selective exploration and reporting from a large space of methodologically defensible analyses. AI agents may amplify this longstanding problem by making such exploration inexpensive and scalable. To address this, we introduce the m-value (multiverse value), the probability that an analysis path would produce a claim at least as extreme as the reported one. We further introduce Agentic Bootstrap, which estimates the m-value by using AI agents to sample plausible analysis paths. Applied to the human immigration study, 13.5% of reported human analyses fell in the most extreme 5% of the analysis space (m<0.05). Scientific evidence should therefore be evaluated not only by a single reported analysis but also by its position within the distribution of analyses that could reasonably have been reported. Agentic Bootstrap makes this distribution observable and turns it into a criterion for scientific credibility.

• The paper shows that persona-conditioned AI agents can produce divergent yet robust analyses on identical datasets, highlighting analysis-space uncertainty.
• It introduces two technical novelties: the m-value to quantify analytical uncertainty and the Agentic Bootstrap to map divergent analytic paths.
• Empirical experiments across political science, public health, psychology, and biology reveal that belief-conditioned analytic choices systematically bias effect estimates.

Analytical Divergence in AI-Driven Empirical Science: Insights from "The Agentic Garden of Forking Paths"

Overview and Motivation

This paper provides a systematic investigation of analytical variability in empirical research through the lens of autonomous AI agents. Grounded in the tradition of the "garden of forking paths" and researcher degrees of freedom [gelman2013garden], the study demonstrates that persona-conditioned AI agents, when exploring the same dataset and scientific question, can reach divergent, often opposing, scientific conclusions. The key insight is that even analyses passing rigorous AI and expert human methodological review (i.e., with no major methodological errors) can yield sharply differing outcomes, mirroring ideological divergence observed among human researchers.

The work adopts four high-impact scientific domains (political science, public health, psychology, and biology) as testbeds and introduces two technical novelties: (1) the "m-value" as a calibrated tail probability quantifying analysis-space uncertainty and (2) the "Agentic Bootstrap," a protocol for empirically estimating the m-value by instrumenting AI agents to catalog extensive analysis pathways. The implications address both empirical reproducibility in the LLM era and the statistical auditing of scientific evidence.

Persona Priming and Systematic Divergence

Empirical research rarely specifies a unique analysis pipeline. Each dataset and question admits a combinatorial space of plausible, methodologically defensible analysis choices, including variable operationalization, covariate selection, modeling methods, and inclusion/exclusion criteria. The study operationalizes this space by tasking AI coding agents, primed with opposing personas reflecting domain-specific ideological stances, to conduct multi-stage analyses on identical questions and datasets. In each experiment, persona was the only randomized factor; data, prompt, and computational environment were held constant.

The results show striking systematicity: persona-primed agents reported effect estimates and inferential claims strongly aligned with their assigned beliefs, both in domains with directional ideological divides (e.g., immigration-welfare) and those with belief in/no-effect dichotomies (e.g., social media harm). Quantitatively, in political science (the immigration-welfare question), AI agents reproduced 72% of the ideological effect estimate gap observed in a large-scale human multi-analyst study [borjas2026ideological].

Figure 1: Persona-primed AI agents reach divergent conclusions across four scientific questions when only the persona prompt is varied.

On permuted-null data (i.e., destruction of the treatment-outcome link), persona-driven divergence persisted, confirming that observed differences originate from selective analytical exploration rather than real patterns in the underlying data. This aligns with concerns articulated in classical literature regarding flexibility-induced false-positive rates [simmons2011false, ioannidis2005most] and modern empirical findings that belief alignment can shape analytic choices [breznau2022observing].

Mechanisms: Exploration and Selection Dynamics

The agentic divergence arises via two temporally distinct, yet compounding, mechanisms: analytical exploration (agents traverse different regions of the analysis spec space) and final selection (selection of belief-consistent specifications for reporting). Round-wise logs of analyses reveal ensemble-level drift—gap near zero at initialization, growing monotonically across rounds, and further amplified during the final report selection phase.

Figure 2: Persona bias progressively increases during exploration rounds and is further amplified by selective reporting.

Linguistic analysis of agents' rationale traces demonstrates belief-aligned interpretation of the same empirical evidence, echoing phenomena of motivated reasoning in scientific practice.

Mapping the Analytical Space with Agent Trajectories

By instrumenting agents to log every candidate analysis during their multi-round exploration, the authors construct an empirical "agentic specification curve." For the ISSP immigration-welfare case, over 4,300 methodologically vetted specifications were logged. The resulting effect estimate distribution covers most of the spectrum present in human analyst outputs (including opposing signs of the effect).

Figure 3: Agentic specification curve displaying effect estimate variability and the analytical choices (e.g., immigration stock vs. flow, control strategy) driving this variation.

The specification curve reveals substantial overlap between opposing personas, confirming that divergence is not due to mutually-exclusive analytic regimes but arises from systematic over-sampling of belief-consistent analytic choices. Odds ratio analysis identifies certain choices (e.g., immigration measure) as most predictive of reported effect polarity. A CatBoost classifier trained solely on analytic decisions yields an AUC of 0.92 for predicting effect sign, validating that choices, not data, primarily drive divergent conclusions.

Quantifying Analysis-Space Uncertainty: m-value and Agentic Bootstrap

The standard inferential p-value only captures sampling uncertainty for a single fixed analytical path. To address model/analysis specification uncertainty, the study introduces the m-value—a tail probability over the empirical distribution of defensible analysis paths (i.e., the probability that a path drawn from this set is at least as extreme as the reported claim in the chosen test statistic).

Figure 4: Contrast of p-value (sampling tail probability) and m-value (analysis-space tail probability), the Agentic Bootstrap workflow, and the comparison of human and agentic multiverse distributions.

The Agentic Bootstrap empirically estimates the m-value distribution by aggregating thousands of agent-generated, review-passed analysis paths. Application to the human ISSP immigration-welfare study shows that 13.5% of human specifications fell in the most extreme 5% of the agentic reference space ($m<0.05$), and 40% of statistically significant human-reported findings ($p<0.05$) are analysis-space outliers ($m<0.05$), highlighting a strong prevalence of selective reporting. This analysis demonstrates that confidence intervals capturing only sampling variability may severely under-represent true uncertainty in the presence of analytical degrees of freedom.

Implications and Theoretical Considerations

This work demonstrates that the central reproducibility risk of AI-augmented data analysis is not the production of technically flawed analyses, but the ease of searching—and selection biasing—a vast latent space of statistically defensible but substantively divergent analyses. Persona-primed AI agents are efficient simulators of the human analytical multiverse, amplifying both the generation and observability of forking paths.

The m-value and Agentic Bootstrap offer principled diagnostics that extend multiverse/specification curve analysis [steegen2016increasing, simonsohn2020specification], delivering a measure (empirically calibrated and computable at scale) for a reported analysis’s typicality amongst all plausible analytic paths. This can act as a guardrail for both peer review and scientific meta-analysis.

Going forward, the integration of analysis-space diagnostics in empirical research pipelines—especially with rapidly evolving foundation model toolchains—is a necessary check on the interpretability and credibility of evidence claims. The protocol is agent- and prompt-dependent; so reproducibility mandates transparent declaration of all settings. The general approach is readily extensible to other domains with analysis multiplicity, from computational biology and psychology to AI benchmarking and meta-analytical synthesis.

Prospective Developments and Open Questions

This work opens several lines for future research: systematic evaluation of human-AI collaboration in empirical reporting (to analyze emergent effects of joint model-human analysis selection); broader application of Agentic Bootstrap to literature-wide meta-analyses; agentic re-analyses as a training curriculum for LLM scientific critique and summary; and development of standardized agent protocol registries for reproducible analysis-space auditing.

An important technical question is bounding the calibration and sensitivity of the m-value as the diversity and autonomy of agent analyst frameworks increase. Another is modeling (and possibly regularizing) joint human-AI interaction effects within the analytic selection process.

Conclusion

This study provides a rigorous and formalized account of how AI agents, via persona conditioning and autonomous exploration, recapitulate belief-aligned analytical divergence long documented among human researchers. It reveals that with modern LLM agents, analytical flexibility is both more scalable and more readily auditable. By introducing the m-value and Agentic Bootstrap, the paper establishes an empirical foundation for quantifying analysis-space uncertainty alongside traditional sampling error. The protocol positions AI-based analytical reproducibility as both a feature and a risk; robust future empirical science will require widespread adoption of analysis-space diagnostics to ensure the credibility of reported findings.