Closing the Loop to Discover Psychological Theories with an Automated Cognitive Scientist

Abstract: Across the sciences, autonomous systems are increasingly being used in closed-loop discovery, proposing new theories and designing and running experiments to test them. This approach is yet to be applied in the field of cognitive science, where the central bottleneck is theory-building: the creative step of turning the accumulated failures of existing models into better ones. Theory generation has remained manual even as data collection, modeling, and experiment design have been automated. We present the Automated Cognitive Scientist (AutoCog), a fully autonomous agentic-AI system that closes this loop. Large-language-model agents advocate competing theories, each expressed as an executable cognitive model, design experiments that best discriminate them, collect behavioral data from participants recruited online, score theories against collected data based on their generative performance, diagnose why they fail, and synthesize a better successor. Repeating this cycle allows them to search the space of theories, models, and experiments. In the domain of decision-making, AutoCog recovered known decision-making strategies from simulated behavior, including unconventional ones, showing that its discoveries are ultimately driven by the data rather than strictly bound by the priors of the underlying LLMs. When run with human participants, it produced theories that outperformed the established theories it was seeded with and generalized to held-out studies in two different experimental settings. It also surfaced a novel theory of multi-cue decision-making in which choices show diminishing sensitivity to feature values. The distinctive predictions of this theory were confirmed in a preregistered study with new participants. AutoCog demonstrates how an automated discovery system can be used to turn cognitive theory-building into an explicit, executable, and cumulative science.

• The paper introduces AutoCog, an automated agent-based system that integrates theory formation, experiment design, behavioral data collection, and revision.
• It demonstrates robust recovery of canonical and non-canonical psychological models, achieving low mean squared error under varied noise conditions.
• The system outperforms traditional models in human data experiments, enabling scalable, auditable closed-loop discovery and innovative theory revision.

AutoCog: Autonomous Closed-Loop Discovery of Psychological Theories

Introduction and Motivation

The paper "Closing the Loop to Discover Psychological Theories with an Automated Cognitive Scientist" (2606.26448) introduces AutoCog, a fully autonomous agentic-AI framework for psychological theory discovery. Unlike prior systems constrained to automating data collection, hypothesis testing, or model fitting, AutoCog operationalizes closed-loop scientific discovery in cognitive psychology—integrating theory formation, experiment design, behavioral data collection, theory comparison, and revision. The system's automation extends to the generative, creative aspects of theory-building, conventionally considered resistant to mechanization in empirical disciplines such as psychology.

The impetus for the approach is the persistent bottleneck in manual theory-building due to open-ended conceptual search required for human cognition. While substantial progress has been realized in closed-loop discovery in domains such as chemistry, materials science, and mathematics, closing the loop in cognitive science demands mechanisms that both explore and meaningfully update mechanistic theories in response to empirical falsification—beyond fitting candidate models to pre-assembled datasets.

System Architecture and Methodology

AutoCog operates as a cyclic closed-loop system (Figure 1), in which each execution involves:

• Experimental Design (Stage 1): LLM agents representing competing theories (encoded as executable models) propose experiments that are maximally discriminative. This follows an adversarial design principle—each agent constructs an experiment likely to empirically discredit its competitor. Proposed metrics for quantifying performance are generated alongside the experimental designs.
• Behavioral Data Collection (Stage 2): Automated administration of the constructed experiments, yielding behavioral datasets from human participants, cognitive models, or foundation models.
• Analysis and Arbitration (Stages 3–4): Theories are scored via forward simulation, not fitted post hoc, to evaluate generative adequacy against empirical data according to pre-registered metrics. An LLM-based arbitrator interprets model–data discrepancies and issues structured verdicts about theory revision or replacement.
• Theory Revision (Stage 5): Based on arbitration, underperforming theories are either programmatically revised (via code synthesis) or replaced at both the description (natural language) and implementation levels. Verification via simulation guarantees behavioral improvement before replacement.

  Figure 1: Schematic of the AutoCog discovery loop, showing end-to-end automation from experiment design to theory revision across N cycles.

This agentic, multi-agent architecture emphasizes open-ended search over both experimental and model spaces. It is designed for generalizability, with theory and experiment modules specified through typed interface contracts that abstract away from the specifics of the decision task.

Ground-Truth Recovery: Synthetic Case Study

To validate the discovery dynamics, AutoCog was tested on synthetic data generated from known canonical and non-canonical cognitive strategies (e.g., Take-the-Best [TTB], Tallying, Weighted Additive [WADD], Alternating, Perseveration, Random). The evaluation criterion was the mean squared error (MSE) between the discovered and the ground-truth model's probabilistic choice profile, measured on held-out diagnostic tasks.

Key findings are:

• Exact Recovery in the Noise-Free Regime: For noiseless simulated datasets, the best surfaced theories achieved $MSE_{\hat{p}(B)}=0.0025,\, SEM=0.0011$—statistically indistinguishable from the generating mechanism, despite initialization with confounding seed models.
• Robustness to Noise: As action noise increased to $\epsilon=0.5$ and $\epsilon=0.75$, recovery performance gracefully degraded but remained substantially better than random or seed theories.
• Diverse Mechanism Recovery: Non-canonical strategies were successfully surfaced (e.g., Alternating $M=0.003$, Perseveration $M=0.005$). For more challenging spurious strategies (e.g., single-cue or anti-majority), extended cycles were required to transcend the LLM's priors and converge on the generating rule.

  Figure 2: Quantitative recovery of generating theories under increasing action noise, for canonical and non-canonical strategies.

A LLM-as-judge rubric confirmed high mechanism similarity between discovered and generating models for low-noise cases, and a principled degradation of mechanism similarity with increasing behavioral stochasticity.

Human Data: Discovery of Decision-Making Theories

The system was extended to behavioral data from human participants in a two-alternative multi-attribute decision-making task, configured in both binary and cardinal cue spaces. The key result is the progressive surfacing of population-invariant, task-invariant cognitive models that outperform canonical seeded theories, generalize to unseen task designs, and yield novel psychological hypotheses.

• Trajectory of Model Improvement: Across five closed-loop cycles, $MSE_{\hat{p}(B)}$ for the winning model decreased from $0.093$ down to $0.0097$ (SEM$=0.0018$), an order-of-magnitude reduction.
• Cumulative Theory Generalization: The winning model, a Non-linear Subjective Weighting Model, unified classical heuristics as special cases by positing a free exponent on subjective weighting of cue validities—a representational innovation not enforced by explicit parsimony or diversity constraints.
• External Generalization: When challenged on diagnostic stimuli from Hilbig et al. (2014), the surfaced model achieved $MSE_{\hat{p}(B)}=0.0377$ (SEM$\epsilon=0.5$0), outperforming TTB, Tallying, and linear WADD.

  Figure 3: Discovery trajectory with theory lineage, model code, and fit metrics showing the stabilization on a non-linear subjective weighting model over cycles.

Extension to Complex Decision Spaces and Prospective Validation

Expanding to cardinal-valued cues permitted discrimination among finer-grained utility transformations. The system autonomously discovered a Diminishing Returns WADD model, hypothesizing concave transformation of feature utilities—a curvature not predicted by seeded heuristics.

Remarkably, the theory was prospectively validated via preregistered human studies that tested three nontrivial predictions:

• Model Discrimination: The Diminishing Returns WADD model produced match rates with human choices significantly higher than competitors (e.g., difference to TTB: $\epsilon=0.5$1, $\epsilon=0.5$2).
• Steep-vs-Flat Region Effects: Participants preferred low-range advantages (steep regions of the utility curve) over high-range (flat regions), consistent with concave transformation.
• Range-Shift Effects: Preferences for a given cue structure were attenuated when shifted to higher absolute value regions, a non-linear effect captured by the model.

The distinctive predictions held in preregistered, prospectively collected data, underscoring the system's capacity for genuine theoretical innovation, not mere refitting.

Figure 4: Closed-loop discovery and external prospective validation of the Diminishing Returns WADD model in a cardinal-cue decision task.

Process Visualization and Systemic Auditability

The modular, agent-native architecture supports fine-grained process auditability and reproducible automation. Additional figures depict the explicit computational pipeline, including adversarial design, data collection, arbitration, and revision:

Figure 5: Adversarial experimental design with LLM-driven experiment proposals and verification.

Figure 6: Automated collection of behavioral data from humans or simulators, employing the same computational pipeline.

Figure 7: Forward simulation-based theory analysis and neutral arbitration, scoring all theories on all datasets.

Figure 8: Theory revision loop with program synthesis and aggregate-fit validation.

Practical and Theoretical Implications

AutoCog provides a generalized, extensible paradigm for fully autonomous theory discovery—turning computational/theoretical psychology into an explicit, executable, and cumulative enterprise. The integration of agentic proposal, self-verification, and accumulation of behavioral constraints induces pressure for generalizability and explanatory adequacy, rather than mere post hoc fit.

Practically, this architecture can support both in silico-theory preselection (with behavioral foundation models or cognitive simulators) and direct human-data-driven cumulative theory search. The open-ended, logged process enables human researchers to continuously audit, intervene, or modify discovery trajectories—shifting the creative role toward the specification of desiderata, evaluation criteria, and design/bias constraints.

From an AI perspective, AutoCog demonstrates the viability of fully agentic, program-synthesizing, multi-agent scientific discovery frameworks. This approach is not only domain-agnostic within psychology but, with appropriate representational scaffolding, extensible to any empirical science amenable to closed-loop experiment–theory iteration. The implications for AI-driven autonomous science are far-reaching.

Future work may target explicit diversity objectives, alternative discovery paradigms beyond adversarial contest, richer human-in-the-loop oversight, and generalization to higher multiplicity or hierarchical theory spaces.

Conclusion

AutoCog realizes a closed-loop discovery system in cognitive science, automating not only modeling and experiment design but the generative process of theory-building from empirical behavioral evidence. It achieves strong empirical results in recovering and outperforming canonical psychological models, surfaces novel theories that withstand preregistered prospective testing, and establishes a foundation for scalable, auditable scientific automation in psychology and related fields (2606.26448).