Science sandboxes measure the scientific capability of AI agents

Abstract: Scientific progress depends not only on finding solutions, but on learning the rules that explain why they work and using that understanding to design better experiments. We introduce science sandboxes, a framework for studying this capability in AI agents through repeated cycles of experimentation, feedback, and hypothesis revision. Science sandboxes invite an agent to query the natural world in different ways, ranging from "wet" physical experiments, to "damp" predictive models trained on empirical data, to "dry" invented rules. By establishing a common experimental loop and a protocol for evaluating agents within it, science sandboxes allow assessment of both quantitative performance on specific metrics and qualitative scientific reasoning, across a spectrum of empirical verifiability. Here, we instantiate this framework in two biological settings, models of regulatory genomics and protein fitness prediction, and examine the capabilities of frontier agents. Across these settings, we could see when agents successfully optimized a quantitative metric without understanding the rules underlying the system. In particular, their scientific reasoning deteriorated when they encountered systems whose rules fell outside familiar biological priors. By highlighting such failure modes, science sandboxes make the frontier of scientific capability measurable and provide a controlled setting in which to study and ultimately expand it.

• The paper introduces a novel evaluation framework called **science sandboxes**, which formalizes scientific investigation as a closed loop involving agents selecting informative experiments, interpreting observations, and updating hypotheses.
• The **MPRAbox** implementation demonstrated the evaluation of AI agents in a library-selection problem, showing that AI models like Claude Opus 4.7 may achieve high performance without necessarily inferring the correct underlying rules for regulating DNA sequence design.
• The findings underscore the importance of evaluating not only the performance metrics but also the qualitative reasoning and capacity for hypothesis revision in assessing the scientific capability of AI agents.

Scientific capability as a closed-loop property

“Science sandboxes measure the scientific capability of AI agents” (2608.30165) proposes an evaluation framework designed to distinguish scientific reasoning from outcome optimization. The central premise is that scientific capability cannot be adequately measured by whether an agent produces a high-scoring artifact or a correct final answer. Scientific investigation requires an agent to select informative experiments, interpret observations, formulate explanatory hypotheses, revise those hypotheses when contradicted, and use the resulting theory to design subsequent experiments.

A science sandbox formalizes this process as a closed loop. The agent submits specimens to a sealed oracle, receives an experimentally derived report, records its interpretation, and selects the next experiment. The specimens may be DNA sequences, proteins, chemicals, cells, engineered constructs, or abstract strings. The oracle may be wet, using physical experiments; damp, using a predictive model trained on empirical data; or dry, applying an invented rule with exact ground truth known to the sandbox designer. This abstraction preserves the sequential structure of empirical inquiry while permitting controlled variation in realism, scalability, and verifiability.

The framework evaluates two separable outputs. The first is quantitative performance: whether the agent improves a prespecified metric. The second is qualitative scientific reasoning: whether the agent identifies explanatory regularities, designs discriminative experiments, and updates its theory appropriately. The distinction is essential because optimization can succeed through correlations, local search, or exploitation of familiar priors without producing a correct account of the system being optimized.

MPRAbox and the experimental-design objective

The first implementation, MPRAbox, casts regulatory sequence design as a library-selection problem. In each round, an agent must submit 50,000 distinct 200-base-pair DNA sequences. These sequences are evaluated by a sealed in silico MPRA oracle based on Malinois, a convolutional neural network trained on 776,474 experimental measurements from three cell types. The oracle generates activity labels for the submitted library. A fresh sequence-to-activity model is then trained from random initialization on those 50,000 sequence-label pairs and evaluated against 14 hidden test sets.

The objective is therefore not to identify individual high-activity enhancers. It is to select a library that yields a model with broad generalization across heterogeneous sequence distributions. The evaluation suite contains five empirically labeled sets and nine Malinois-labeled sets, covering held-out MPRA sequences, UK Biobank and GTEx variants, DNase hypersensitive sites, Sei regulatory classes, random genomic windows, and synthetic DNA. The aggregate score is the mean Pearson correlation across these 14 sets.

This formulation makes library composition itself the experimental variable. Genomic sequences, synthetic sequences, regulatory-class stratification, sequence diversity, activity calibration, and negative controls can all be treated as competing hypotheses about what constitutes informative training data. Because the agent receives only aggregate scores, it cannot directly inspect sequence-level labels or determine which evaluation distributions drove a result.

The human reference strategies exhibited substantial variation in performance. Libraries composed primarily of genomic regulatory sequences generally outperformed fully synthetic libraries at the 50,000-sequence budget, although the paper also notes that synthetic sequences can be useful for controlled perturbation of motifs and sequence grammar. This result is specific to the Malinois-based downstream-training objective and should not be interpreted as a general finding that genomic libraries are intrinsically superior for all regulatory sequence-design tasks.

One-shot frontier-agent performance

The authors evaluated Claude Opus 4.7, GPT-5.5, and Gemini 3.5 Flash in five independent one-shot replicates, with and without prior knowledge of the human-designed strategies. In the blind condition, Claude performed best:

Agent	Median Pearson $r$, no prior knowledge	Median Pearson $r$, prior knowledge
Claude Opus 4.7	0.774	0.781
Gemini 3.5 Flash	0.680	0.751
GPT-5.5	0.655	0.760
Best human strategy	0.763 mean	0.763 mean

Claude’s five blind libraries each matched or exceeded the mean performance of the strongest human-selected strategy, whose mean score was $r = 0.763$. GPT and Gemini did not exceed that reference in the blind condition. Their lower performance was associated with a preference for synthetic sequence designs based on controlled motif variation, spacing, and frequency. Claude instead selected naturally occurring regulatory DNA, emphasizing broad coverage and negative examples for calibration.

The agents’ written rationales reveal that performance was partly mediated by prior biological knowledge. Claude reasoned that natural regulatory sequences preserve genomic context and that a library containing only “interesting” elements would bias the model toward predicting activity universally. GPT argued for synthetic sequences because they permit causal variation of motifs against controlled backgrounds. That reasoning is scientifically plausible, but it was mismatched to the particular oracle and evaluation distribution. The result illustrates a central point of the paper: a coherent scientific rationale can still be empirically incorrect, and benchmark performance depends on the relationship between the rationale and the hidden data-generating process.

Providing the agents with descriptions and scores for the 14 human strategies improved all three systems, but especially GPT and Gemini. GPT rose from $r = 0.655$ to $r = 0.760$, and Gemini from $r = 0.680$ to $r = 0.751$, whereas Claude improved only from $r = 0.774$ to $r = 0.781$. The agents did not simply reproduce the best human strategy. Their notebooks indicate that they treated the reference results as evidence about mechanisms and proposed modifications, including mixtures of genomic sequences, synthetic controls, topic stratification, and rare-class enrichment.

This condition demonstrates both transfer and dependence on prior information. Agents can use compressed experimental evidence to improve design, but their initial hypotheses are strongly shaped by the semantic framing of the task and by knowledge acquired during pretraining. Their success in MPRAbox therefore does not by itself establish de novo discovery of regulatory principles.

Iterative experimentation and hypothesis revision

The long-horizon evaluation gave Claude 30 sequential design rounds, with two runs conducted without and two with prior knowledge of the human strategies. All four trajectories eventually exceeded the strongest human-selected reference strategy, although the quantitative gains over one-shot performance were modest. The authors intentionally emphasize that the primary value of the long-horizon condition is not additional hill climbing but the observation of experimental reasoning over time.

The trajectories displayed several behaviors characteristic of scientific investigation. Agents initially compared broad classes of libraries, including random DNA, genomic regulatory sequences, synthetic sequences, and cross-species sequences. They then increasingly adopted controlled comparisons that varied one design feature at a time. In one run, the agent recognized that a comparison involving chicken sequences had simultaneously changed the human regulatory component, making the result uninterpretable. It subsequently articulated the experimental-design principle that causal claims require isolation of the relevant variable.

Agents also revised hypotheses in response to unexpected observations. One run produced a surprisingly high score for random DNA, contradicting the agent’s expectation that a library with no biological structure would perform poorly. The agent revised its theory, proposing that part of the evaluation signal was driven by sequence composition and that the library needed to teach regulatory grammar beyond this “free” compositional signal. Whether that revised interpretation was fully correct is less important than the fact that it was generated in response to an anomalous result and used to motivate subsequent experiments.

The four trajectories converged on different design principles. One favored naturally occurring regulatory contexts over motif-injected sequences. Another found that rare regulatory classes could improve performance when enriched without sacrificing broad coverage. A third reported gains from adding chicken regulatory sequences but not more distant species. A fourth found that GC-content stratification improved a strong DHS-based design. The lack of convergence is consistent with a complex design landscape, but it also makes it difficult to determine whether the agents discovered stable causal principles or exploited trajectory-specific correlations.

Dry oracles and the separation of optimization from understanding

The dry-oracle experiments provide the paper’s strongest test of de novo rule discovery. The authors replaced the Malinois oracle with 14 hidden functions involving nucleotide composition, positional dependence, substring occurrence, compressibility, parity, prime counts, Fibonacci-indexed positions, modular arithmetic, a cellular automaton, and a hidden cipher mapping nucleotide pairs to English letters.

The surrounding task remained similar: the agent submitted libraries, a model was trained using oracle-generated labels, and the agent received only an aggregate generalization score. Three framings were compared. The agent was told either that it was designing regulatory DNA, that it was optimizing an unspecified black-box function over the DNA alphabet, or that it was operating over the symbolic alphabet $\{0,1,2,3\}$.

Framing materially changed exploration. The MPRA framing elicited hypotheses about regulatory motifs and biological sequence structure. The unframed DNA condition shifted attention toward composition and short patterns. The symbolic condition encouraged simpler frequency-based analyses. In some cases, these priors provided useful proxies for the hidden rule and improved quantitative performance. However, they did not reliably result in rule identification.

The hidden English-word cipher is particularly diagnostic: none of the framings led the agent to hypothesize that pairs of symbols encoded letters or that the decoded sequence contained natural-language words. In the Fibonacci-position condition, the agent developed a composition-based explanation that predicted some score variation but was not the actual generating rule. Thus, the agent could exploit a correlated feature while maintaining a false causal theory.

This is the paper’s most important conceptual result. Improving a quantitative metric is not equivalent to inferring the mechanism that produces the metric. A benchmark that reports only final performance would classify these trajectories as successful, despite their failure to recover the governing rule. The dry oracles make this distinction measurable because the ground-truth rule is known to the evaluator.

CodonBox and structured induction

CodonBox extends the analysis to an invented biological system. Each input is a nucleotide-like sequence that is parsed into hidden codons, translated into a 16-residue hydrophobic/polar chain, and folded using a two-dimensional lattice model. Fitness is the number of favorable nonconsecutive hydrophobic contacts in the optimal fold, with a maximum score of 9.

The agent receives only the final fitness score. It is not told that codons exist, what their length is, how codons map to residues, or that the translated chain undergoes lattice folding. Eight worlds vary alphabet size, codon length, silent positions, and additive versus interaction-based codon rules. Claude is given 500 one-sequence experiments in each world.

Quantitative optimization is intentionally easy. A repeated-character sequence produces repeated codons; if the repeated codon maps to hydrophobic residues, the resulting all-hydrophobic chain reaches the maximum fitness. The agent reached the maximum score within the first 10 rounds in every run. This result directly demonstrates why score maximization is an insufficient measure of scientific capability: the agent could attain the optimum while the translation and folding rules remained unidentified.

The agent correctly inferred codon length for tables containing 16 and 64 entries and also identified three-nucleotide codons in a larger alphabet condition with 512 possible codons. For a four-nucleotide codon system with 256 entries, it inferred the codon structure and learned many individual codons but failed to recover the complete table. With a six-symbol alphabet and three-nucleotide codons, it failed to discover the codon structure and instead constructed increasingly elaborate theories based on individual nucleotides and short patterns.

The most successful trajectories changed the experimental strategy rather than merely increasing the number of observations. In a world where only the middle codon position mattered, controlled substitutions of the outer positions revealed that they were silent. In a second world, the middle position was silent and the two outer positions jointly determined the output. The agent initially modeled nucleotide effects independently, but a contradiction forced it to test positional combinations and eventually recover the interaction rule.

The hardest world combined a six-symbol alphabet, four-character codons, a silent position, and a three-position interaction, yielding 1,296 possible codons. The agent eventually inferred four-character segmentation but did not identify the silent position or the general mapping. It reverted to cataloging and reusing individually successful codons. The result isolates a failure mode in experimental induction: when the hypothesis space becomes combinatorial, the agent may continue accumulating cases instead of searching for a lower-complexity structural explanation.

Limitations and open questions

The framework’s conclusions are constrained by the oracle choices and the experimental scale. MPRAbox uses Malinois both to label submitted libraries and to generate labels for much of the evaluation suite. Consequently, agents optimize a surrogate model whose inductive biases, training distribution, and architectural limitations shape the task. Five evaluation sets use empirical labels, but nine use Malinois labels, and the downstream model shares the oracle’s broad modeling assumptions even though its weights are independently initialized. The reported correlations therefore measure generalization within a partly model-defined environment, not regulatory-genomics competence in an unrestricted biological setting.

The human baselines are also sampling strategies rather than a controlled sample of expert experimental designs. Their performance depends on implementation choices, sequence sources, library size, and the fixed downstream architecture. The one-shot agent comparison is based on five replicates per condition, and the long-horizon analysis includes only Claude because of computational cost. It consequently cannot establish robust comparative claims about long-horizon performance across model families.

The qualitative evaluation of lab notebooks is potentially informative but underformalized in the presented material. Stated reasoning may not faithfully represent the causal basis of an agent’s actions, and a notebook can contain post hoc explanations for successful or unsuccessful experiments. The proposed use of independent AI judges could improve scalability, but automated assessment of whether a hypothesis genuinely explains an oracle remains itself a validation problem.

CodonBox and the dry MPRAbox rules provide exact ground truth and strong control over novelty, but their invented mechanisms may not represent the statistical structure, noise, confounding, and measurement constraints of real science. Conversely, wet oracles introduce cost, latency, irreproducibility, and experimental noise. The paper therefore leaves open how performance and rule-discovery criteria should be calibrated across the wet–damp–dry spectrum, and how much evidence is required before an agent’s explanation should count as scientifically adequate.

A further open question is whether the observed failures arise primarily from the underlying LLMs, the agent harnesses, the finite experimental budgets, or the scoring interface. The study uses provider-native coding environments and limits the long-horizon comparison to one model. It does not disentangle exploration policy, memory architecture, tool use, inference-time compute, and model capability. Nor does it determine whether explicit hypothesis representations, active-learning objectives, uncertainty estimates, or intervention-focused experiment selection would improve rule discovery rather than merely score optimization.

Conclusion

Science sandboxes provide a practical framework for evaluating AI agents through repeated cycles of experiment, feedback, and hypothesis revision. MPRAbox shows that frontier agents can achieve strong performance in a biologically familiar design problem, with Claude reaching a median $r$0 without prior strategy information and all four long-horizon runs exceeding the best human reference. Dry MPRAbox and CodonBox, however, show that high scores can coexist with incorrect or incomplete theories of the underlying system. The paper’s principal contribution is therefore methodological: scientific capability must be evaluated not only by what an agent finds, but by whether its experiments support the discovery of general rules that explain why the result occurs.