Sibyl-AutoResearch Framework
- Sibyl-AutoResearch is an autonomous research framework that converts trial outcomes into research judgment by linking trial signals with subsequent behavior changes.
- It employs scientific trial-and-error harnesses to ensure evidence maturity, traceability, and auditable state transitions in research workflows.
- The SIBYL implementation uses a file-backed state machine with role-specific memory routing to evolve research practices and repair recurring process failures.
Sibyl-AutoResearch is a framework and concrete system, SIBYL, for autonomous research centered on Scientific Trial-and-Error Harnesses rather than on paper generation alone. It treats autonomous research as a process in which bounded trials, negative evidence, evidence-maturity states, role-specific memory routing, and harness self-repair must remain inspectable as concrete state transitions. In this formulation, the decisive question is not merely whether agents can read papers, run code, and draft prose, but whether trial outcomes can be shown to change later behavior and whether recurring process failures can be shown to change the harness itself (Wang et al., 21 May 2026).
1. Motivation and problem formulation
Sibyl-AutoResearch is motivated by the claim that executable workflows do not automatically produce research judgment. The framework defines research judgment as experience-backed behavior: over time, the system should form better priors, distrust fragile metrics, downgrade weak claims, and avoid repeating the same infrastructure mistakes. Its diagnosis of existing autonomous research systems is organized around six failure modes: paper completion hides evidence immaturity, pilot collapse, bad objective, unrouted memory, trial-volume drift, and static harness (Wang et al., 21 May 2026).
These failure modes are operational rather than rhetorical. In this view, a system can finish a paper while metrics are broken, baselines are missing, or numbers are stale; pilot signals can be narrated as broad conclusions without an explicit notion of evidence maturity; metric-driven loops can optimize whatever is visible even after the metric is discovered to be brittle or mis-defined; and “lessons learned” can remain in reflections without altering plans, gates, or role behavior. Sibyl-AutoResearch responds by making update paths from trials to later behavior explicit and auditable (Wang et al., 21 May 2026).
This emphasis differs from broader Auto Research visions that organize the full lifecycle into literature review, ideation, methodology planning, experimentation, writing, review, rebuttal, and dissemination. That broader line of work treats research as a modular, interpretable, and optimizable workflow, whereas Sibyl-AutoResearch argues that workflow execution alone is insufficient unless trial outcomes are converted into later planning, validation, claim scope, scheduling, critique, writing, and harness repair (Liu et al., 26 Apr 2025).
2. Scientific Trial-and-Error Harnesses and evidence maturity
A Scientific Trial-and-Error Harness is the research environment around the agent. It includes state and roles, tools and compute, memory and reflection, gates and decisions, artifact contracts and traceability, resource policy, and repair/self-evolution mechanisms. Each trial is explicitly bounded by scope, resources, iteration cap / stop conditions, and artifact requirements (Wang et al., 21 May 2026).
The framework specifies seven harness functions, labeled H1–H7.
| Function | Requirement |
|---|---|
| H1 Trial orchestration | Each trial has a question, expected evidence, dependencies, outputs, and stop conditions |
| H2 Evidence maturity | Distinguish execution completion, pilot signal, analysis-ready evidence, paper-ready evidence, and audited claim |
| H3 Traceability | Tie behavior updates to artifacts such as configs, logs, tables, reviews, and writing changes |
| H4 Routed research memory | Route lessons to planner checks, experimenter sanity checks, critic objections, supervisor gates, and writer restrictions |
| H5 Perspective separation | Ensure role-specific authority so objections become tasks, plan mutations, stopped branches, or claim downgrades |
| H6 Resource-aware trial policy | Use GPU, tokens, and attention budgets to change sanity-check ordering, allocation, and monitoring |
| H7 Harness self-evolution with protected constraints | Turn recurring process failures into new gates, overlays, telemetry requirements, repair tasks, and artifact contracts |
A second foundational construct is the evidence maturity ladder. Sibyl-AutoResearch distinguishes Execution completion, Pilot signal, Analysis-ready evidence, Paper-ready evidence, and Audited claim. Claims can advance only after validation and artifact links, and negative results can increase maturity by debunking stories. This structure is intended to separate “code ran” from “claim is auditable” and to block the collapse of pilots into general conclusions (Wang et al., 21 May 2026).
3. SIBYL implementation and file-backed architecture
SIBYL is the concrete implementation of the framework. It is a file-backed autonomous research system in which plans, task definitions, role outputs, validation decisions, reflections, evolution records, experiment artifacts, and writing outputs are represented as files rather than hidden in transient conversational state. The architecture includes an artifact-backed state machine, decision gates and quality gates, evolution memory, and a validated claim registry (Wang et al., 21 May 2026).
The role structure is explicit. The system uses planner, experimenter, critic, supervisor, skeptic, methodologist, writer, and editor roles. The planner reads previous state, evolution overlays, and evidence maturity and emits a task plan. The experimenter runs code and produces logs, metrics, tables, and figures. Validation agents, typically including supervisor and critic, decide whether outputs warrant refine, pivot, block, or scoped advancement. The writer consumes the validated claim registry rather than raw logs, so pilots cannot be promoted into main claims by prose alone (Wang et al., 21 May 2026).
Memory is layered. Reflection artifacts are normalized into structured issue records with categories such as system, experiment, writing, analysis, planning, pipeline, ideation, and efficiency. These records are aggregated into evolution records and then routed into role-specific overlays. The current corpus is described as containing 416 patterns, including 212 experiment, 89 writing, and 84 analysis patterns. The framework judges memory by behavior change rather than by length: a lesson is effective only if it reduces repeated failures, changes plans, or modifies gates (Wang et al., 21 May 2026).
4. Conversion units: from trial signals to behavior and harness updates
Sibyl-AutoResearch formalizes two auditable conversion units.
The first is trial-to-behavior conversion. A trial at iteration produces a signal, and the framework requires an observable trace path to a later behavior update at iteration . The paper describes this conceptually as:
The behavior change may be a changed plan or branch priority, a new validation task, a narrowed or downgraded claim, a scheduling change such as sanity checks before expensive runs, or altered critique and writing behavior. A valid conversion event therefore requires three artifacts: a specific trial signal, a trace path through memory, gates, or planning, and a downstream behavior change (Wang et al., 21 May 2026).
The second is trial-to-harness-behavior conversion. Here the signal is a recurring process failure, such as duplicate result files, stale tables, unsupported statistics, missing telemetry, or feature-count mismatches. The framework describes this as:
The harness update may be a new gate, a prompt overlay, a telemetry requirement, a scheduler policy change, a protected constraint, or a repair task. Together, the two conversion units define agent–harness co-evolution: agents change how they research, and the harness changes how research is allowed to proceed (Wang et al., 21 May 2026).
This conversion-centric formulation implies a different unit of evaluation from paper-centric systems. A plausible implication is that autonomous research is assessed less by whether a polished artifact exists and more by whether later conduct can be causally linked to earlier evidence and failures.
5. Retrospective audits, conversion events, and recovered failures
The framework is supported by a retrospective audit over real SIBYL workspaces rather than by a comparative benchmark claim. The audit identifies 8 high-confidence conversion events, with median latency of one iteration and maximum latency of three iterations. These traces are presented as evidence that the proposed conversion units are recoverable from realistic autonomous-research workspaces, not as proof that the system outperforms alternative frameworks (Wang et al., 21 May 2026).
The audited workspaces include projects on sparse autoencoder absorption, failed sparse autoencoder replication, dynamic weight decay, diffusion-LM acceleration, image augmentations, and a diffusion-LM caching pilot. In the dynamic weight-decay case, trial signals about controller instability led to controller repair, stability tests, and a refined three-tier claim taxonomy. In diffusion-LM acceleration, unsupported statistics and mismatched accept-rate claims caused the speedup story to be reframed as an interference taxonomy, pairwise speedup multiplication claims to be dropped, and stronger full-scale gates to be added. In sparse autoencoder absorption, source-to-paper validation scripts, duplicate detection, and feature-count verification became prerequisites before writing could proceed (Wang et al., 21 May 2026).
The paper also reports a recovered-failure registry with five naturally occurring failure classes. These include duplicate result files, CI inversion, stale headline numbers, feature-count mismatch, and unsupported statistics. In the cited examples, byte-identical replicates led supervisors and critics to downgrade evidence and to introduce duplicate detection and single-source analysis as prerequisites; confidence intervals that did not contain their point estimates caused a source-to-paper validation script to be added; a stale 4.1× headline ratio was reduced to 2.7× after validation; a run using 1,024 features while the paper claimed 16,384 triggered feature-count verification; and fabricated Wilcoxon values together with inconsistent accept rates caused claims to be downgraded and the paper to be reframed (Wang et al., 21 May 2026).
A separate aggregate review-to-action audit examined 51 project-iteration snapshots for generated reviews and 39 snapshots for internal supervisor reviews. For rows in which the internal supervisor score decreased by at least 0.25, the next plan was dominated by experiment/control tasks and validation/harness changes. The reported interpretation is that the system reacts to serious criticism by doing more science rather than merely rewriting (Wang et al., 21 May 2026).
6. Relation to broader AutoResearch systems and scope limits
Sibyl-AutoResearch occupies a specific position within the wider AutoResearch literature. It does not reject full-lifecycle automation, but it argues that modular execution of literature review, ideation, experimentation, writing, review, and dissemination does not by itself yield research judgment. Related systems instead illustrate complementary emphases: full-lifecycle multi-agent orchestration (Liu et al., 26 Apr 2025), self-evolving memory via failure logs and guarded updates (Liu et al., 13 May 2026), autoresearch-guided multimodal memory discovery (Liu et al., 1 Apr 2026), perpetual RL-based code-editing for architecture discovery (Jain et al., 7 Mar 2026), two-level pipeline redesign for sequential social dilemmas (Gallego, 28 May 2026), auditable LLM-guided optimization for aerospace control (Jain et al., 18 Jun 2026), and agentic descriptor design in materials science (Cobelli et al., 14 May 2026).
Within that landscape, Sibyl-AutoResearch is distinguished by its insistence on evidence maturity, claim-evidence boundaries, routed memory, and harness self-evolution with protected constraints. Systems such as AutoResearch-RL or aerospace AutoResearch show how a frozen environment, a mutable target file, and a credibility layer can support auditable search over code or hyperparameters; systems such as EvolveMem and Omni-SimpleMem show how failure logs and iterative experimentation can reconfigure memory architectures; and two-level autoresearch systems show that an outer researcher can redesign an inner pipeline. Sibyl-AutoResearch generalizes these themes into an explicit theory of how trial signals and recurring failures must be converted into later behavior and harness changes (Wang et al., 21 May 2026).
The framework’s own scope limits are stated explicitly. It makes no comparative performance claim; the evidence is retrospective and comes from a single system co-developed with the framework; the recovered-failure registry uses naturally occurring failures rather than injected ones; generated reviews are not authoritative; and the current evidence is concentrated in computational AI/ML projects, not in wet-lab or robotic domains. Future evaluation protocols proposed in the source include prospective fixed-budget comparisons of different harness designs, injected failure stress tests, cross-project memory tests, harness-evolution tests, and ablations of roles and validators (Wang et al., 21 May 2026).
Sibyl-AutoResearch therefore marks a shift from artifact-centric autonomous research to conversion-centric autonomous research. In this formulation, the decisive empirical object is not simply a finished paper or a successful run, but an auditable trail showing that trial history becomes research judgment and that the harness itself changes in response to its own failures (Wang et al., 21 May 2026).