Auto: The AGI Compiler

Abstract: Every LLM agent run re-derives its behavior token by token on a frontier model: brilliant, expensive, slow, and unbounded. We present Auto, a compiler that records live agent behavior, measures which parts are secretly deterministic, extracts them into verified programs or distilled specialists, and emits cognition binaries: WebAssembly artifacts whose manifests carry measured guarantees and whose declared capabilities are physically enforced by the sandbox. A tiered runtime executes compiled behavior behind conformally calibrated guards; guard trips deopt to the reference agent, and the captured trace recompiles back down, so nothing is figured out twice. We use "AGI compiler" in one narrow, testable sense: a system that autonomously converts novel experience into permanent, verified, near-free skill while measuring what it does not know. On AUTO-BENCH, a benchmark we introduce and pre-register, 87.1% of 560 recorded frontier-agent spans are witnessed-deterministic (three of the four censused task families measure 100.0%). On a 300-item stream with three scheduled distribution shifts, the closed loop compiles three artifact generations and drives marginal cost from 59 to 2 micro-dollars per item (6.4x end-to-end) at 96.9% parity on witnessed inputs with zero errors. The same stream also quantifies the failure modes: a loose guard silently mislabels 48.9% of compiled answers, and an unfaithful deopt reference causes the verification gate to refuse recompilation. Calibration and reference fidelity, not model capability, decide whether cheap stays correct. Code: https://github.com/RightNow-AI/auto

• The paper introduces Auto, an AGI compiler that proceduralizes agent cognition by transforming live behavior into persistent, verifiable binaries.
• It outlines a multi-stage process including deterministic tracing, IR synthesis, distillation, and contract verification to ensure reliable artifact generation.
• Empirical results show significant cost reductions, high determinism levels, and robust guard calibration for secure, scalable agent deployment.

Auto: The AGI Compiler—Formal Overview

Motivation and Context

The execution of LLM-based agents remains fundamentally inefficient: every inference run re-derives token-level decisions from scratch, incurring substantial latency, cost, and variability, even on tasks exhibiting stable deterministic behavior. This paradigm mirrors interpreted code, forgoing the economic and reproducibility gains historically afforded by compilation in legacy software systems. Existing approaches—including semantic caching, model routing, and knowledge distillation—only reduce marginal costs and do not yield persistent, verifiable artifacts. The "AGI compiler" thesis posited here is precise: a requisite property of general intelligence is proceduralization, i.e., the transformation of experience into permanent, verified skill. "Auto" instantiates this process, compiling agent cognition into reproducible binaries with guarantees and enforced capability boundaries.

System Architecture and Methodology

Auto is architected as a classical compiler and runtime toolchain capable of transforming live agent behavior into confined binaries:

• Recording: All spans of agent execution are traced, capturing prompts, tool acts, branches, and I/O with cost and latency tracking.
• Determinism Census: Spans are labeled as witnessed-deterministic if multiple executions yield identical, error-free outputs. This pre-compilation measurement is non-extrapolative—compilation is only attempted on observed spans.
• Ir Construction and Synthesis: Traced behavior is lowered into a typed task graph IR, encapsulating capability effects, uncertainty classes, and resource limits. Symbolic behavior is extracted via enumerative search over a DSL; for remaining spans, LLM-guided CEGIS proposes candidate programs validated against recorded inputs, with counterexample generation ensuring proposal soundness.
• Distillation: Residual non-symbolic spans are distilled into decision trees or MLPs arranged per witnessed outputs, a process leveraging weighted witness counts.
• Contract Verification: The behavioral contract, articulated as input-output pairs, thresholds, budgets, and judged properties, becomes the type system for artifact admission. Differential replay, optionally augmented by LLM-based semantic equivalence for generative outputs, forms the verification gate. Inconclusive or failing checks strictly preclude artifact emission.
• Artifact Generation and Confinement: Verified cognition binaries are emitted as WebAssembly modules, physically confined by declared capability imports. Zero-imports enforce purity, and tool-using artifacts restrict host access to precisely auditable functions.
• Tiered Runtime: Execution is governed by conformally calibrated guards—typically lexical sketches calibrating abstention thresholds, with designated miscoverage probabilities ($\alpha$). In-distribution requests are served by compiled artifacts (Tier-1, microsecond latency and zero marginal cost), others deopt to the reference agent (Tier-0), triggering trace capture and recompilation (the ratchet).

Auto-Bench Benchmark Protocol and Metrics

Auto-Bench is introduced as a protocol measuring compilation feasibility rather than model capability. Four headline metrics are defined:

• H1: Ratchet Curve — Cost per task as a function of cumulative experience; tracks amortized marginal cost reduction.
• H2: Determinism Census — The fraction of behavior spans that are witnessed-deterministic, measured prior to compilation.
• H3: Parity-Gated Compression — Cost and latency reductions achieved by compiled artifacts at specified parity thresholds, including explicit refusals for spans failing the contract.
• H4: Calibrated Ignorance — The fidelity of guard abstention mechanisms in rejecting out-of-distribution queries without silent error.

The protocol is pre-registered, reproducible, and attaches spend ledgers and evaluation run IDs to every outcome. Diverse task families—including ticket triage, multi-stage pipelines with tools, information extraction, policy routing, summarization, and distribution-shifting streams—are benchmarked.

Core Empirical Results

Determinism Census

The witnessed-deterministic fraction is pivotal: 87.1% of agent spans across four task families reach determinism levels enabling compilation (three families achieve 100% determinism). This empirical census directly validates the thesis; significant agent behaviors are structurally amenable to compilation.

Compression and Verification Outcomes

Compiled artifacts deliver three-orders-of-magnitude cost reductions (e.g., reducing per-item cost from 59 to 2 micro-dollars) and microsecond-latency execution at 96.9% parity with reference agents on witnessed data, with zero errors. The verification gate's refusal patterns are nuanced: deterministic extraction (JSON output) rebuffed due to the current lack of structural synthesis primitives, and generative summarization remains non-compilable at scale, failing contract thresholds despite partial determinism.

The Ratchet Curve

On a 300-item novelty stream, Auto triggers three artifact generations synchronized to distribution shifts, maintaining economic advantage (6.4× reduction in cumulative spend) and verified parity. The ratchet loop ensures that novel experience is captured, compiled, and amortized, avoiding repeated payment for decisions previously computed and verified.

Failure Modes and Calibration

Failure analysis demonstrates that loose guard calibration can admit post-shift novelty, resulting in silent mislabeling (48.9% compiled answers incorrect in one configuration), while generic model references as deopt stall recompilation at the gate due to contract vocabulary mismatches. Only agent-faithful references close the loop fully. Guard exposure is measured precisely: lexical calibration admits cousins of witnessed data, with $\alpha$ quantifying exposure. Semantic embedding calibration is earmarked as the prospective upgrade for future versions.

Calibrated Ignorance

Guard calibration achieves perfect rejection for out-of-distribution probes in policy routing tasks, with lexical noise detected as the primary calibration floor elsewhere. Artifact generalization is strictly controlled: only witnessed distributions, unless holdout tests show generalization, are memorizable.

Theoretical and Practical Implications

Auto operationalizes proceduralization for LLM agent cognition by providing a formal methodology and measurable infrastructure for turning live behavior into persistent, contract-verified, confined binaries. The measured high fraction of deterministic spans suggests that a significant portion of agent workloads—classification, extraction, routing—are structurally suitable for compilation, promising substantial cost savings and consistency in deployment. The ratchet loop (autonomous recompilation) enforces continual adaptation, averting staleness amid distribution changes.

Verification as the type system ensures fidelity and defensibility of artifacts, mitigating typical risks in naive distillation or caching approaches. Physical capability confinement in WebAssembly artifacts enforces declared boundaries, relevant to operational safety.

Auto's limitations are transparently documented: designed corpora, initial compile primitives, lexical calibration, and judge dependency. These highlight compelling directions for future research:

• Synthesis advances for structured outputs and generative spans
• Semantic guard mechanisms to minimize calibration exposure
• Robustification of judge verification pipelines
• Re-running Auto-Bench on real-world traffic for broader empirical validation

The architecture's modularity and reproducibility (open source) facilitate further integration in agent deployment stacks.

Conclusion

Auto advances agent cognition compilation from a theoretical desideratum to an empirically substantiated system. It rigorously quantifies the compilable fraction of agent tasks, delivers persistent binaries at marginal cost, and formalizes the ratchet loop as the mechanism for permanent skill acquisition. The system is rigorously benchmarked, with detailed analyses of both operational gains and failure modes. Auto positions compilation as a central infrastructural element for scalable, cost-effective, and reproducible agent deployment, laying foundational groundwork for future proceduralization mechanisms in AGI architectures.