Cost-Effective Agent Harnesses for Abstract Reasoning and Generalization on ARC-AGI-1

Abstract: Recent progress on ARC-AGI-1 from disclosed architectures has come broadly from two regimes: heavy test-time compute over frontier models (evolutionary search, exhaustive sampling, extended chain-of-thought), or benchmark-specific training in which small models are fine-tuned on ARC data, often with task-specialized architectures. We study a third regime: an open-weight model in non-thinking mode (DeepSeek V3.2) under a strict budget, with no ARC-specific fine-tuning. We study what is recoverable through architecture alone, building agentic harnesses that decompose pattern-discovery and program-synthesis stages explicitly. First, we introduce an Explorer-Definer Pipeline that separates pattern discovery from executable transformation synthesis, implemented as a two-stage agent pipeline. Next, we present the Reflective Orchestrator, which augments the pipeline with autonomous exploration of new transformations when previous hypotheses fail on training pairs. On the ARC-AGI-1 public 400-task evaluation set, the pipeline reaches 57.50% pass@2 at \$0.25 per task, and the orchestrator reaches 67.25% pass@2 at \$0.62 per task. Together these architectures lift a 15.50% one-shot baseline by ~52 points without benchmark-specific training or heavy test-time compute. Furthermore, the orchestrator-driven lift tests a falsifiable diagnostic the pipeline produces; unbiased pass@k analysis suggests the pipeline is generation-bound, not selection-bound (selection via training-pair accuracy captures ~95% of the candidate ceiling) and predicts that significant improvement requires broader generation, not better ranking. The orchestrator implements this prediction via adaptive re-exploration and confirms it (unbiased pass@1 lift +9.81 pp, matching selection-mediated pass@2 lift). An additional pipeline ablation identifies its think tool as a significant component, with removal reducing pass@2 by 5.75 pp.

• The paper introduces structured agent harnesses that decompose abstract reasoning into pattern discovery and program synthesis stages.
• It demonstrates that adaptive re-exploration with a Reflective Orchestrator yields significant accuracy gains, with a pass@2 improvement and cost efficiency.
• Empirical analysis shows that tool-mediated scratchpad channels and candidate deduplication enhance performance without domain-specific training.

Cost-Effective Agent Harnesses for Abstract Reasoning on ARC-AGI-1

Architectural Regimes for ARC-AGI-1 Solving

The work rigorously investigates the intermediary architectural regime for solving ARC-AGI-1, distinct from high-compute frontier-model search and benchmark-specialized model training. Using DeepSeek V3.2 in non-thinking mode with strictly controlled budget and no ARC-specific fine-tuning, the authors design explicit agentic harnesses that systematically decompose the task into structured stages of pattern discovery and program synthesis. The core architectures introduced are the Explorer-Definer Pipeline and the Reflective Orchestrator. Architectural lifts are evaluated under fixed model conditions on the public ARC-AGI-1 400-task set, reporting significant accuracy gains realized purely by architecture and tool design, without reliance on domain-specific training or heavy inference compute.

Figure 1: The PatternExplorer agent showing structured note-taking and hypothesis generation for systematic pattern exploration.

Pipeline Decomposition and Role Specialization

The Explorer-Definer Pipeline achieves high abstraction by splitting ARC solving into two conceptually orthogonal stages: a fleet of PatternExplorer agents generate independent, exhaustive natural-language descriptions of candidate patterns, mediated by deliberate scratchpad token budgets and structured logging tools; subsequently, a TransformationDefiner agent synthesizes executable Python transformations conditioned on the compressed explorer artifacts and training input-output pairs. Verification is achieved by deterministic execution and exact-match scoring on training pairs, serving as an unbiased reward signal for candidate ranking. Notably, the pipeline is cost-efficient, achieving 57.50% pass@2 at $0.25 per task, exceeding chain-of-thought baselines and nearly doubling accuracy over previous single-pass architectures.

Figure 2: The Explorer-Definer Pipeline operational flow with transformation synthesis staged after parallel pattern exploration.

Adaptive Re-Exploration and the Reflective Orchestrator

Above the pipeline's architectural bottlenecks, trace-level analysis reveals generation-bound limitations: refinement within a fixed conceptual frame fails when upstream pattern abstractions are incorrect, and selection policies cannot overcome candidate pool scarcity. The Reflective Orchestrator addresses this by introducing a dynamic agentic loop wherein synthesis failures trigger autonomous spawning of focused explorer fleets conditioned on structured negative guidance from intermediate train-pair feedback. This allows mid-loop injection of new abstraction trajectories, confirmed by unbiased accuracy analysis to yield generation-driven lifts. At equivalent explorer budget, the orchestrator attains 67.25% pass@2 at $0.62 per task—a statistically significant +9.75 percentage point improvement over the static pipeline.

Figure 3: The TransformationDefiner re-cast as a Reflective Orchestrator facilitating adaptive spawn-based re-exploration.

Empirical Analysis of Cost–Accuracy Frontier

Extensive Pareto sweeps across explorer count ($N$), sampling temperature ($t$), and synthesis candidate count ($M$) demonstrate that explorer fleet diversity dominates accuracy improvements, while additional downstream sampling delivers rapidly diminishing returns. Intermediate temperature optimally balances diversity and quality in explorer trajectories. Ablations highlight the non-trivial contribution of structured scratchpad channels: removal of the definer's think tool induces a -5.75 percentage point drop in pass@2 despite reduced cost, substantiating the architectural merit of tool-mediated deliberation. Pipeline refinement yields only marginal effects, validating that deterministic selection and deduplication mechanisms absorb most candidate pool gains.

Figure 4: $N \times t \times M$ Pareto surface for the Explorer-Definer Pipeline showing accuracy saturation dependent on explorer fleet size and temperature.

Generalization Across Models and Variants

Replication on Qwen3-235B-Instruct, a distinct model family, confirms the architectural lifts generalize: each lever yields positive transitions through baseline, CoT, pipeline, and orchestrator architectures, with the same monotonic ordering and paired-bootstrap significance. The absolute accuracy is model-dependent, but the underlying harness-driven improvement trajectory is robust—reinforcing the claim that agentic harness structure, not model idiosyncrasy, induces the dominant accuracy gains in this regime.

Figure 5: Cross-model Pareto comparison on a matched ARC-AGI-1 subset, showing analogous lifts for both DeepSeek V3.2 and Qwen3-235B-Instruct across architectures.

Selection-Guided Candidate Deduplication

Deduplication by predicted test grid ensures selection amplifies candidate diversity: convergent wrong-rule definers are merged, preventing dominance by majority errors and retaining dissenting correct candidates. This is pivotal for highly ambiguous ARC tasks, illustrated on task 3a301edc, where arithmetic and geometric rules equally fit training pairs but only geometric abstraction solves the held-out test pair. Reflective Orchestrator’s spawn mechanism directly enables this dissention, confirming its status as a relief valve for wrong-abstraction failure modes and underpinning the generation-bound diagnostic confirmed by unbiased pass@$k$ estimators.

Figure 6: Task 3a301edc, a canonical ambiguous ARC instance requiring abstraction switching enabled by orchestrator spawn.

Methodological Rigor in Compute-Efficiency Claims

The evaluation protocol is stringent: all cost calculations are token-normalized and wall-clock independent, paired-bootstrap confidence intervals are reported for architectural deltas, and pass@$k$ estimation isolates generation vs. selection contributions. Ablation and sweep methods control for run-to-run variance and isolate causal effects of architectural tooling, ensuring auditability and reproducibility.

Figure 7: Cost–accuracy frontier for four primary architectures, elucidating distinct monotone moves along the sub-\$1 per task boundary.

Implications and Future Directions

Agent harness engineering recovers over 50 percentage points in absolute accuracy at dramatically lower inference cost than previously disclosed systems in this region. The findings establish that structured agentic decomposition, adaptive re-exploration, and tool-mediated scratchpad channels comprise orthogonal architectural levers with compounding effects. The methodology for generation-bound diagnostic, followed by targeted architectural intervention, offers a paradigm for future compute-efficient, general-purpose reasoning systems. Extensions suggested include further harness-level ablations (especially spawn-off), signal-driven branching policies, upstream certainty-weighted explorer ranking, cross-family panel expansion, and evaluation on ARC-AGI-2.

Conclusion

The study demonstrates that agent harnesses structured around explicit stage decomposition and adaptive exploration robustly elevate ARC-AGI-1 abstract reasoning performance without domain-specific training or excessive test-time compute. Architectural interventions targeting generation-bound bottlenecks—especially the Reflective Orchestrator’s spawn mechanism—are confirmed to yield substantial accuracy and efficiency improvement. The results generalize across models and situate the harnesses as cost-effective, scalable approaches for symbolic reasoning task benchmarks, offering a foundation for subsequent harness innovation and theoretical analysis of agent-driven generalization.