FrogNano: Training a 4B Coding Agent via Online Task Synthesis
This lightning talk examines FrogNano, a compact 4-billion-parameter coding agent that achieves 61.5% on SWE-bench Verified without behavioral distillation from larger models. The presentation explores three core contributions: how interface design becomes a capability bottleneck for small models, how synthetic tasks become more effective when calibrated against the evolving policy's solve rate, and how executable verification behavior predicts success more reliably than textual reasoning cues. The talk traces the policy-guided curriculum synthesis loop, analyzes why the fifth iteration broadens task categories while reducing scaffolding, and examines the failure modes that still limit repository-level reasoning at 4 billion parameters.Script
A 4-billion-parameter model just solved 61% of real repository engineering tasks without ever watching a larger model work. FrogNano learns purely through reinforcement learning on synthetic problems that adapt to what it can almost solve right now.
The harness matters more than you think. Switching from an elaborate multi-tool interface to five simple commands increased the base model's solve rate from 8% to 37%, while a larger model performed identically on both. Small models break on complex tool choreography that bigger models ignore.
Here's the core loop. Generate a candidate task, run the current policy on it, measure how often it succeeds. If the solve rate lands in your target band, keep the task. If it's too hard or too easy, refine the problem statement and try again. The policy determines the next curriculum, and the curriculum determines the next policy.
By iteration 5, bug fixes dropped from 85% to 41% of the curriculum, while feature requests, refactors, and performance tasks expanded. Problem statements got shorter, not longer, yet the tasks became less scaffolded and more semantically heterogeneous. Harder doesn't always mean bigger.
Successful trajectories run tests, especially after their final edit. Among mixed-success task groups, solutions were 8 percentage points more likely to verify after editing. Confidence language appeared in 65% of failures, so what the agent says about its reasoning matters less than whether it actually checks its work.
FrogNano climbed from 43% to 61% on SWE-bench Verified across five calibrated curricula, but 91% of remaining failures trace to reasoning gaps, not tool access or context limits. Wrong root cause, misread specs, and incomplete patches still dominate. The agent learned to verify, but it hasn't yet learned to diagnose reliably. To explore the full technical breakdown or create your own video walkthrough of this research, visit EmergentMind.com.
