When AI Doubts Its Own Verified Work
Large language model agents often break working solutions after receiving criticism, even when the criticism is wrong. This presentation explains Thinking-in-Bets, a test-time method that separates an agent's pre-feedback assessment from its response to user complaints. By recording claims, evidence, and confidence values before reading feedback, the method reduces false reversals from 12.2% to 0.02% in coding tasks while preserving the ability to accept valid corrections. We examine how the approach works across software engineering, academic reasoning, and autonomous coding, and explore when to apply Always-On, On-Demand, or Draft-First verification schedules.Script
Two agents have code passing all 153 tests. A user complains but gives no failing case. One agent rewrites the code and breaks three tests. The other keeps the passing code and asks which test failed. What made the difference?
The researchers discovered that a vague complaint lowers the model's confidence on correct and incorrect requirements by roughly the same amount. The criticism functions as a nonspecific downward shift, not a reliable diagnostic signal, causing agents to abandon verified work.
Thinking-in-Bets records a Claim Checklist before the agent reads the next user message. Each claim includes supporting evidence, a confidence value, and rejected alternatives. When feedback arrives, the agent maps it to affected claims, checks specific objections, and revises only claims whose updated status warrants change.
The revision policy uses claim-level selective updating. A strong claim with vague criticism triggers a request for the failing case. A weak claim with specific criticism is targeted for repair. This allows verified requirements to remain stable while uncertain or contradicted requirements are revised.
Across 10,323 requirement checks, false reversals on satisfied requirements fell from 12.2 percent with the standard method to 0.02 percent with Thinking-in-Bets. Valid corrections remained possible. This is not a modest quality improvement but the prevention of a specific causal sequence in which criticism causes the model to discard previously verified work.
Always-On Betting records claims before every reply and suits tested artifacts under review. On-Demand Betting writes a full checklist only when confidence is low, reducing token overhead by 85 percent during unfinished coding. Draft-First Betting audits the least certain claim before submission, improving accuracy and calibration without labeled data. If you want to see how test-time claim separation reshapes interactive AI workflows, visit EmergentMind.com to explore this paper further and create your own explainer videos.
