AREX: Towards a Recursively Self-Improving Agent for Deep Research
Abstract: Deep research requires agents to find answers that jointly satisfy multiple constraints. Discovering such answers is costly, whereas verifying a candidate can often be decomposed into tractable constraint-wise checks. This discovery--verification asymmetry suggests that a research agent should do more than simply search longer: it should recursively improve its current answer by verifying intermediate results and using the partially verified state to guide subsequent refinement. We introduce AREX, a family of Recursively Self-Improving (RSI) deep research agents. AREX alternates between an inner research loop that gathers evidence and constructs a provisional answer, and an outer self-improvement loop that audits the answer constraint-wise, identifies unresolved claims, and launches targeted follow-up research. To sustain RSI over long horizons, AREX learns an autonomous context-update tool that compresses growing interaction history into a compact improvement state preserving verified evidence and unresolved constraints, without relying on an external model. We train AREX on verified synthetic tasks and high-quality trajectories through agentic mid-training and long-horizon reinforcement learning. To mitigate sparse final rewards during long horizon learning, we emphasize key steps where decisive evidence is acquired or erroneous research directions are corrected. We instantiate a dense 4B model and a 122B-A10B Mixture-of-Experts model. Across BrowseComp, WideSearch, DeepSearchQA, Humanity's Last Exam (HLE), and other reasoning and tool-use benchmarks, AREX substantially outperforms comparable-scale baselines and remains competitive with models using substantially more activated parameters.
First 10 authors:
Summary
- The paper presents a recursive self-improving framework that integrates evidence tracking, tool use, and confidence-based validation to guide deep research.
- It employs a bi-level loop structure combining an inner research loop with an outer self-improvement loop to incrementally build and verify provisional answers.
- Ablation studies highlight that both the autonomous context updates and targeted RL supervision are critical for achieving high accuracy and efficiency.
AREX: Recursively Self-Improving Deep Research Agents
Motivation and Problem Definition
The AREX architecture targets the deep research regime, where agents must synthesize information from heterogeneous sources to produce answers that satisfy multiple interdependent constraints. This entails not only exploration in a large, sparsely informative search space but also rigorous verification against individual task constraints, often in the presence of conflicting or partial evidence. The core insight behind AREX is the exploitation of discovery–verification asymmetry: while the discovery of a fully satisfactory solution is computationally expensive, the constraint-wise verification of candidate solutions is significantly cheaper and decomposable. Prior LLM-based agents have focused on extending inference trajectories, but these generally lack structured, recursive mechanisms for leveraging partially verified progress to guide targeted refinements, often resulting in inefficient or stalled searches.
AREX Architecture and Framework
AREX introduces a hierarchical, bi-level recursive self-improvement framework, comprising an inner research loop and an outer self-improvement loop. The inner loop is responsible for incremental construction of a provisional answer, integrating tool use, document retrieval, evidence tracking, and intermediate reasoning. Once a provisional answer is formed, it is coupled with a supporting evidence set and an explicit confidence estimate.
The outer self-improvement loop operates as a meta-controller: if the confidence estimate surpasses a threshold, the provisional answer is accepted; otherwise, the loop diagnoses which constraints remain unverified or lack sufficient evidence. Depending on this diagnosis, the loop either targets a refined subproblem for further investigation (preserving verified progress) or restarts from scratch if the trajectory is judged to be irrecoverably noisy or misleading.
Figure 1: AREX’s recursive self-improvement framework, illustrating bi-level control between the inner research loop and the outer self-improvement loop.
A central innovation of AREX is its learned, autonomous context-update mechanism. Rather than relying on naive context truncation or heuristic summarization, AREX invokes a dedicated update_context tool to compress growing interaction histories into a concise improvement state. This includes verified evidence, candidate rejection logs, unresolved constraints, and the next-step research plan, tightly coupled to the agent’s current objective. Unlike external summarization, this process is internal to the agent’s workflow, preserving relevance and enabling efficient long-horizon operation.
Training Pipeline and Data Construction
AREX utilizes a multi-stage, agentic mid-training pipeline. The process begins with progressively staged training:
- Browse-intensive training: Establishes tool-use, web navigation, and iterative evidence acquisition by imitating high-quality trajectories.
- Reasoning-intensive training: Focuses on long-form reasoning, multi-step deduction, and hypothesis verification.
- Mixed-capability consolidation: Merges the two domains, with explicit focus on critical decision points rather than routine transitions.
A key challenge in long-horizon trajectories is credit assignment, as the sparse final reward does not reveal which intermediate steps were decisive. To address this, AREX designates key steps—such as initial acquisition of answer-relevant evidence, path rejection and course correction, or substantial context updates—and applies targeted supervision and auxiliary RL shaping rewards to these events.
Figure 2: Left: Average loss per step after full-trajectory mid-training, indicating that key steps (evidence acquisition, redirection, context update) remain underfit compared to ordinary steps. Right: Ablation results on BrowseComp demonstrating the impact of each training component on accuracy.
Empirical Results and Comparative Performance
AREX is instantiated as both a dense 4B-parameter model (Turbo) and a Mixture-of-Experts 122B-total (10B active) model (Base), both leveraging Qwen3.5 foundations. The models are evaluated on a diverse suite of benchmarks, including BrowseComp, WideSearch, DeepSearchQA, GAIA, the xbench-DeepSearch-2510, and Humanity's Last Exam (HLE). AREX-Base, with only 10B active parameters, demonstrates competitive or superior performance to much larger open-source and closed-source models, including Qwen3.5-397B, DeepSeek-V4-Pro, and MiroThinker-H1, on multiple dimensions:
Figure 3: Benchmark performance of AREX on a representative suite of deep research, information synthesis, and agentic reasoning tasks.
On BrowseComp, AREX-Base matches or surpasses the accuracy of Qwen3.5-397B and outperforms specialized research agents. On WideSearch-en, AREX-Base achieves the highest reported score. The empirical analysis underscores a strong parameter-to-capability efficiency, with AREX-Turbo (4B) exceeding Qwen3.5-35B on five out of six benchmarks.
Mechanism Analysis and Ablations
Ablative analysis isolates the contribution of AREX's architectural and training modules. Disabling the autonomous context update (ACU) tool and outer self-improvement loop results in marked performance degradation (59.6 BrowseComp accuracy vs. 82.5 in the full system). ACU alone confers an 11.8 point gain; enabling only the outer loop (without ACU) adds 10.2 points. The interplay between inner and outer loops yields a cumulative improvement exceeding 22 points. The learned confidence score effectively separates correct from incorrect results, enabling robust confidence-based acceptance/rejection criteria.
Figure 4: Left: Accuracy improvements attributable to ACU and the outer self-improvement loop. Right: Histogram of confidence score distributions for correct vs. incorrect outputs, demonstrating effective error separation.
Further ablations show that omitting progressive multi-round training, key-step focused supervision, or step-aware RL substantially degrades performance, confirming that each training component is complementary and critical for end-to-end reliability.
Practical and Theoretical Implications
AREX enables structured recursive progress, mitigating the typical inefficiencies seen in long-horizon research agents. The learned context-update tool avoids redundant reasoning and preserves decision-relevant state, while recursive refinement ensures that model effort is concentrated on unresolved or weakly supported sections of the solution. This permits both increased longitudinal depth and higher sample efficiency, critical for advanced tool-augmented research applications in domains ranging from automated literature review to scientific synthesis and regulatory compliance auditing.
The framework formalizes partial verification as a dynamic operator for research-state transition, offering a principled direction for future agentic AI design. The results suggest that autonomous research-state maintenance and decision-critical credit assignment are, jointly, necessary ingredients for high-fidelity, scalable agentic systems. The architecture supports extensions, such as model-based estimation of step importance, finer-grained research-state decomposition, or integration with external verification or critique modules.
Conclusion
AREX presents a robust, recursively self-improving agentic architecture for deep research tasks, substantiated by superior empirical performance and efficiency across standard benchmarks. The approach leverages discovery–verification asymmetry, recursive refinement loops, and learned context management to deliver systematic long-horizon research capability. The findings reinforce the hypothesis that recursive verification-guided state compression, in conjunction with step-aware optimization, is a key enabler for the next generation of tool-augmented AI research agents.
Whiteboard
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
A simple explanation of “AREX: Towards a Recursively Self-Improving Agent for Deep Research”
1. What is this paper about?
This paper introduces AREX, an AI “research assistant” that gets better at answering hard questions by checking its own work and improving it step by step. Instead of just searching more, AREX repeatedly:
- builds a draft answer,
- checks it against a checklist of requirements,
- keeps the parts that are proven correct,
- fixes what’s still uncertain,
- and repeats until confident.
The big idea: discovering a perfect answer is hard, but checking parts of an answer is often easier. AREX uses this to guide smarter follow-up research.
2. What questions are the researchers trying to answer?
In simple terms, they ask:
- How can an AI tackle complex questions that have many conditions to satisfy (like a checklist), not just one?
- Can an AI improve its own partial answers by verifying them piece by piece, rather than starting over or wandering aimlessly?
- How can an AI remember long, messy research histories without getting confused or forgetting important facts?
- Can this approach beat other AI systems on tough research and reasoning tests?
3. How does AREX work? (Methods in everyday language)
Think of AREX like a detective with two modes and a tidy notebook.
- Inner research loop (the detective at work): The AI searches the web, reads sources, gathers clues, and writes a draft answer. It notes which clues support which parts and how confident it is.
- Outer self-improvement loop (the supervisor check): The AI evaluates the draft using a checklist of constraints (the conditions the final answer must meet). If confidence is high enough, it stops. If not, it decides:
- Refine: keep the good, verified parts and focus next round on the missing or weak parts, or
- Restart: if the current path is messy or misleading, begin fresh.
- Autonomous context updating (keeping a clean notebook): During long research, the AI’s “notes” can get overwhelming. AREX uses a special tool, update_context, to clean up and compress its history into a compact “improvement state.” This keeps:
- verified facts (with sources),
- what’s still unresolved,
- rejected ideas (so it doesn’t repeat mistakes),
- and the next plan.
- It throws out repeated or outdated bits so the next steps are clear.
- Training (how it learned):
- Synthetic practice tasks: The team created carefully designed “puzzles” that require multiple steps and evidence.
- Teacher examples: Strong models showed how to research properly (like worked examples), including tool use and citing evidence.
- Key-step focus: Not every step in a long project matters equally. AREX gets extra practice on critical moments—like when it finds a decisive source or corrects a wrong path.
- Reinforcement learning: The AI gets rewarded for good final answers, plus small extra rewards for those key, high-impact steps. This helps it learn which moves matter most in long, complex searches.
4. What did they find, and why does it matter?
The researchers built two versions:
- AREX-Turbo (about 4 billion parameters),
- AREX-Base (Mixture-of-Experts: 122B total, ~10B active per step).
Across several tough benchmarks that require deep web search, reasoning, and tool use (like BrowseComp, WideSearch, DeepSearchQA, GAIA, HLE, xbench), AREX:
- clearly outperforms other models of similar size,
- competes well with much larger systems,
- and shows especially strong results on tasks that need careful multi-step research and synthesis.
Why this matters:
- It shows that “verify-then-improve” beats just “search more.”
- The context-update tool helps the AI stay organized during very long tasks, avoiding repeated mistakes and confusion.
- AREX uses its computing power efficiently (doing well even when only ~10B parameters are active), which is practical for real-world use.
5. What’s the impact of this research?
This approach points to smarter, more reliable AI research assistants. Instead of guessing and hoping, AREX:
- builds answers like a careful investigator,
- keeps proof (citations) for each claim,
- knows what’s still missing,
- and steadily improves until it’s confident.
Potential benefits include:
- better tools for students, journalists, and scientists who need trustworthy, multi-source answers;
- more efficient long-term problem solving (fewer dead ends, less repetition);
- a foundation for safer AI research agents that can explain and verify their reasoning.
In short, AREX shows that making verification the driver of the process—turning partial answers into clearer next steps—can make AI much better at deep, complicated research.
Knowledge Gaps
Below is a single, focused list of concrete knowledge gaps, limitations, and open questions that the paper leaves unresolved. Each point is intended to be actionable for follow-up research.
- Confidence calibration: How well is the answer-level score s in [0,100] calibrated across tasks and domains, and does miscalibration cause premature Accept or unnecessary Refine/Restart decisions?
- Termination policy design: What are principled, adaptive methods to set and update the confidence threshold τ per task (or per constraint), rather than using a fixed global threshold?
- Per-constraint verification signals: Can the agent produce calibrated, constraint-wise satisfaction probabilities (and rationales) to replace or augment a single answer-level confidence score?
- Monotonic improvement guarantees: Under what conditions does the outer loop’s Refine/Restart policy provably reduce uncertainty or error over rounds (e.g., monotone progress or bounded regret)?
- Failure mode taxonomy for RSI: What systematic failure modes arise (e.g., oscillation between candidates, confirmation bias within ACU states, repeated rediscovery), and how can they be detected and corrected online?
- Fidelity of autonomous context updating (ACU): What objective metrics quantify information retention, provenance fidelity, and lossiness of update_context over many updates?
- ACU robustness to errors: How often do ACU-induced omissions or distortions propagate to later rounds, and can a verification/auditing pass detect and repair ACU-induced state drift?
- Independent auditing of ACU: Would using a separate verifier (model or rule-based checker) to audit ACU outputs reduce self-justification and hallucination risks compared to self-produced updates?
- Provenance granularity: How precisely are citations preserved (passage-/span-level vs page-level), and do citations continue to support claims after repeated ACU compressions and refinements?
- Restart vs Refine sensitivity: How sensitive is performance to the outer-loop decision boundary v(k), and what learning or meta-control strategies optimize this decision under variable tool noise and task hardness?
- Key-step detector generalization: The rule-based key-step detectors are domain-specific; how well do they transfer to unseen tasks, tools, and evidence types without re-engineering?
- Shaping-reward side effects: Does the auxiliary key-step bonus induce spurious behaviors (e.g., gaming detector heuristics) or reduce exploration of equally valid alternative paths?
- Credit assignment beyond key steps: Can more general, learnable credit assignment (e.g., counterfactual baselines, hindsight relabeling, temporal-difference critics) outperform heuristic key-step shaping?
- Tool-failure robustness: How does AREX handle flaky search results, timeouts, rate limits, or injected noise, and can it learn robust recovery policies (e.g., redundancy, backoff, alternative tools)?
- Prompt-injection and adversarial web content: What defenses (e.g., content sanitization, policy sandboxes, retrieval isolation) are needed for safe browsing in agentic settings?
- Temporal validity and staleness: How reliably can the agent detect outdated sources and perform time-aware reasoning, and can it maintain or update time-sensitive beliefs across rounds?
- Abstention and no-answer behavior: When constraints remain unmet after budget exhaustion, can calibrated abstention or partial answers outperform returning the highest-confidence (possibly wrong) answer?
- Budget-aware planning: How should AREX allocate inner-loop steps, ACU frequency, and outer-loop rounds under tight cost/latency constraints while preserving accuracy?
- Scalability of RSI depth: What happens when tasks require more than 5 outer-loop rounds or >300 inner steps—do returns diminish, and how should depth be scheduled adaptively?
- Cross-task memory: Can verified findings be persisted across episodes (lifelong memory) without contaminating future tasks or amplifying bias?
- Multilingual and multimodal coverage: How does AREX perform on non-English sources and multimodal evidence (PDF tables, figures, diagrams, code, datasets), and what extensions are needed?
- Domain and source diversity: Synthetic and teacher-derived data may underrepresent messy, real-world information; how does performance change with heterogeneous, evolving web corpora?
- Dataset transparency and replication: Are the synthesized tasks, trajectories, and detectors released with enough detail to fully reproduce training and evaluation?
- Benchmark realism: Do current benchmarks capture adversarial content, temporal drift, paywalled sources, and open-ended research goals typical of real investigations?
- Fair-comparison controls: Were all baselines run with identical tools, budgets, and environments; if not, what is the effect size of interface differences?
- Backbone dependence: How portable is AREX to non-Qwen backbones, different MoE routings, and smaller/larger active-parameter budgets?
- Ablations for core components: What is the isolated contribution of ACU, structured finish, outer-loop Refine/Restart, key-step shaping, and step-aware GPO under controlled, identical setups?
- Constraint discovery from natural queries: Beyond templated tasks, how reliably can AREX infer and maintain a correct set of latent constraints directly from arbitrary user questions?
- Handling under-specified or conflicting constraints: How does AREX detect underdetermined problems, request clarification, or reconcile fundamentally incompatible constraints?
- Negative evidence management: How systematically is disconfirming evidence tracked, and can explicit contradiction-detection modules reduce confirmation bias?
- Information re-discovery metrics: How often does the agent revisit rejected candidates or repeat unproductive queries, and can ACU or planning be modified to minimize redundancy?
- Security and privacy: What guardrails ensure compliance when browsing (e.g., PII handling, licensed content), and how are sensitive outputs controlled in long-horizon loops?
- Energy and compute footprint: What are the training/inference costs of RSI vs single-trajectory agents, and can distillation or policy reuse reduce overhead without regressing accuracy?
- Theoretical framing: Can the RSI process be formalized as a POMDP with per-constraint belief states, enabling analysis of optimal policies, stopping rules, and regret bounds?
- Uncertainty-aware planning: Would Bayesian ensembling or epistemic uncertainty estimates improve decision-making for Refine/Restart, tool selection, and exploration breadth?
- Human-in-the-loop integration: When and how should human feedback intervene (e.g., verifying citations, adjusting constraints) to amplify reliability without high oversight costs?
Practical Applications
Below are practical applications of the paper’s AREX framework, organized by near-term deployability and longer-horizon opportunities. Each item includes sector links, prospective tools/workflows, and key assumptions or dependencies.
Immediate Applications
- Evidence-grounded research copilot for multi-constraint questions
- Sectors: consulting, policy analysis, legal, finance
- Tools/products/workflows: constraint-wise research briefs with citations and an answer-level confidence; “Refine/Restart” loops to target unresolved claims; exportable audit appendix (sources, timestamps)
- Assumptions/dependencies: access to web search/enterprise document stores; basic source reliability heuristics; human-in-the-loop sign-off
- Systematic literature review and evidence table builder
- Sectors: healthcare, pharma, academia
- Tools/products/workflows: PICO-style constraint tracking; autonomous context updates to manage thousands of abstracts; PRISMA-style flow and evidence tables
- Assumptions/dependencies: APIs for PubMed/Embase/Cochrane; de-duplication; strict provenance retention; clinical claims reviewed by domain experts
- Regulatory compliance and policy conformance checking
- Sectors: finance (KYC/AML), energy (grid codes), healthcare (HIPAA), privacy (GDPR/CCPA)
- Tools/products/workflows: map requirement checklists to documentary evidence; “partially verified state” highlights gaps; confidence-gated submission packages with citations
- Assumptions/dependencies: up-to-date regulation repositories; robust document ingestion; mandatory human approval for regulated filings
- Legal research triage and e-discovery support
- Sectors: legal
- Tools/products/workflows: case law retrieval with constraint-wise validation (jurisdiction, date, holding); contract clause localization vs RFP/regulatory constraints; confidence-ranked memo drafts
- Assumptions/dependencies: access to legal databases; citation pin-cites; attorney review for interpretation
- Competitive intelligence and market landscaping
- Sectors: software, biotech, consumer goods
- Tools/products/workflows: attribute-by-attribute product comparisons; unresolved-constraint flags (e.g., missing pricing, region availability); rolling updates via autonomous context refresh
- Assumptions/dependencies: OSINT/data feeds; source credibility scoring; periodic human curation
- Patent prior-art and freedom-to-operate triage
- Sectors: IP strategy, R&D
- Tools/products/workflows: query reformulation for CPC/IPC codes; constraint checks on claims overlap and priority dates; evidence dossier with family links
- Assumptions/dependencies: patent databases (USPTO, EPO, WIPO); claim-parsing tools; IP counsel validation
- Vendor/RFP response evaluation and scorecards
- Sectors: procurement, IT
- Tools/products/workflows: auto-check vendor docs against multi-constraint RFPs; “Refine” loop to chase missing proofs; scorecards with cited passages
- Assumptions/dependencies: structured RFP requirement models; document normalization; procurement oversight
- Software engineering research assistant
- Sectors: software
- Tools/products/workflows: multi-repo code search and doc triangulation; bug reproduction steps with evidence and confidence; “Restart” to pivot from dead ends; python sandbox for quick checks
- Assumptions/dependencies: code indexers, CI logs, IDE integrations; sandboxed execution; developer review
- Data quality assessment and lineage verification
- Sectors: data platforms, analytics
- Tools/products/workflows: schema/SLAs as constraints; evidence from catalog/lineage logs; compact improvement state retains verified checks and open issues
- Assumptions/dependencies: access to data catalogs, lineage graphs, monitoring; governance policies for changes
- Customer support knowledge synthesis with confidence gating
- Sectors: SaaS, telecom, retail
- Tools/products/workflows: verified answers with citations to KB/tickets; escalate unresolved constraints; track confidence thresholds before auto-reply
- Assumptions/dependencies: connectors to KB/CRM; guardrails for outdated/unsupported guidance
- Investigative journalism and fact-checking
- Sectors: media
- Tools/products/workflows: cross-source corroboration; conflict localization and targeted follow-ups; publishable source log with timestamps
- Assumptions/dependencies: news/APIs/archives; editorial oversight; risk/legal review
- Enterprise knowledge curation and wiki updates
- Sectors: cross-industry
- Tools/products/workflows: continuous synthesis with unresolved-constraint trackers; change logs and confidence deltas; citation-preserving edits
- Assumptions/dependencies: content lifecycle governance; identity/permissions; periodic human review
Long-Term Applications
- Continuous, living systematic reviews and clinical guideline maintenance
- Sectors: healthcare, pharma
- Tools/products/workflows: streaming evidence intake; automated “Refine” cycles when new RCTs appear; confidence-aware guideline updates
- Assumptions/dependencies: high-recall ingestion, bias/risk-of-bias assessments; formal validation and regulatory acceptance
- Closed-loop scientific discovery agents
- Sectors: biotech, materials, chemistry
- Tools/products/workflows: integrate lab robotics/ELNs; constraints from assay quality and safety; recursive refinement of hypotheses from experiment results
- Assumptions/dependencies: lab APIs, safe autonomy constraints, IRB/biosafety compliance; rigorous causal evaluation
- Autonomous compliance copilot with real-time regulation drift tracking
- Sectors: finance, energy, privacy
- Tools/products/workflows: change detection in statutes/rulings; auto-proposed policy deltas with unresolved-constraint lists; periodic audit packages
- Assumptions/dependencies: authoritative feeds; legal sign-off; versioned evidence archives
- Self-maintaining enterprise knowledge graphs with verified claims
- Sectors: software, manufacturing, retail
- Tools/products/workflows: convert partially verified states into KG updates; confidence-weighted edges; contradiction resolution workflows
- Assumptions/dependencies: KG platforms; provenance and conflict-resolution policies; data stewards
- AI agent auditor (“verification-first” oversight)
- Sectors: AI governance, safety
- Tools/products/workflows: constraint-wise audits of other agents’ outputs; confidence thresholds for deployment; evidence trails for red-teaming
- Assumptions/dependencies: agent orchestration hooks; standardized audit schemas; organizational adoption
- Multimodal incident root-cause analysis across logs and documents
- Sectors: cloud/SRE, manufacturing, aviation
- Tools/products/workflows: unify telemetry, tickets, runbooks as constraints; recursive refinement to isolate causal chains; compact incident state for war rooms
- Assumptions/dependencies: secure access to logs/PII; time-sync and schema normalization; human adjudication
- Advanced RPA with constraint-aware verification and restart logic
- Sectors: operations, back-office
- Tools/products/workflows: long-horizon task chains that verify intermediate states; automated “Restart” on noisy trajectories; audit-ready traces
- Assumptions/dependencies: robust tool connectors; reliability/SLA guarantees; exception-handling playbooks
- Strategic forecasting with evidence-weighted scenarios
- Sectors: policy, finance, supply chain
- Tools/products/workflows: scenario constraints (signals, timelines) with confidence and unresolved gaps; periodic refinement as new data arrives
- Assumptions/dependencies: high-quality macro/micro data feeds; calibration pipelines; decision-theoretic evaluation
- Agentic MLOps instrumentation using key-step telemetry
- Sectors: AI/ML platforms
- Tools/products/workflows: log “key steps” and shaping signals to improve training and debugging; trajectory analytics for credit assignment
- Assumptions/dependencies: instrumentation standards; privacy-safe telemetry; RL/finetuning infrastructure
- Generalized memory and context management SDK (ACU as a service)
- Sectors: software, agent platforms
- Tools/products/workflows: drop-in autonomous context-update component for long-horizon agents; APIs to preserve verified evidence, unresolved constraints, and next-step plans
- Assumptions/dependencies: integration with agent frameworks; evaluation harnesses for regressions; token-cost management
Notes on feasibility and cross-cutting dependencies
- Reliability and safety: Many deployments need human-in-the-loop gating, conservative confidence thresholds, and domain-specific validation (especially medical/legal/financial use).
- Data and tools: Performance depends on access to high-quality search/browsing, proprietary databases, document parsers/OCR, code/index search, sandboxed compute, and robust connectors.
- Provenance and auditing: The structured finish interface and constraint-wise state are only useful if citations are preserved, timestamped, and versioned.
- Scalability: Autonomous context updating reduces context bloat, but large-scale use still needs memory stores, caching, and cost controls.
- Governance: For regulated sectors, organizational policies must define when “partially verified” is acceptable, audit procedures, and escalation paths.
Glossary
- Activated parameters: The subset of parameters actually used during inference in a Mixture-of-Experts model. Example: "a Mixture-of-Experts model with 122B total parameters and 10B activated parameters (Base)."
- Active context window: The maximum token budget the agent can condition on during an episode. Example: "The agent operates within a 128K-token active context window."
- Agentic mid-training: An intermediate training phase focused on developing autonomous tool-use and decision-making behaviors before final RL. Example: "through agentic mid-training and long-horizon reinforcement learning."
- Autonomous context updating: A model-initiated process that consolidates interaction history into a compact, decision-relevant research state. Example: "Rather than generic summarization, autonomous context updating performs trajectory consolidation and research-state refreshing."
- Confidence threshold: The minimum confidence score required for the outer loop to accept an answer. Example: "Let denote the confidence threshold."
- Constraint-wise checks: Verification decomposed per required condition of a candidate solution. Example: "evaluating a proposed candidate can often be decomposed into substantially simpler constraint-wise checks."
- Critical-interval exposure: Training that increases emphasis on decisive moments within long trajectories to aid credit assignment. Example: "together with critical-interval exposure for improving credit assignment in long research trajectories."
- Credit assignment: The problem of attributing final outcomes to the specific intermediate actions that caused them. Example: "Long trajectories also create a sparse credit-assignment problem: final rewards do not reveal which intermediate actions produced decisive progress."
- Discovery–verification asymmetry: The phenomenon where discovering a full solution is expensive while verifying parts of a candidate is comparatively easier. Example: "We observe that deep research tasks exhibit a fundamental discovery--verification asymmetry."
- Effective context: The compressed, refreshed state the model conditions on after invoking the context-update tool. Example: "We denote the effective context available to the model at step by ."
- Evidence-grounded: Based on verifiable, cited observations rather than unsupported claims. Example: "we apply evidence-grounded answer filtering."
- Finish interface: A structured tool call that externalizes the provisional answer, supporting evidence, and confidence score. Example: "the model invokes a structured finish interface."
- Improvement state: A compact representation of progress that preserves verified evidence and unresolved constraints to guide subsequent research. Example: "a compact improvement state preserving verified evidence and unresolved constraints"
- Inner research loop: The process that gathers evidence, updates hypotheses, and constructs a provisional answer for the current objective. Example: "AREX alternates between an inner research loop that gathers evidence and constructs a provisional answer"
- KL penalty: A regularization term that discourages the learned policy from drifting too far from a reference policy using KL divergence. Example: "We add a KL penalty against a reference policy:"
- Long-horizon reinforcement learning: RL over extended sequences with sparse feedback, typical in deep research tasks. Example: "long-horizon reinforcement learning."
- Mixture-of-Experts (MoE): An architecture with multiple expert sub-networks where only a subset is activated per input. Example: "a 122B-A10B Mixture-of-Experts model."
- Outer self-improvement loop: The controller that audits the provisional answer, decides to accept, refine, or restart, and formulates the next objective. Example: "and an outer self-improvement loop that audits the answer constraint-wise, identifies unresolved claims, and launches targeted follow-up research."
- Provenance: The origin and traceability of evidence sources supporting claims. Example: "When sources conflict, the model seeks evidence with stronger authority, more direct provenance, or greater temporal relevance."
- Recursively Self-Improving (RSI): A process where an agent repeatedly refines its solution by leveraging verification to guide subsequent research rounds. Example: "Recursively Self-Improving (RSI) deep research agents."
- Reference policy: A baseline policy used to compute KL regularization during optimization. Example: "We add a KL penalty against a reference policy:"
- Research trajectory: The sequence of intermediate analyses, actions, and observations during investigation. Example: "the loop assesses whether the current research trajectory is recoverable"
- Reward shaping: Adding auxiliary rewards to emphasize informative intermediate steps without replacing the main outcome reward. Example: "we introduce step-level reward shaping to emphasize informative intermediate decisions."
- Rollout: A sampled interaction sequence (trajectory) generated by a policy to evaluate and improve performance. Example: "Independent research rollouts are further used to discard tasks that can be solved through shallow retrieval or remain unsolved after extensive investigation."
- Search-augmented reasoning: Reasoning enhanced by external search and tool use to gather and verify information. Example: "We evaluate AREX on six benchmarks spanning four complementary regimes of search-augmented reasoning."
- Step-aware Group Policy Optimization: A turn-level policy optimization approach that balances learning across steps in long trajectories. Example: "Step-aware Group Policy Optimization"
- Structured Answer Externalization: The mechanism that converts the terminal inner-loop state into a structured answer, evidence set, and confidence score. Example: "Structured Answer Externalization"
- Temporal validity: The extent to which information remains correct at the relevant time. Example: "temporal validity of the answer."
- update_context (tool): A tool that consolidates the accumulated interaction history into a compact, reusable research state. Example: "AREX provides an explicit update_context tool."
Collections
Sign up for free to add this paper to one or more collections.