OpenSIR: Autonomous Self-Improving Reasoner

• OpenSIR is a self-improving reinforcement learning framework where language models generate, solve, and refine diverse tasks through self-play.
• It uses alternating teacher–student roles and dense internal reward mechanisms to evaluate task novelty, correctness, and difficulty.
• The framework demonstrates practical gains in mathematical problem solving, open-ended QA, and specialized domains via recursive self-reflection and dynamic curriculum expansion.

An Open-Ended Self-Improving Reasoner (OpenSIR) is a reinforcement learning-based framework in which a LLM autonomously evolves its reasoning abilities by generating, solving, and refining novel tasks without external supervision or static reward models. OpenSIR leverages self-play, internal consistency judgments, and dynamic exploration strategies to enable continual and scalable improvement across increasingly complex reasoning domains, such as mathematics, scientific problem solving, and open-ended question answering. The concept is realized through alternation of teacher–student roles, dense self-judging mechanisms, recursive self-improvement algorithms, and co-evolutionary approaches, resulting in models that discover and master ever more sophisticated reasoning strategies and problem spaces (Kwan et al., 1 Nov 2025).

1. Conceptual Foundations and Motivation

The OpenSIR paradigm emerges from the limitations of reinforcement learning with verifiable rewards (RLVR). RLVR achieves strong performance in domains where outcomes are checkable (e.g., math proofs, code synthesis), but most real-world reasoning is open-ended, lacking generic, scalable, and hack-resistant reward signals (Xu et al., 16 Jun 2025). Self-play offers a route to bypass hand-crafted rewards and data-labeling bottlenecks. OpenSIR unifies approaches where an LLM plays both generator and solver roles, calibrating task difficulty and diversity internally, and using self-consistency, solve-rate, and process reward attribution to measure improvement (Kwan et al., 1 Nov 2025, Zhang et al., 2 Feb 2026, Kwan et al., 29 May 2026).

Key properties distinguishing OpenSIR frameworks include:

• Fully autonomous curriculum: Models invent tasks at the edge of their capability.
• Internal self-judgment: Quality and progress are evaluated by the model itself or frozen copies thereof, typically without recourse to human-labeled data or external oracles.
• Open-endedness: The space of tasks, skills, and reasoning strategies grows unbounded by design—not confined to fixed datasets or benchmarks.
• Continuous improvement: Teacher and student (or analogous Challenger/Solver) roles co-evolve to push the reasoning frontier autonomously.

2. Core Algorithms and Self-Improvement Mechanisms

Classical OpenSIR implementations alternate between teacher (task generation) and student (problem-solving) modes within a single LLM policy. The primary self-improvement loop, as realized in (Kwan et al., 1 Nov 2025), involves:

1. Problem Generation: From a seed pool of problems, the teacher samples references and generates novel candidate problems, filtering for format and validity.
2. Solution Sampling: For each generated problem, multiple solution attempts are sampled. The modal answer determines solve-rate: $s_{q} = \mathrm{count}(a^*)/G$ where $a^*$ is the most common answer among $G$ samples.
3. Scoring and Selection: Problems are scored by novelty (embedding-based diversity), solution difficulty (calibrated via solve-rate), length, and formatting. Solutions are scored for correctness and alignment with expected answer format.
4. Policy Update: Teacher and student experiences are aggregated; rewards normalized by group mean and variance (Group Relative Policy Optimization), and the LLM parameters updated using a clipped policy-gradient objective with a KL-regularization to reference policy.
5. Curriculum Expansion: Validated and rewarding problem–solution pairs are added to the pool, enabling the model to bootstrap from trivial seeds to advanced concepts.

Variants refine this architecture by incorporating:

• Dense process rewards: Token-level attribution for credit assignment, e.g., by leveraging model gradients with respect to judge decisions as in Grad2Reward (Zhang et al., 2 Feb 2026).
• Self-consistency and dynamic filtering: Outlier or trivial prompts are removed via solve-rate, token variance, or difficulty calibration dynamics (Xu et al., 16 Jun 2025).
• Explicit co-evolution: Challenger/Solver roles (akin to teacher/student) evolve separately but interact via a frozen or lagged judge, as in SCOPE (Kwan et al., 29 May 2026).
• Recursive self-reflection: Multi-stage reasoning via structured "thinking" and "reflection" modalities informed by preference-based optimization and recursive feedback (Buehler, 2024).

A representative OpenSIR pseudocode core is shown below (Kwan et al., 1 Nov 2025):

Initialize policy π_θ P₀ ← { “What is 1+1?” } for t in 1…T: # Teacher generates new problems from pool P_{t-1} # Student samples solutions # Rewards computed for novelty, solve-rate, diversity # Update policy via group-normalized PPO # Pool expansion with validated new problems

3. Rewarding Mechanisms and Dynamic Evaluation

OpenSIR frameworks replace fixed or binary reward signals with dense, internally generated metrics reflecting both process and outcome properties, tailored for open-ended reasoning:

• Reasoning Reflection Reward (R³): Measures, at a per-token level, how much each reference output token reflects the preceding reasoning, with variance-based key-token selection and masked baseline subtraction to isolate true reasoning-dependent contributions (Xu et al., 16 Jun 2025).
• Gradient-based Attribution: Grad2Reward computes, via a single backward pass, gradient × embedding contributions from a frozen self-judge for each token, yielding precise credit assignment and stable process-level RL (Zhang et al., 2 Feb 2026).
• Solve-rate calibration: Difficulty reward is configured to maximize problems solved neither too easily nor rarely; often using triangular functions centered on an intermediate solve-rate (e.g., 0.7) (Kwan et al., 1 Nov 2025).
• Diversity scoring: Embedding or concept grammar-based metrics promote exploration of new conceptual spaces. Table 1 illustrates the key reward components for teacher and student roles in a canonical OpenSIR loop (Kwan et al., 1 Nov 2025):

Reward Component	Teacher (Problem)	Student (Solution)
Main criterion	solve-rate/difficulty	correctness
Diversity (embedding/concept)	yes (γ score_div)	—
Length, formatting	yes (λ score_len, δ FOM)	yes (δ FOM)

A plausible implication is that dense and dynamically modulated reward signals are essential for open-ended reasoning improvement, preventing stagnation in trivial or unattainable subspaces.

4. Open-Endedness, Co-Evolution, and Multi-Agent Extensions

Co-evolutionary principles extend traditional self-play. SCOPE (Kwan et al., 29 May 2026) and DGM-style architectures (Zhang et al., 29 May 2025) instantiate this through:

• Challenger/Solver alternation: The Challenger crafts document-grounded tasks, optimizes for frontier difficulty and novelty, and is rewarded for producing challenging but solvable problems.
• Self-judge as Rubric Generator: A frozen base model generates detailed task-specific evaluation rubrics and quality gates (entity identifiability, source relevance), grades response sequences, and detects reward hacking (e.g., answer inflation).
• Multi-agent self-reflection and critique: Dynamic Alignment (Anantaprayoon et al., 5 Dec 2025) suggests multi-policy instantiation with cross-evaluation, enforcing robustness against circular drift and bias.
• Archive growth: Darwin Gödel Machine-inspired methods build explicit archives of agent variants, sampling parents not only by performance but also by low child count (novelty incentive). Multiple child lineages evolve in parallel, extracting diverse reasoning methodologies.

Empirical evidence indicates that such co-evolutionary loops yield both rapid and sustainable accuracy gains across open-ended benchmarks—surpassing data-driven RL approaches using curated prompts (Kwan et al., 29 May 2026).

5. Practical Implementations and Empirical Results

OpenSIR frameworks have been demonstrated across mathematical problem-solving (GSM8K, Minerva), open-ended QA (HealthBench, ResearchQA), and specialized domains (biological materials science):

• OpenSIR (self-play math) (Kwan et al., 1 Nov 2025): Llama-3.2B improves from 73.9% to 78.3% on GSM8K and from 28.8% to 34.4% on College Math, overtaking GRPO RL baselines (see Table below). Gemma-2-2B achieves 38.5% to 58.7% on GSM8K.
• SCOPE (open-domain self-play) (Kwan et al., 29 May 2026): Qwen2.5-7B sees a +10.4 point gain, Qwen3-8B +5.4, OLMo-3-7B +7.8, with transfer gains up to +13.8 across short-form QA tasks.
• Grad2Reward (dense process reward) (Zhang et al., 2 Feb 2026): Provides 2–5 point improvements across medical and research benchmarks, with 1.8× faster convergence.
• DRO (long-form reasoning) (Xu et al., 16 Jun 2025): Achieves ParaRev pairwise win-rate of 58.8% versus Claude judge, and 67.1% Pass@1 on FinQA—outperforming aggregate-certainty and ROUGE-based RL.

Model	GSM8K	College Math	Avg. gain
Base	73.94%	28.78%	–
GRPO	79.72%	33.33%	+3.1 pts
OpenSIR	78.28%	34.42%	+3.6 pts

Key ablations highlight the necessity of dynamic curriculum shaping, diversity constraints, and disciplined reward gating to avoid trivialization or collapse (Kwan et al., 1 Nov 2025, Kwan et al., 29 May 2026).

6. Challenges and Future Directions

Several open research challenges remain:

• Compute efficiency: Iterative sampling, recursive self-reflection, and dense reward computation increase training cost. Dynamic data filtering reduces load but scalability across domains and model sizes requires further innovation (Xu et al., 16 Jun 2025).
• Alignment drift and robustness: Without periodic human feedback, cumulative self-improvement can drift from expected values. Hybrid schemes incorporating sparse expert evaluations or inter-agent peer review are being investigated (Anantaprayoon et al., 5 Dec 2025).
• Formal verification and compositionality: Applying OpenSIR to domains like proof discovery, scientific design, or non-mathematical reasoning requires integration with formal verifiers, execution engines, or symbolic modules for rigorous step validity (Zhang et al., 29 May 2025).
• Generalization and task transfer: Empirical studies suggest cross-domain transfer is feasible under Grad2Reward and Dynamic Alignment, but systematic quantification across unstructured open-ended tasks remains incomplete (Zhang et al., 2 Feb 2026, Anantaprayoon et al., 5 Dec 2025).
• Multi-agent evolution: Incorporating collaborative and adversarial agents—each exploring different inductive biases or reasoning strategies—promises improved diversity, resilience against local optima, and emergent meta-reasoning (Anantaprayoon et al., 5 Dec 2025, Zhang et al., 29 May 2025).

A plausible implication is that OpenSIR-style architectures may eventually serve as a scalable substrate for continual, autonomous discovery and learning—extensible to intelligence amplification, scientific research, and the development of integrative agentic systems.

• (Kwan et al., 1 Nov 2025) OpenSIR: Open-Ended Self-Improving Reasoner
• (Kwan et al., 29 May 2026) SCOPE: Self-Play via Co-Evolving Policies for Open-Ended Tasks
• (Zhang et al., 2 Feb 2026) Grad2Reward: From Sparse Judgment to Dense Rewards for Improving Open-Ended LLM Reasoning
• (Xu et al., 16 Jun 2025) Direct Reasoning Optimization: LLMs Can Reward And Refine Their Own Reasoning for Open-Ended Tasks
• (Zhang et al., 29 May 2025) Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents
• (Anantaprayoon et al., 5 Dec 2025) Dynamic Alignment for Collective Agency: Toward a Scalable Self-Improving Framework for Open-Ended LLM Alignment
• (Buehler, 2024) PRefLexOR: Preference-based Recursive Language Modeling for Exploratory Optimization of Reasoning and Agentic Thinking