Self-Improvement in Large Language Models

• Large language models are systems that autonomously generate, verify, and curate training data to enhance reasoning and task performance.
• They utilize iterative generation–verification loops with techniques like chain-of-thought and self-reflection to achieve significant accuracy gains.
• Empirical results show that self-improvement pipelines in models over 7B parameters can outperform traditional human-labeled fine-tuning methods.

LLMs can self-improve by autonomously generating, verifying, and curating their own training data, thereby enhancing their reasoning and task capabilities beyond what is achievable with static human-labeled datasets. Unlike traditional fine-tuning or reinforcement learning from human feedback (RLHF), self-improvement approaches leverage the inherent ability of sufficiently large LLMs to evaluate, reflect on, and refine their own outputs or even invent optimization algorithms for their own advancement. This paradigm encompasses supervised, reinforcement learning, preference optimization, and meta-skill frameworks, and is characterized by iterative closed-loop pipelines in both unimodal and multimodal domains. Empirical evidence demonstrates absolute accuracy and capability gains on reasoning, translation, agentic, and web-based tasks—sometimes matching or exceeding performance achieved via external labels or reward models—while also revealing structural dependencies on model scale, verification capacity, and data quality control.

1. Principles and Theoretical Foundations

LLM self-improvement rests fundamentally on a sampled generation–verification–distillation loop. This paradigm proceeds as follows: (i) Generation—sample diverse candidate responses for each prompt; (ii) Verification—score each candidate using the model itself (or a frozen copy), employing metrics such as majority vote, model-consistency, or chain-of-thought rating; (iii) Distillation—fine-tune the model or update its policy on the filtered/corrected samples. A definitive quantity, the generation–verification gap (GV-Gap), governs the expected utility increase: improvement is possible if the verification mechanism reliably distinguishes better candidates, yielding a positive gap between the reweighted distribution’s performance and the model’s baseline (Song et al., 2024).

Mathematically, for a model $f$, its expected utility, $J(f) = \mathbb{E}_{x,y} [u(x, y)]$, increases via distillation of $f[w(u_f)]$—the $f$-weighted distribution where weights $w$ are a function of proxy utility scores. If the gap $gap(f) = J(f[w(u_f)]) - J(f) > \epsilon_{update}$ (update error), net improvement is achieved. The gap scales linearly with $\log(\text{FLOPs})$ in large models under stable verification schemes, but remains nonpositive for small (≤1B parameter) models or for tasks where verification is as hard as generation (Song et al., 2024).

2. Core Methodologies and Pipelines

A diverse range of self-improvement frameworks has been instantiated for both general LLMs and task-specialized systems. The most widely studied methodologies include:

• Chain-of-Thought (CoT) Prompting + Self-Consistency: Iteratively prompt LLMs for multiple CoT rationales per input; select majority or high-confidence rationales as pseudo-labels; fine-tune on these to boost reasoning accuracy. This protocol, first formalized as LMSI (Large Model Self-Improvement), yields state-of-the-art in multi-step/mathematical reasoning without any ground-truth labels, improving 74.4%→82.1% on GSM8K and 90.0%→94.4% on OpenBookQA (Huang et al., 2022).
• Self-Judging and Reinforcement Learning: Models serve both as agent and judge, generating solutions and providing binary or scalar rewards (e.g., correctness or 1–10 ratings) to their own outputs. Rewards derived from self-evaluation are used in REINFORCE or PPO loops, as in SIRLC (Self-Improvement by Reinforcement Learning Contemplation) (Pang et al., 2023) and Self Rewarding Self Improving (Simonds et al., 12 May 2025), resulting in significant gains—such as an 8% gain in integration tasks with Qwen 2.5 7B and a 5.6% absolute increase in BigBench-Hard answering accuracy.
• Self-Reflection and Iterated Refinement: Frameworks such as SELF (Self-Evolution with Language Feedback) train models to generate critiques of their own outputs and subsequently refine them, using these improved responses to further fine-tune the base model. The iterative loop increases both stepwise and final accuracy, with each round typically adding 1–2% absolute gain in math and general tasks (Lu et al., 2023).
• Autonomous Data Engineering and Continuous Self-Evolution: In the LANCE paradigm, LLMs act as fully autonomous data engineers, continually generating, scoring, diversifying, cleaning, and ingesting their own data—reducing dependency on external annotators or reward models and boosting performance by up to 3.36 absolute points across multiple benchmarks (Wang et al., 2024).
• Importance Weighting and Distributional Filtering: Importance weighting identifies and discards self-generated samples with high distribution shift from a small valid set, mitigating self-reinforcing model collapse and outperforming standard LMSI filtering (Jiang et al., 2024).
• Preference Optimization and DPO: Self-improvement pipelines in both text and multimodal domains often combine supervised fine-tuning on high-quality self-generated samples with Direct Preference Optimization (DPO) on model-scored preference pairs, maximizing the model’s preference-aligned likelihood (Wang et al., 2024, Deng et al., 2024).

3. Empirical Results and Scaling Behavior

Controlled experiments confirm robust self-improvement effects across reasoning, generation, translation, agentic action, and vision–language tasks:

Model/Method	Domain	Baseline	Post Self-Improvement	Gain	Reference
PaLM-540B (LMSI)	Reasoning	74.4%	82.1% (GSM8K)	+7.7%	(Huang et al., 2022)
Qwen2-7B (LANCE)	Multitask	61.42	64.78	+3.36	(Wang et al., 2024)
SIRLC (Flan-T5-Large)	Reasoning	31.6%	37.2%	+5.6%	(Pang et al., 2023)
Qwen 2.5 7B (SR-SI, IntBee)	Calculus	35%	43%	+8%	(Simonds et al., 12 May 2025)
R³V (Qwen-VL, vision-language)	VQA Reasoning	48.47%	64.37%	+32.8% rel	(Cheng et al., 2024)
TriPosT (LLaMA-7B, Date)	Math Reasoning	29.9%	37.0%	+7.1%	(Yu et al., 2023)
Blosom (Llama-3.1-8B, MuSiQue)	Long-context QA	50.8	55.0	+4.2	(Li et al., 2024)

Scaling analyses show that positive relative generation–verification gap emerges only for larger models (≥7B–13B), and that self-improvement saturates after 2–3 rounds of iteration (Song et al., 2024). Notably, smaller models acquire self-refinement capability only if scaffolded by interactive demonstrations from stronger LMs (Yu et al., 2023).

4. Modalities, Applications, and Automation

Self-improvement methodologies span both text-only and multimodal LLMs (MLLMs):

• Computer Agents: LLMs, bootstrapping from no tools, can self-augment by generating Python/bash software to expand their operating capabilities (retrieval, web navigation, editing), thus recursively widening their solution space for arbitrary computer tasks (Sheng, 2024).
• Web and Vision-Language Agents: Models fine-tuned on self-generated/filtered trajectories or vision–language preference pairs outperform purely supervised or RLHF alternatives in web navigation and VQA tasks, with agentic models such as Qwen-1.5-72B showing a 31% lift in WebArena task completion (Patel et al., 2024) and R³V boosting vision-language reasoning by 23–60% (Cheng et al., 2024).
• Preference-Based/Judge-Free Optimizations: New techniques in MLLMs eliminate expensive, computationally heavy model-based judgment loops, using lightweight contrastive embedding-based filters to assemble high-quality preference data with significant reductions in hallucination and cost (Deng et al., 2024).
• Algorithm Discovery and Meta-Optimization: Recent research demonstrates that LLMs can generate and refine executable code for new self-improvement algorithms, achieving >6% absolute accuracy gains on mathematical reasoning by inventing model-merge strategies superior to human-designed approaches (Ishibashi et al., 2024).
• Autonomous Data Generation in Math Reasoning: Fully unsupervised “Crescent” frameworks show that LLMs can generate diverse, deduplicated math problem sets and solutions—improving both self and distilled smaller models with no seed data or external reward models (Sun et al., 19 Feb 2025).

5. Challenges, Limitations, and Model Dependencies

While the self-improvement paradigm is empirically successful, key limitations are apparent:

• Verification Reliability: The capacity to self-improve depends crucially on the informativeness and reliability of the self-verification mechanism. In tasks where verification is as hard as generation, or where the verifier is systematically overoptimistic (e.g., failure to identify invalid plans in classical planning), closed self-refinement loops can stagnate or degrade (Valmeekam et al., 2023, Song et al., 2024).
• Scale Threshold: Small models (e.g. ≤1B parameters) lack sufficient generation–verification gap and cannot self-improve without external assistance. Efficacy scales favorably with pretraining FLOPs and model size (Song et al., 2024).
• Drift, Collapse, and Overconfidence: Iterative loops are subject to model collapse (loss of diversity), “reward hacking,” and steadily rising overconfidence or self-bias, observable as increased expected calibration error (ECE) with successive self-improvement rounds. Post-hoc calibration is required for reliable confidence estimation (Huang et al., 3 Apr 2025).
• Quality Control and Data Filtering: Importance-weighting or DSE-filtering of self-generated data is essential. Filtering based solely on answer correctness is insufficient; removing correct-but-distribution-shifted samples yields substantial additional benefits (Jiang et al., 2024). Tiny human-written valid sets (∼5% of train) can ground such filtering without requiring full supervision.
• Plateau and Generalization: Gains typically saturate after 2–4 training loops. Out-of-task or truly open-ended cross-domain self-improvement remains a challenge, and the integration of stronger model judges, hybrid symbolic verifiers, or continually evolving data/message diversity are active research areas (Song et al., 2024, Deng et al., 3 Oct 2025).

6. Outlook and Unifying Paradigms

Sustained progress in LLM self-improvement is leading to higher autonomy in both unimodal and multimodal domains, including autonomous data engineering, algorithm discovery, and compositional reasoning. Empirical trends suggest compute, rather than annotation, is now the primary bottleneck. Future research aims toward:

• Multi-modal, multi-task, and open-ended self-evolution (Deng et al., 3 Oct 2025).
• Hybrid verification leveraging both model-based and symbolic checks.
• Meta-optimization—models iteratively improving not just their outputs but their own improvement algorithms (Ishibashi et al., 2024).
• Safe, aligned, and scalable self-improvement loops with explicit calibration, reward-hacking safeguards, and adaptive data diversity management (Song et al., 2024, Huang et al., 3 Apr 2025, Deng et al., 2024).

Self-improvement paradigms portend a shift from passive, label-bound LLM learning to truly autonomous, recursively self-improving systems with implications for continual AI advancement across a range of linguistic, reasoning, agentic, and scientific domains.