Self-Optimizing Language Models

• Self-Optimizing Language Models are a design space where models or controllers iteratively improve their own behavior through autonomous, closed-loop optimization.
• A narrow SOL instance uses a frozen transformer paired with a policy network that dynamically allocates compute per token, achieving up to 7.3% accuracy gains on benchmarks.
• SOL techniques extend to self-generated supervision, preference learning, and co-evolving evaluators, but they require careful feedback management to avoid model collapse.

A plausible unifying definition of Self-Optimizing LLMs (SOL) is a family of model-centered systems in which a LLM, or a controller attached to it, participates in improving its own behavior through closed-loop optimization over computation, prompts, data, rewards, memory, or code. In the supplied literature, the term is used in both a narrow and a broad sense. In the narrow sense, "Self-Optimizing LLMs" denotes a frozen LLM coupled to a lightweight policy network that allocates compute per token during decoding (Akhauri et al., 11 May 2026). In the broader sense, the literature on self-improving LLMs describes an automated lifecycle spanning data acquisition, data selection, model optimization, inference refinement, and autonomous evaluation, with the model itself driving each stage (Yang et al., 26 Mar 2026).

1. Conceptual scope and defining properties

The system-level account of self-improvement identifies two essential properties: autonomy, meaning that the improvement loop operates without ongoing human annotation or manual correction once deployed, and continuity, meaning that outputs or experiences from earlier stages are reused to generate stronger supervision signals for subsequent updates (Yang et al., 26 Mar 2026). Within that framework, the model is not merely the object being optimized; it is also a generator of data, a selector of training signal, a participant in evaluation, and, in some settings, an optimizer of its own inference policy.

This broader conception subsumes multiple technical patterns. Some methods optimize what computation is spent where, as in per-token compute control during decoding (Akhauri et al., 11 May 2026). Others optimize what the model learns from, by generating instructions, rationales, critiques, preferences, or rubrics and then training on those artifacts (Wang et al., 2022, Huang et al., 2022, Lu et al., 2023, Yuan et al., 2024, 2405.14103). A third group optimizes how the model is embedded in a larger system, for example through trainable memories, multi-agent workflows, or external optimization loops that preserve a frozen backbone while changing routing, prompting, or coordination policies (Li et al., 14 May 2026, Wang et al., 19 Nov 2025, Wen et al., 5 Feb 2025). A fourth line extends self-optimization down to the implementation layer, where a code-generating model modifies its own source code or where agentic optimizers target GPU kernels against hardware-derived Speed-of-Light bounds (Sheng et al., 2022, Lin et al., 19 Mar 2026).

This diversity suggests that SOL is best treated not as a single algorithm but as a design space. The common structure is recursive: the model, or a system organized around it, produces signals that later condition its own future behavior. The central technical question is therefore not whether self-optimization is possible in some abstract sense, but which feedback loops remain stable, informative, and aligned under repeated reuse.

2. Dynamic compute allocation as the narrow SOL formulation

In "Compute Where it Counts: Self Optimizing LLMs," SOL refers to a frozen decoder-only transformer paired with a small policy network that decides, at every decoding step, how much compute to spend (Akhauri et al., 11 May 2026). The base LLM parameters are never updated. Instead, the policy outputs a discrete action

$$a_t \in \{1,\dots,A\} \mapsto \big(k(a_t),\,p(a_t),\,q(a_t)\big),$$

where $k$ is the token-attention keep fraction, $p$ the structured MLP activation keep fraction, and $q$ the activation quantization bit-width. The action is broadcast uniformly across all transformer layers for that decode step, so the optimization is temporal rather than per-layer.

The controller reads the previous-step final-layer hidden state, the embedding of the current token, the embedding of the previous action, and an eight-dimensional budget/progress vector that includes progress through the episode, an effective-step flag, deviations from requested attention/pruning/precision budgets, and the budgets themselves (Akhauri et al., 11 May 2026). Training uses teacher-forced episodes and samples multiple counterfactual compute schedules for the same token path. The task reward is token log-likelihood, the compute penalty is a hinge penalty on deviation from requested episode-average budgets, and the return is discounted with $\gamma = 0.85$. Policy optimization uses group-relative policy optimization with a PPO-style clipped objective and an entropy bonus.

The three controlled efficiency axes are implemented with existing inference-time mechanisms: Quest for token-level attention sparsity, TEAL-style structured activation pruning in the MLP, and ZeroQuant-style fake quantization of MLP outputs (Akhauri et al., 11 May 2026). Episodes are finite, typically $T=16$, and may be followed by a dense KV-cache refresh. This design is motivated by KV pollution: even short bursts of compression can cause delayed perplexity spikes in later tokens, with token sparsity producing the strongest tail pollution.

Empirically, SOL improves the quality-efficiency Pareto frontier at matched budget over fixed static allocation and strong random schedule search across Llama-3.2-1B, Llama-3.2-3B, Llama-3.1-8B-Instruct, and DeepSeek-R1-Distill-Llama-8B (Akhauri et al., 11 May 2026). The paper reports that SOL discovers a better quality-efficiency pareto-front across all experiments and improves MMLU accuracy by up to 7.3% over uniform budget allocation strategies. The controller is also small relative to the base model, approximately 8.56M parameters for the 8B Llama case, and is trained at modest cost compared with pretraining or full fine-tuning.

3. Self-generated supervision, preference learning, and language feedback

A large part of the SOL literature treats self-optimization as the generation of new supervision from the model's own outputs. "Self-Instruct" bootstraps instruction-following by having GPT-3 generate instructions, inputs, and outputs, then filtering invalid or similar instances before fine-tuning the same architecture (Wang et al., 2022). The resulting synthetic corpus contains 52,445 instructions and 82,439 instances, and tuning GPT-3 on it improves Super-NaturalInstructions performance from 6.8 to 39.9 ROUGE-L, a 33.1-point absolute gain, while leaving only a 5% absolute gap behind InstructGPT-001 on a human evaluation of 252 expert-written instructions (Wang et al., 2022). This established a basic SOL pattern: the model manufactures its own task distribution, then learns from it.

"LLMs Can Self-Improve" applies a different form of self-generated supervision to reasoning tasks (Huang et al., 2022). A pretrained model samples multiple Chain-of-Thought paths on unlabeled questions, extracts final answers, uses majority-vote self-consistency as a confidence signal, and fine-tunes on rationale-augmented pseudo-labels. On PaLM-540B, this improves GSM8K from 74.4% to 82.1%, DROP from 78.2% to 83.0%, OpenBookQA from 90.0% to 94.4%, and ANLI-A3 from 63.4% to 67.9% without any ground-truth label (Huang et al., 2022). An important ablation shows that fine-tuning on reasoning is critical: answer-only pseudo-labels yield much smaller gains than rationale-inclusive training.

SELF (Self-Evolution with Language Feedback) pushes this pattern further by explicitly teaching a model to critique and refine its own responses (Lu et al., 2023). It first learns meta-skills for self-feedback and self-refinement from a synthetic corpus of tuples $(p,r,f,\hat r)$, then iteratively uses those meta-skills on unlabeled prompts to generate improved answers, filter them via self-feedback, and fine-tune again. On GSM8K, Vicuna-7B rises from 16.43% direct accuracy to 29.64% after self-evolution, and inference-time self-refinement raises it further to 31.31%; on SVAMP, the same pipeline reaches 49.40% direct and 50.20% with self-consistency (Lu et al., 2023). The paper also reports that SELF outperforms an RLHF baseline trained with a learned reward model derived from the same source data, arguing that language feedback provides a richer optimization signal.

Preference-based self-optimization forms another major branch. "Self-Rewarding LLMs" uses the model itself, via an LLM-as-a-Judge prompt, to assign rewards to candidate responses and then train with Iterative DPO (Yuan et al., 2024). Starting from Llama 2 70B, three iterations improve both instruction-following and judging ability, and the resulting model outperforms many existing systems on AlpacaEval 2.0, including Claude 2, Gemini Pro, and GPT-4 0613 (Yuan et al., 2024). "Online Self-Preferring LLMs" replaces binary preferences with self-judged preference strengths and optimizes a soft-preference cross-entropy objective on online self-generated pairs (2405.14103). The paper reports state-of-the-art alignment performance across two human preference datasets, stronger robustness than RLHF when offline data are limited, and successful self-improvement without external supervision once the base model is sufficiently proficient in self-preferring (2405.14103).

4. Co-evolving evaluators, agentic memories, and structural self-optimization

Several systems extend SOL beyond self-generated labels into co-evolving evaluators and persistent external structures. EvoLM treats a single Qwen3-8B model as both a policy and a rubric generator, while a small frozen judge scores answers under rubric-conditioned criteria (Li et al., 5 May 2026). The rubric generator produces instance-specific JSON rubrics with weighted criteria and discrete scoring levels, and is trained using preference pairs constructed entirely from the policy's own temporal history. In the reported setting, EvoLM rubrics improve RewardBench-2 from 36.6% for GPT-4.1-prompted rubrics to 46.0%, a 25.7% relative improvement, and the co-trained policy reaches 69.3% average on the OLMo3-Adapt suite, outperforming policies trained with GPT-4.1 prompted rubrics by 3.9% and with SkyWork-RM by 16% (Li et al., 5 May 2026). The key mechanism is co-evolution: as the policy improves, rubrics must become more discriminative, and sharper rubrics in turn provide more informative rewards.

Solvita exemplifies a different route: continuous self-optimization around a frozen backbone through specialized agents and trainable graph-structured memories (Li et al., 14 May 2026). The Planner, Solver, Oracle, and Hacker each pair a frozen LLM with a role-specific knowledge network updated via reinforcement signals such as pass/fail verdicts, certification quality, and adversarial break success. The framework reports new state of the art among code-generation agents, nearly doubling the accuracy of single-pass baselines in the abstract; detailed results include GPT-5.4 on CodeContests rising from 40.00% for single-pass to 82.42% for Solvita (Li et al., 14 May 2026). Here the optimized object is not the backbone weights but the routing policy over memories, tests, and attack strategies.

Other work moves the optimization target again. "Self-Programming Artificial Intelligence Using Code-Generating LLMs" lets a T5-style code generator read its own source code, emit candidate modifications, execute and evaluate them, and adopt the best variant as its new implementation (Sheng et al., 2022). The reported architecture and task changes yield MNIST accuracy improvements from about 92% to about 98%, CIFAR-10 from about 20% to about 70%, and EMNIST from about 70% to about 87% (Sheng et al., 2022). In "SOLID," self-optimization takes the form of iterative negotiation between an optimization agent and an LLM agent via dual prices and deviation penalties, retaining convergence guarantees under convexity assumptions and improving annualized returns relative to an optimizer-only portfolio baseline (Wang et al., 19 Nov 2025). SOLOMON reorganizes reasoning around Thought Generators, a Thought Assessor, and a Steering subsystem, using multi-model thought pools and execution error logs to improve code generation for semiconductor layout design; all SOLOMON instances outperform o1-preview on Basic Shapes 1, and the Claude-based SOLOMON surpasses o1-preview in three of four task categories (Wen et al., 5 Feb 2025). ORPP, finally, treats role-playing prompts as a constrained search space for prompt-level self-optimization, using iterative improvement on only 10 training questions and then transferring the resulting role design by few-shot prompting; across reasoning benchmarks, it often matches or exceeds broader prompt-optimization methods such as OPRO and SPO (Duan et al., 3 Jun 2025).

5. Failure modes, collapse, and constraints

The strongest negative result in the supplied corpus is "Collapse of Self-trained LLMs," which studies the most literal form of SOL: repeatedly training GPT-2 on its own outputs (Herel et al., 2024). The protocol is fully online with batch size 1: the model generates a sequence, performs one gradient update on that sequence, discards it, and repeats. Under this naive self-training loop, validation loss on real data rises, self-train loss on generated data falls toward zero, outputs become increasingly repetitive, and the model eventually collapses into trivial loops. The appendix traces a concrete trajectory from coherent text at iteration 0, to visible repetition by iteration 50, to heavily repetitive bullet-style text by iteration 100, and finally to a one-token loop that outputs only "-" by iteration 133 (Herel et al., 2024). Larger learning rates accelerate collapse, and larger GPT-2 variants collapse faster.

This result does not show that every self-optimizing scheme must fail, but it sharply limits one common intuition: minimizing loss on the model's own current outputs is not, by itself, a stable notion of self-improvement (Herel et al., 2024). The empirical picture is consistent with low-entropy mode collapse driven by self-reinforcing distribution shift. The paper therefore questions the viability of naive schemes in which self-generated data become the sole training target and warns that the increasing amount of artificial text on the web could create the same failure mode at ecosystem scale.

The overview literature broadens this critique into a general hazard model for self-improving systems (Yang et al., 26 Mar 2026). Reported challenges include data autophagy and model collapse under recursive synthetic training, flawed or biased feedback signals from self-judging procedures, reward hacking under proxy objectives, ineffective self-refinement when the model lacks true evaluative competence, and benchmark contamination or metric obsolescence under adaptive pressure. These concerns recur across otherwise very different methods. OSP explicitly motivates soft preference strengths as a way to reduce overfitting relative to hard binary preferences (2405.14103). SELF preserves meta-skill data across iterations to avoid forgetting (Lu et al., 2023). Dynamic-compute SOL introduces episodic control and KV refresh because approximate computation at one step can pollute future KV states (Akhauri et al., 11 May 2026). The shared lesson is that self-optimization is governed by the stability of the feedback loop, not merely by the presence of recursion.

6. Evaluation infrastructure and future directions

The broadest roadmap comes from the lifecycle view of self-improving LLMs, which argues for end-to-end systems that integrate data acquisition, data selection, model optimization, inference refinement, and autonomous evaluation, and identifies specialized domain-centric agents, unified self-improvement benchmarks, and adjustable balances between automation and human oversight as central future directions (Yang et al., 26 Mar 2026). This suggests that mature SOL systems will likely be hybrid rather than monolithic: part policy learner, part evaluator, part data engine, and part systems optimizer.

Evaluation itself has become a first-class technical problem. Dynamic benchmarks such as REALTIME-QA, FreshQA, LiveBench, LiveCodeBench, and OKBench are presented as countermeasures to temporal drift and adaptive overfitting (Yang et al., 26 Mar 2026). For low-level compute optimization, SOL-ExecBench makes this principle explicit by benchmarking 235 CUDA kernel optimization problems from 124 production and emerging AI models against analytically derived Speed-of-Light bounds on NVIDIA Blackwell GPUs (Lin et al., 19 Mar 2026). Its SOL Score measures how much of the gap between a release-defined scoring baseline and the hardware SOL bound a candidate kernel closes, and its harness includes GPU clock locking, L2 cache clearing, isolated subprocess execution, and anti-reward-hacking checks (Lin et al., 19 Mar 2026). This kind of infrastructure is likely to be increasingly important as self-optimizing systems extend from language behavior to their own execution stack.

A plausible implication is that the field is converging on a layered view of SOL rather than a single recipe. One layer optimizes inference allocation, as in per-token compute control (Akhauri et al., 11 May 2026). Another optimizes supervision generation, as in self-instruction, self-reward, self-preference, and language-feedback loops (Wang et al., 2022, Yuan et al., 2024, 2405.14103, Lu et al., 2023). Another optimizes system structure, including rubrics, memories, prompts, multi-agent topologies, and code (Li et al., 5 May 2026, Li et al., 14 May 2026, Duan et al., 3 Jun 2025, Sheng et al., 2022). The open research problem is how to compose these layers without triggering the collapse mechanisms already documented for naive recursive training (Herel et al., 2024). The current literature therefore supports both a positive and a restrictive conclusion: self-optimization is technically feasible across many axes, but stable self-optimization requires external grounding, selective feedback, and evaluation protocols that remain informative as the model itself becomes an optimizer.