---
title: Proposer–Solver Loop in Adaptive Systems
url: https://www.emergentmind.com/topics/proposer-solver-loop
type: topic
---

# Proposer–Solver Loop in Adaptive Systems

A proposer–solver loop is an iterative interaction paradigm central to a diverse set of research domains, including large language models (LLMs), multimodal self-improving systems, combinatorial optimization, and decentralized block production in blockchain protocols. This mechanism deploys two distinct roles: the Proposer, responsible for generating candidate queries, tasks, or solutions, and the Solver, which attempts to resolve these proposals through reasoning, search, or optimization processes. The loop is closed via mutual feedback, producing rewards or difficulty signals that guide both agents towards the frontier of performance, often in a fully unsupervised or self-play manner. The architecture is foundational to automated curriculum discovery, scalable self-improvement, and adaptive orchestration in agentic systems.

## 1. Formal Structure and Roles

At its core, the proposer–solver loop splits an underlying computational model or system into two agent views:

1. **Proposer** ($\pi_\phi$): Generates candidate tasks, queries, questions, or model components grounded in raw input spaces (images, text, topic specifications, etc.). For multimodal reasoning (as in EvoLMM [2511.16672]), the proposer outputs visually grounded math questions from images; in LLM self-play frameworks (PasoDoble [2511.11881], Self-Questioning Language Models [2508.03682]), it generates challenging queries or code snippets.
2. **Solver** ($\pi_\theta$): Attempts to solve, reason over, or optimize the proposer's outputs, returning answers, solution trajectories, or code completions. The solver’s response consistency, correctness, or coverage typically serves as the feedback signal.

In reinforcement learning (RL) or adversarial setups, both agents are parameterized—often as adapters atop a frozen base model—updated via policy-gradient methods with intrinsic, structure-driven rewards.

Key mathematical definitions (as in EvoLMM):

- **Empirical answer distribution:** $p(a|x,q) = (1/N) \sum_i \mathbb{1}[y_i=a]$
- **Proposer policy:** $\pi_\phi(q|x)$
- **Solver policy:** $\pi_\theta(y|x,q)$

## 2. Algorithmic Loop and Learning Procedures

The iterative loop can be formalized as:

1. Sample raw input (image $x$, topic $t$, ground-truth $y^*$, etc.)
2. Proposer outputs candidate $q$ or $x$.
3. Solver samples $N$ trajectories or answers $\{y_1, ..., y_N\}$.
4. Evaluate internal consistency (e.g., answer entropy $H(x,q)$), majority voting, or external verification (unit tests, search engine RAG).
5. Compute continuous or discrete rewards for both agents:

   - **Solver reward:** Based on answer correctness, agreement, brevity, or code unit test pass rates.
   - **Proposer reward:** Often an entropy-based band-pass (EvoLMM), majority-vote window (SQLM), inverse solver success rate (PasoDoble). Rewards peak at the “edge of competence”—neither trivial nor impossible queries.

6. Update policies via REINFORCE, PPO, or specialized curriculum RL with KL regularization, baselines, and adaptive controller mechanisms.

As a concrete example, EvoLMM leverages a frozen Qwen2.5-VL backbone with LoRA adapters for proposer and solver, each updated via self-generated rewards. Hyperparameters include $N=5$ samples per question, Gaussian band for proposer reward ($\mu_H=0.9$, $\sigma_H=0.35$), continuous solver reward softening ($\gamma=0.7$), and KL controller step size ($\eta$) [2511.16672].

## 3. Reward Shaping and Curriculum Dynamics

Reward architectures in proposer–solver loops are engineered for automated curriculum discovery and agent co-evolution:

- **Band-pass rewards:** Entropy-driven Gaussian rewards peak when solver uncertainty is neither near 0 (trivial) nor maximal (chaotic), incentivizing proposers to synthesize “just-hard-enough” tasks [2511.16672].
- **Majority-vote windows:** Proposer rewards assigned only when the solver’s responses are neither unanimous nor completely disagreeing, pushing for intermediate difficulty [2508.03682].
- **Adversarial/inverse accuracy:** In dual-play, the proposer’s reward is inversely proportional to solver accuracy, with diversity regularization to avoid repetitive collapse [2511.11881].
- **External tool feedback:** In search self-play, proposer queries must be verifiable via all search results; reward is $1 -$ solver success rate, driving adversarial escalation of query complexity [2510.18821]. RAG-based verification is crucial to prevent reward hacking.
- **Optimization feedback:** In interactive optimization, the human proposer iteratively refines the mathematical model based on solver outputs and constraint feedback, supporting explicit direct manipulation and gallery archiving [2009.03163].

The curriculum emerges dynamically: as the solver’s competence increases, the proposer must escalate task difficulty to retain non-trivial reward, resulting in continual agentic frontier advancement [2601.07055, 2510.23595].

## 4. Applications Across Domains

### LLM Self-Play and Co-evolution
Proposer–solver loops are foundational to recent unsupervised self-play RL methods that enable LLMs to improve reasoning and generalization without curated datasets:

- **EvoLMM**: Closed-loop unsupervised multimodal reasoning with continuous self-rewarding and entropy calibration (Qwen2.5-VL backbone). Delivers $\sim$3% absolute gains on ChartQA, MathVista, and MathVision benchmarks solely from raw images [2511.16672].
- **PasoDoble**: Dual-play adversarial training with clipped rewards and diversity constraints; offline decoupling for stability; up to 600 RL updates, outperforming label-dependent RLVR baselines [2511.11881].
- **Self-Questioning Language Models**: Asymmetric self-play for mathematical and programmatic reasoning, utilizing majority-vote and code unit tests for unsupervised validation [2508.03682].

### Search Agents and RAG-Driven Verification
Frameworks such as Dr. Zero and SSP build automated curricula via iterative search-query generation and solver verification:

- **Dr. Zero**: Data-free self-evolution in open-domain search, optimizing question hop groups and minimizing compute via HRPO; adversarial curriculum learning with group-level baselines [2601.07055].
- **Search Self-Play**: Co-evolution of search queries and document retrieval, RAG-verification, and competitive reward shaping [2510.18821].

### Decentralized Optimization: Ethereum PBS
In blockchain consensus, proposer–solver loops underpin block production auctions. Validators (proposers) select execution payloads from competitive builders (solvers), with economic incentives tuned for MEV extraction, decentralization, and censorship resistance [2506.18189, 2305.19037].

- **Formal loop**: Builders solve for $\pi^*$ maximizing block value $R(\pi)$, submit bids to proposers, proposers select the highest bid, and payouts enforced by committee or burn auction [2506.18189].

### Physics-Informed ML and Code Synthesis
Solver-in-the-loop is crucial in domains demanding semantic or physical correctness:

- **Turbulence closure**: Neural network closures exposed to a differentiable ODE solver, gradients propagated across many time-steps, trajectory-based loss for non-Gaussian statistics [2411.13194].
- **Logic programming**: ASP solver in-the-loop for logic puzzle encoding, using solver feedback to filter and fine-tune LLM-proposed partial programs; best-of-$N$ sampling for robust inference [2512.17093].

## 5. Empirical Results and Implementation Characteristics

Across published implementations, proposer–solver loops yield consistent gains in target metrics:

- **Multimodal math-reasoning**: EvoLMM achieves $2$–$3\%$ improvement over strong baselines using only raw images and self-generated queries [2511.16672].
- **LLM Reasoning**: SQLM boosts three-digit multiplication accuracy from $0.791$ to $0.948$ after 100 RL steps, algebra gains of $16\%$, and Codeforces programming up $7\%$ [2508.03682]; PasoDoble attains strong advances with buffer-based training and diversity regularization [2511.11881].
- **Search Agents**: Dr. Zero matches or surpasses supervised baselines on 5/7 QA tasks, HRPO yields $0.326$ EM vs. $0.320$ for GRPO, and achieves significant compute reductions [2601.07055]; SSP shows $8$–$11$ point boost over fixed-opponent setups [2510.18821].
- **Blockchain**: PBS increases median block value $2$–$3\times$, builder HHI $\sim0.17$, relay HHI up to $0.40$, but exposes tradeoffs in centralization and censorship [2305.19037].
- **Turbulence modeling**: Solver-in-the-loop closures reproduce high-order flatness up to $p=10$, scaling exponents $\xi_p$ with MSE $10^{-3}$, and unbiased energy flux statistics at $\mathcal{R}\sim10^{12}$ [2411.13194].
- **Logic code synthesis**: ASP solver-in-the-loop boosts exact match rates from $31\%$ (greedy baseline) to $66.8\%$ with best-of-$N$ and regeneration+backtracking [2512.17093].

## 6. Challenges, Controversies, and Stability Mechanisms

The efficacy of the proposer–solver loop depends on careful reward engineering, anti-hacking measures, and stability-promoting algorithmic choices:

- **Reward hacking avoidance**: Validity reward clips (PasoDoble), format checks (Dr. Zero, MAE), and RAG-verification (SSP) are enforced to prevent the proposer from exploiting ambiguous or unanswerable queries.
- **Curriculum collapse**: Diversity rewards, entropy band-pass filters, and buffer eviction of stale questions prevent degeneration into trivial or repetitive tasks.
- **Training stability**: Offline decoupling of agent updates (PasoDoble), group-level baselines (Dr. Zero), and frequency-scheduled proposer updates (SQLM) improve reward monotonicity and reduce gradient variance.
- **Centralization and censorship**: Protocol-level issues in builder concentration and relay trust remain unresolved in Ethereum PBS, as institutionalized via formal centralization and censorship metrics [2305.19037]. Burn auctions and committee smoothing have been proposed as mitigations [2506.18189].

## 7. Future Directions and Extensions

The proposer–solver loop paradigm is generalizable across data modalities, agent architectures, and dynamic environments:

- Scaling to higher-dimensional, continuous domains (e.g., full Navier–Stokes LES [2411.13194]).
- Integration with external automated verifiers (e.g., ASP solvers, physics engines, interpreters) for semantic code generation and combinatorial reasoning [2512.17093].
- Protocol-level enshrinement of decentralized block assembly and MEV redistribution in blockchain consensus mechanisms [2506.18189].
- Automated curriculum discovery for human–algorithm hybrid teams in interactive optimization [2009.03163].

A plausible implication is that proposer–solver loops will increasingly serve as a universal mechanism for autonomous self-improvement and co-evolution in agentic systems, provided that robust reward shaping and adversarial stability protocols are maintained.

Source: https://www.emergentmind.com/topics/proposer-solver-loop