Agentic Proposing: Enhancing Large Language Model Reasoning via Compositional Skill Synthesis

Abstract: Advancing complex reasoning in LLMs relies on high-quality, verifiable datasets, yet human annotation remains cost-prohibitive and difficult to scale. Current synthesis paradigms often face a recurring trade-off: maintaining structural validity typically restricts problem complexity, while relaxing constraints to increase difficulty frequently leads to inconsistent or unsolvable instances. To address this, we propose Agentic Proposing, a framework that models problem synthesis as a goal-driven sequential decision process where a specialized agent dynamically selects and composes modular reasoning skills. Through an iterative workflow of internal reflection and tool-use, we develop the Agentic-Proposer-4B using Multi-Granularity Policy Optimization (MGPO) to generate high-precision, verifiable training trajectories across mathematics, coding, and science. Empirical results demonstrate that downstream solvers trained on agent-synthesized data significantly outperform leading baselines and exhibit robust cross-domain generalization. Notably, a 30B solver trained on only 11,000 synthesized trajectories achieves a state-of-the-art 91.6% accuracy on AIME25, rivaling frontier-scale proprietary models such as GPT-5 and proving that a small volume of high-quality synthetic signals can effectively substitute for massive human-curated datasets.

• The paper introduces Agentic Proposing, a framework that enhances large language model (LLM) reasoning by treating problem synthesis as compositional logic engineering, where reasoning patterns are decomposed into atomic, executable skill modules.
• The framework employs a multi-granularity policy optimization (MGPO) approach, which includes dynamic pruning of misaligned skills and curriculum-based skill sampling, and shows a 13.6% improvement in verifier validity and a significant improvement in solver performance on contest mathematics benchmarks.
• Training results demonstrate that a 30B solver achieves state-of-the-art performance on LiveCodeBench and contest mathematics tasks, outperforming even some frontier proprietary models.

Motivation and problem statement

Reinforcement learning approaches to LLM reasoning, exemplified by DeepSeek-Math and DeepSeek-R1, depend on large volumes of high-difficulty, verifiable problems. Human annotation is expensive, and existing synthesis pipelines face a structural trade-off: template- or prior-constrained methods (MetaMath, WizardMath, DESIGNER) guarantee validity but confine generation to a narrow, pre-defined space, while relaxing constraints to raise difficulty frequently yields inconsistent or unsolvable instances. The paper's central reframing is that problem synthesis should be treated not as monolithic text generation but as compositional logic engineering: reasoning patterns are decomposed into atomic, executable skill modules that a specialized agent orchestrates under verification feedback.

Framework overview

The synthesis process is formalized as a POMDP $$(\mathcal{S}, \mathcal{A}, \mathcal{O}, P, R, \gamma)$$, with the key modeling claim that logical solvability is a latent property unobservable from surface dialogue history alone; the agent must therefore actively probe the environment via tool-use and reflection. Each skill $k$ is a structured tuple $\langle \iota, \mu, \delta, \tau \rangle$ encoding intent, construction method, difficulty effect, and tool hint. A mapping operator $\Phi$ converts composed skills into natural-language constraints conditioning the proposer policy. The action space spans cognitive actions (internal reflection $\tau^{\text{think}}$), interactive tools (sandboxed execution $\tau^{\text{exec}}$, dynamic skill pruning $\tau^{\text{edit}}$), and terminal submission. The design is justified by an emergent-compositionality lemma: RL agents can learn unseen compositions of atomic skills when rewards target the composed task.

Training proceeds in three stages:

1. Skill acquisition: a teacher model (Qwen3-235B) distills atomic skills from mixed corpora via rejection sampling against a quality threshold, forming the autonomous library $\mathcal{K}_{\text{self}}$.
2. Agentic SFT: behavioral cloning on expert trajectories exhibiting reflection, tool calls, and pruning, filtered by a verifier ensemble so only trajectories ending in valid problems are retained.
3. MGPO post-training: reinforcement learning with curriculum-based skill sampling and multi-granularity rewards.

Two mechanisms deserve emphasis. Dynamic pruning lets the agent remove misaligned skills mid-draft; this raises verifier validity from 68.7% to 82.3%, and the agent exercises it in 14.5% of trajectories. Curriculum sampling weights skill categories inversely to EMA-tracked proficiency ($p(c) \propto 1/(m_c + \epsilon)$), which empirically increased sampling of combinatorial-proof skills 3.4× and prevents collapse into a low-difficulty regime.

Multi-Granularity Policy Optimization

MGPO addresses sparse credit assignment in long synthesis chains. It builds on the KL-constrained objective's closed-form optimal policy and a zero-sum weighting property: sample weights defined as the difference between centered advantages and centered implicit rewards cancel the intractable partition function. Advantages are fused at two granularities—trajectory-level standardization of the terminal reward and stage-level standardization of process rewards within subgroups sharing the same cognitive stage indicator $\sigma_t$—combined as $k$0. Weights pass through an asymmetric $k$1 gate on the importance ratio, with $k$2 to attenuate noisy off-policy updates.

The reward structure enforces two properties worth noting: invalid problems receive exactly zero total reward, and difficulty bonuses scale with $k$3 only for solvable instances ($k$4), where $k$5 is Pass@16 estimated by an external prober pool spanning Qwen3 1.7B–30B and gpt-oss-20b, switched when stabilized accuracy drops below 30%. This targets problems at the solver's reasoning frontier—the regime where group-relative advantage signals remain informative.

Main results

All comparisons use fixed budgets of 10,000–11,000 trajectories and identical GRPO recipes for downstream solvers, isolating data quality as the variable.

4B solvers on contest mathematics. Training Qwen3-4B-Instruct-2507 on Agentic Proposer data yields an overall Mean@64 gain of +4.1 points over the zero-shot baseline (34.2% → 38.3%), including +4.5 on AIME25 and +5.5 on HMMT. Strikingly, most baselines—including MetaMath, WizardMath, NuminaMath, R-Zero, and problems generated by GPT-5.2-High, Gemini-3-Pro, Claude-4.5-Opus, and DeepSeek-V3.2—degrade performance relative to zero-shot. The paper attributes this to advantage saturation: diagnostic evaluation shows the base model already passes 85–89% of MetaMath samples, collapsing group-relative advantages toward zero and inducing entropy collapse under GRPO. This is a strong, somewhat contradictory claim relative to the premise that more reasoning data helps; it holds specifically for advanced base models trained with group-based RL at small budgets.

30B solvers. A Qwen3-30B-A3B-Thinking solver trained on 11,000 mixed math/code trajectories reaches 91.6% on AIME25 (+6.6 over zero-shot), 93.5% on AIME24, and gains of roughly +5 points on LiveCodeBench v5/v6, surpassing OpenMathReasoning, PromptCoT 2.0, and other open-source collections. The paper claims this rivals frontier proprietary models such as GPT-5 and Gemini-3-Pro while outperforming open-source models with up to 20× more parameters—an assertion that should be read as benchmark-specific (Mean@64 on contest math and Best-of-5 on LiveCodeBench) rather than a general capability claim.

Cross-domain transfer. With domain-targeted synthesis, a 4B solver improves +5.3 overall across scientific benchmarks, with the largest gains on graduate-level tasks: +7.3 on SuperGPQA and +6.3 on GPQA, plus +4.4 on OlympicArena's text-only subset. The authors interpret this as evidence that agentic synthesis fosters cross-disciplinary robustness rather than domain-specific overfitting, though no baseline comparison is reported for this table.

Ablations

Three ablations isolate component contributions, all measured as downstream 4B AIME averages:

Configuration	AIME Avg.
One-shot proposing	31.5
+ Tool-use only	33.8
+ Reflection only	33.4
Full iterative pipeline	38.3
Standard GRPO	31.8
MGPO w/o stage-level advantage	35.1
Full MGPO	38.3

Tool-use and reflection each contribute ~2 points independently, but their combination within the Draft–Check–Refine loop yields +6.8, indicating synergy rather than additivity. Stage-level advantage accounts for half of MGPO's total gain, confirming that intermediate credit assignment matters for long synthesis chains. On proposer specialization, equipping raw GPT-5.2 with the skill library and agentic workflow lifts its downstream score from 32.8 to 36.4, yet the specialized 4B proposer still exceeds it at 38.3—supporting the claim that domain-specific RL on agentic trajectories can beat frontier-scale prompting for this task. MGPO gate ablations show the asymmetric gated variant reaches 93.4% proposing accuracy versus 87.8–88.1% for symmetric or ungated variants, and sensitivity sweeps show performance degrades at extreme gate temperatures or extreme values of $k$6 (optimal at 0.5).

Verification reliability is measured against human expert annotation on 100 sampled trajectories: the full committee (three verifiers with proposer-history privilege, majority vote, plus randomized second audit) achieves 98% agreement, versus 89–91% for individual zero-shot verifiers. The sample size here is small, and the semi-rule-based acceptance criteria involve judgment calls, so residual label noise at the ~2% level propagates into all training sets.

Limitations and open questions

Several caveats bear directly on the headline results. First, the framework depends on a heavyweight external apparatus—a three-model verifier ensemble consuming up to 36k tokens per verification, a prober pool, and a 235B teacher for skill acquisition—so the "small model surpasses frontier models" claim concerns the specialized synthesis task, not end-to-end cheapness. Second, the finding that established baselines degrade downstream performance is specific to strong base models under group-relative RL at fixed small budgets; whether it persists at larger data scales or weaker initial policies is not tested. Third, the frontier-model comparisons rest on benchmark subsets (contest math, LiveCodeBench) and particular decoding protocols, leaving general capability parity unestablished. Fourth, the verifier's 98% human agreement is estimated on only 100 samples, and the difficulty-calibration mechanism assumes prober accuracy tracks the target solver's frontier, which may drift as solvers improve. Finally, the compositionality lemma motivating skill modularity is asserted rather than proven for this setting, and the paper does not measure how often novel skill compositions actually emerge during MGPO training.

Conclusion

Agentic Proposing recasts training-data synthesis as a closed-loop sequential decision process over composable reasoning skills, trained with a tailored multi-granularity RL objective. Under tightly controlled fixed-budget comparisons, its trajectories consistently improve downstream solvers where many synthetic and even frontier-generated alternatives degrade them, and a 30B solver reaches 91.6% on AIME25 from only 11,000 examples. The evidence supports the paper's core thesis—that signal precision and difficulty calibration, not dataset volume, are the binding constraint for RL-based reasoning improvement—while leaving open the scalability of the verification apparatus and the generality of the advantage-saturation diagnosis beyond the evaluated regimes.