Rollout Response Recomposition (RoRecomp)

• RoRecomp is a set of techniques that transforms raw rollout streams into optimized signals for tasks like LLM reasoning, reinforcement learning, and network security.
• It employs methods such as decoupling training and inference policies, selecting extreme samples, and synthesizing response fragments to enhance decision-making.
• Empirical results show improvements in accuracy, reduced response lengths, and faster training convergence across various domains.

Searching arXiv for the cited RoRecomp-related papers and adjacent formulations. Rollout Response Recomposition (RoRecomp) is a term used in recent research for methods that alter how rollout information is produced, selected, recombined, or synthesized so that reinforcement learning, test-time search, or online decision-making can use trajectories more effectively. In LLM reasoning, it can denote separating the policy used to generate training rollouts from the policy used to produce inference-time responses and then recomposing the rollout distribution as a controlled perturbation of the inference policy (Wang et al., 17 Jan 2026). In reinforcement learning with verifiable rewards, it can denote reorganizing sampled responses into specialized batches without changing the reward function (Li et al., 30 Sep 2025). In test-time scaling, it can denote synthesizing a final response from an accumulated bank of intermediate claims, constraints, and failure patterns extracted across rollouts (Wang et al., 29 Jan 2026). Related work on vision-LLMs implements the same core idea by recombining response segments across rollouts to densify self-correction signals (Ding et al., 9 Feb 2026). In adaptive network security, the term is used for recomposing online response policies from a precomputed base policy through rollout lookahead (Hammar et al., 21 Jul 2025).

1. Semantic scope of the term

The current literature uses “RoRecomp” for several related but non-identical mechanisms. What remains common is that a raw rollout stream is not consumed in its original form; instead, it is transformed into a more useful object for optimization or control.

Context	Meaning of RoRecomp	Representative paper
LLM RL for reasoning	Decouple rollout generation from inference responses via a residual rollout policy	(Wang et al., 17 Jan 2026)
RLVR reasoning efficiency	Recompose sampled responses into priority and compensation batches	(Li et al., 30 Sep 2025)
Test-time scaling	Recompose a final response from a shared experience bank	(Wang et al., 29 Jan 2026)
VLM self-correction	Recombine response segments to create correction-specific rollouts	(Ding et al., 9 Feb 2026)
Network security control	Recompose online actions from a base policy using rollout lookahead	(Hammar et al., 21 Jul 2025)

This diversity matters because RoRecomp is not tied to a single loss, architecture, or domain. In some papers it is an architectural principle, in others a data-selection rule, a synthesis procedure, or an online control mechanism. The shared technical objective is to improve the quality of the signal extracted from rollouts: exploration signal in reasoning RL, efficiency signal in RLVR, reusable partial knowledge in search, self-correction signal in VLMs, or adaptive control signal in partially observed security environments.

2. Policy-level recomposition in R$^2$PO

In "R$^2$PO: Decoupling Training Trajectories from Inference Responses for LLM Reasoning" (Wang et al., 17 Jan 2026), RoRecomp is the design principle of separating the policy used to generate training rollouts from the policy used to produce inference-time responses, then recomposing the rollout distribution as a controlled perturbation of the inference policy. The motivation is an exploration–exploitation tension in single-head PPO or GRPO: the same parameters must both generate stable, concise, correct inference responses and produce diverse training trajectories that expose alternative reasoning paths, including useful errors. The paper argues that this suppresses exploration, leading to repetitive trajectories and reduced reasoning gains.

R$^2$PO operationalizes this principle by adding a lightweight Residual Rollout-Head atop the policy. The base LM head and the residual head both read the shared backbone hidden states $H$, and the two policies are

$$\begin{aligned} \pi_{\theta} &= \mathrm{Softmax}(f_{\mathrm{LM}}(H)), \ \pi_{\phi} &= \mathrm{Softmax}(f_{\mathrm{RO}}(H)+f_{\mathrm{LM}}(H)). \end{aligned}$$

Here $\pi_\theta$ is the base policy used for inference, while $\pi_\phi$ is the rollout policy used during training. The residual head is a two-layer MLP that outputs a vocabulary-sized logit offset. Zero-initialization of $f_{\mathrm{RO}}$ ensures that $\pi_\phi$ initially matches $\pi_\theta$, avoiding cold-start instability. During training rollouts, responses are sampled from $^2$0; during inference and deployment, only $^2$1 is used and the residual head is discarded.

Optimization is organized as an iterative two-stage procedure under GRPO. In Stage 1, only $^2$2 is optimized, while the backbone and LM head are frozen; rollouts are sampled from $^2$3 and optimized with an exploration reward, GIF by default. In Stage 2, $^2$4 is frozen and $^2$5 is updated using trajectories still sampled from $^2$6, but now scored with correctness and formatting rewards. The reward used for main policy optimization is

$^2$7

Stage 1 uses Group Inverse-Frequency exploration reward, where within each group of $^2$8 rollouts trajectories with rarer reward values receive higher scores:

$^2$9

The reported control knobs are $^2$0 for the KL coefficient, $^2$1 for PPO clipping, and group size $^2$2. The backbones are Qwen2.5-3B and Qwen3-8B, trained on GSM8K or MBPP and evaluated on GSM8K, MATH-500, HumanEval, and APPS. The method achieves average accuracy gains of 3.1% on MATH-500 and 2.4% on APPS across backbones. Selected Pass@1 results include Qwen2.5-3B on MATH-500 improving from GRPO 42.20 to R$^2$3PO 45.60, and Qwen3-8B on MATH-500 improving from GRPO 54.40 to R$^2$4PO 57.20. On GSM8K for Qwen3-8B, GRPO 88.55 and R$^2$5PO 88.48 are reported as on-par (Wang et al., 17 Jan 2026).

The stability analyses are central to the method’s significance. In a perturbation experiment injecting redundant tags, GRPO error rate rose to 0.875% at step 2100, whereas R$^2$6PO maintained 0% error across 100 subsequent samples. The method also mitigated length bias: for correct answers, Base 211.61 tokens, GRPO 211.95, and R$^2$7PO 204.12; for incorrect answers, Base 281.22, GRPO 295.28, and R$^2$8PO 281.59. The residual head adds approximately 315M parameters (10.2%) on 3B and 639M (7.8%) on 8B, only during training; inference latency is unchanged because the residual head is detached (Wang et al., 17 Jan 2026).

3. Batch recomposition for concise reasoning and efficient exploration

In "RoRecomp: Enhancing Reasoning Efficiency via Rollout Response Recomposition in Reinforcement Learning" (Li et al., 30 Sep 2025), RoRecomp is a plug-and-play method for RL with verifiable rewards that improves reasoning efficiency by reorganizing rollout responses into two specialized batches—priority and compensation—without changing the reward function. The stated problem is that outcome-only rewards do not penalize intermediate reasoning steps, so models are incentivized to “think longer” or use more tools until they chance upon a correct outcome. The paper also reports that in small rollout groups, response lengths vary greatly; for DeepSeek-R1 on AIME24, the average discrepancy between the longest and shortest responses for the same problem is 8.3k tokens.

The priority batch isolates the most informative extremes. For each prompt, responses are partitioned into correct and incorrect sets, and the method selects the shortest $^2$9 fraction from $H$0 and the longest $H$1 fraction from $H$2:

$H$3

The remaining responses are stored in a replay buffer. Compensation batches are then formed from the replay buffer to prevent collapse and distribution drift. Their frequency decays according to

$H$4

with lower bound $H$5.

The paper’s interpretation is that short-correct responses carry a clear positive signal for brevity with correctness, while long-incorrect responses carry a clear negative signal against verbose failure. Compensation batches maintain broader exposure to the rollout distribution. This suggests that RoRecomp in this setting is a variance-control device as much as an efficiency device: it changes the distribution of samples seen during updates so that the policy gradient receives less ambiguous signals, while avoiding training exclusively on extremes.

The reported empirical results span three settings. In zero RL training with Qwen2.5-7B, average response length is reduced from 997 to 721 tokens, a 27.7% reduction, while accuracy changes from 45.9% to 45.5%. In agentic RL, average F1 improves from 51.5% to 52.2% while tool calls per trajectory drop from 6.2 to 3.3, a 46.8% reduction. In thinking compression, RoRecomp produces up to 52.5% length reduction: for DeepSeek-R1-Distill-Qwen 1.5B with GRPO, average length drops from 4408 to 2095 while accuracy changes from 49.1% to 48.2%. The method is also compared against ThinkPrune, ConciseRL, AdaR1, and length-penalty reward shaping, with RoRecomp reported as achieving shorter responses with equal or better accuracy in the cited examples (Li et al., 30 Sep 2025).

Ablations make two points explicit. First, the default priority fraction $H$6 is a balance point: $H$7 gives the shortest outputs but lowest accuracy, while $H$8 includes more responses and yields higher accuracy but longer outputs. Second, removing compensation batches causes a rapid early length drop but sharp accuracy degradation after roughly 80 steps, ending about 6% below RoRecomp with compensation. The method therefore depends not only on aggressive sample selection but also on replay-based stabilization (Li et al., 30 Sep 2025).

4. Search-time recomposition from accumulated experience

In "Do Not Waste Your Rollouts: Recycling Search Experience for Efficient Test-Time Scaling" (Wang et al., 29 Jan 2026), RoRecomp appears at the synthesis layer of Recycling Search Experience (RSE). Test-Time Scaling is described as improving reasoning by allocating more inference-time compute to explore the solution space, but standard search strategies are criticized for treating rollouts as disposable. RSE makes search cumulative by distilling rollouts into a shared Experience Bank. RoRecomp then recomposes a final response from accumulated intermediate claims, constraints, partial derivations, and failure patterns.

The Experience Bank contains a positive repository $H$9 for validated intermediate conclusions and a negative repository $$\begin{aligned} \pi_{\theta} &= \mathrm{Softmax}(f_{\mathrm{LM}}(H)), \ \pi_{\phi} &= \mathrm{Softmax}(f_{\mathrm{RO}}(H)+f_{\mathrm{LM}}(H)). \end{aligned}$$0 for structured pitfalls. Distillation enforces independent verifiability, explicit conditions, atomicity, and self-contained statements. Semantic deduplication uses an encoder, all-MiniLM-L6-v2, with cosine similarity threshold $$\begin{aligned} \pi_{\theta} &= \mathrm{Softmax}(f_{\mathrm{LM}}(H)), \ \pi_{\phi} &= \mathrm{Softmax}(f_{\mathrm{RO}}(H)+f_{\mathrm{LM}}(H)). \end{aligned}$$1, and the reported useful range is approximately $$\begin{aligned} \pi_{\theta} &= \mathrm{Softmax}(f_{\mathrm{LM}}(H)), \ \pi_{\phi} &= \mathrm{Softmax}(f_{\mathrm{RO}}(H)+f_{\mathrm{LM}}(H)). \end{aligned}$$2 to $$\begin{aligned} \pi_{\theta} &= \mathrm{Softmax}(f_{\mathrm{LM}}(H)), \ \pi_{\phi} &= \mathrm{Softmax}(f_{\mathrm{RO}}(H)+f_{\mathrm{LM}}(H)). \end{aligned}$$3. Positive recycling reuses verified intermediate conclusions as anchors; negative recycling reuses failure patterns and contradictions as constraints to prune dead ends.

The theoretical contribution models solutions as requiring a finite set of necessary intermediate conclusions $$\begin{aligned} \pi_{\theta} &= \mathrm{Softmax}(f_{\mathrm{LM}}(H)), \ \pi_{\phi} &= \mathrm{Softmax}(f_{\mathrm{RO}}(H)+f_{\mathrm{LM}}(H)). \end{aligned}$$4. Under perfect persistence and monotonicity, the paper states a distribution-free dominance result: for all $$\begin{aligned} \pi_{\theta} &= \mathrm{Softmax}(f_{\mathrm{LM}}(H)), \ \pi_{\phi} &= \mathrm{Softmax}(f_{\mathrm{RO}}(H)+f_{\mathrm{LM}}(H)). \end{aligned}$$5, $$\begin{aligned} \pi_{\theta} &= \mathrm{Softmax}(f_{\mathrm{LM}}(H)), \ \pi_{\phi} &= \mathrm{Softmax}(f_{\mathrm{RO}}(H)+f_{\mathrm{LM}}(H)). \end{aligned}$$6. In the independent coverage corollary,

$$\begin{aligned} \pi_{\theta} &= \mathrm{Softmax}(f_{\mathrm{LM}}(H)), \ \pi_{\phi} &= \mathrm{Softmax}(f_{\mathrm{RO}}(H)+f_{\mathrm{LM}}(H)). \end{aligned}$$7

whereas

$$\begin{aligned} \pi_{\theta} &= \mathrm{Softmax}(f_{\mathrm{LM}}(H)), \ \pi_{\phi} &= \mathrm{Softmax}(f_{\mathrm{RO}}(H)+f_{\mathrm{LM}}(H)). \end{aligned}$$8

The paper states that the baseline requires $$\begin{aligned} \pi_{\theta} &= \mathrm{Softmax}(f_{\mathrm{LM}}(H)), \ \pi_{\phi} &= \mathrm{Softmax}(f_{\mathrm{RO}}(H)+f_{\mathrm{LM}}(H)). \end{aligned}$$9 in the homogeneous case, while RSE reduces this to $\pi_\theta$0. This is the formal basis for the claim that RSE exhibits an exponential sample-complexity gap in a canonical coverage model (Wang et al., 29 Jan 2026).

Within this framework, the technical synthesis describes RoRecomp as a confidence-weighted recomposition extension. Candidate fragments are retrieved from $\pi_\theta$1, canonicalized, stitched into a proof skeleton, checked for contradictions, and filtered by negative constraints from $\pi_\theta$2. The extension proposes a generic reweighting

$\pi_\theta$3

and a confidence score

$\pi_\theta$4

Because these are presented as extensions consistent with RSE rather than as the core paper’s original formalism, they are best understood as one explicit realization of RoRecomp inside an experience-guided search pipeline.

Empirically, RSE improves Pass@1 on HMMT24, HMMT25, IMO-AnswerBench, and HLE-Math-text across Qwen3-30B-Thinking, Qwen3-4B-Thinking, Phi-4-Reasoning, and DeepSeek-V3.2. For Qwen3-30B-Thinking, HMMT24 rises from Base 57.4 to RSE Iter-3 74.4, HMMT25 from 69.0 to 83.9, and HLE-Math-text from 24.0 to 44.8. Positive-only and negative-only recycling both improve over baseline, while full RSE is slightly better than either alone. The paper also reports that PaCoRe undergoes mode collapse, with reasoning length falling from 23,309 tokens to 1,885 tokens by Iter-3 and answer entropy declining to 0.0606, whereas RSE maintains approximately 8,417 tokens and entropy 0.3319 at Iter-3 (Wang et al., 29 Jan 2026).

5. Response recomposition for self-correction in vision-LLMs

"Learning Self-Correction in Vision-LLMs via Rollout Augmentation" (Ding et al., 9 Feb 2026) does not use the exact term RoRecomp, but its technical synthesis explicitly maps the paper’s correction-specific rollouts to the same core idea. The model generates responses of the form

$\pi_\theta$5

where $\pi_\theta$6 is the pre-correction reasoning or answer, $\pi_\theta$7 is a special token signaling correction onset, and $\pi_\theta$8 is the post-correction reasoning or answer. The paper’s starting point is that effective wrong$\pi_\theta$9correct transitions are extremely rare under standard RL, at $\pi_\phi$0, and remain below $\pi_\phi$1 even under prompt-encouraged RL.

The recomposition mechanism constructs augmented trajectories by pairing $\pi_\phi$2 from one rollout with $\pi_\phi$3 from another rollout within the same input group:

$\pi_\phi$4

From $\pi_\phi$5 original rollouts this creates $\pi_\phi$6 augmented samples, categorized as wrong$\pi_\phi$7correct, correct$\pi_\phi$8correct, correct$\pi_\phi$9wrong, and wrong$f_{\mathrm{RO}}$0wrong. The paper keeps the original online rollouts and adds $f_{\mathrm{RO}}$1 augmented samples, balancing positives and negatives. In practice, $f_{\mathrm{RO}}$2 and $f_{\mathrm{RO}}$3.

Training uses GSPO rather than GRPO because the augmentation induces off-policy effects and GSPO uses sequence-level importance weighting:

$f_{\mathrm{RO}}$4

with $f_{\mathrm{RO}}$5. The method also introduces a two-stage response-masking strategy. In Stage I, all tokens in $f_{\mathrm{RO}}$6 are masked from the policy gradient, a KL regularization is applied on $f_{\mathrm{RO}}$7, and optimization focuses only on $f_{\mathrm{RO}}$8. The Stage I reward mixes a shaped correction reward and a format reward:

$f_{\mathrm{RO}}$9

In Stage II, $\pi_\phi$0 is unmasked only for samples with consistent correctness, namely correct$\pi_\phi$1correct or wrong$\pi_\phi$2wrong, while remaining masked for wrong$\pi_\phi$3correct examples.

The empirical result is Octopus-8B, built on Qwen3-VL-8B-Instruct, which reaches an average accuracy of 71.7 across seven benchmarks. This is reported as +9.5 over base Qwen3-VL-8B-Instruct, +1.2 over Qwen3-VL-8B-Thinking, and +1.0 over the strongest RLVR baseline GSPO with 16 rollouts. Per-step total time is 958.1 seconds for Octopus-8B versus 1322.8 seconds for GSPO-16, giving 0.72× training time per step. Ablations show 67.4 without Octopus augmentation and 68.6 with random augmentation, indicating that recomposition is not equivalent to merely increasing sample count (Ding et al., 9 Feb 2026).

The paper also identifies failure modes specific to recomposition. Naively pairing responses from standard RL rollouts can create out-of-distribution samples; reward shaping without masking can produce reward hacking in which the model intentionally emits a wrong $\pi_\phi$4 followed by a trivial correction; and random token-level mixing of response fragments causes training collapse. The method therefore restricts recomposition to swapping full $\pi_\phi$5 segments and couples it with masking and KL stabilization (Ding et al., 9 Feb 2026).

6. Online policy recomposition in adaptive network security

In "Adaptive Network Security Policies via Belief Aggregation and Rollout" (Hammar et al., 21 Jul 2025), RoRecomp is used in a different decision-theoretic sense. The problem is modeled as a POMDP over security states, actions, observations, and costs. Beliefs over security states are estimated through particle filtering; a scalable base policy is computed offline in an aggregated belief space; and at runtime the policy is recomposed by rollout under the current model or simulator, including changes in vulnerabilities, workloads, and configurations.

The paper’s security examples include incident-response and access-control actions such as recover or secure reset a component, deploy decoys, remove malware, throttle access, and adjust firewall rules. For incident recovery, with replicas indexed by $\pi_\phi$6 and compromise indicator $\pi_\phi$7, the stage cost penalizes intrusion persistence and unnecessary recovery:

$\pi_\phi$8

Belief updates can be computed exactly through Bayes’ rule or approximated with particle filtering, where the paper states almost sure convergence of the Monte Carlo estimate to the exact belief as the number of particles grows.

Scalability comes from feature-based aggregation. Original states are mapped to a smaller feature space, beliefs are aggregated to feature-beliefs, feature-beliefs are discretized at resolution $\pi_\phi$9, and a finite aggregated MDP is solved offline to obtain a base policy $\pi_\theta$0 and terminal value approximation $\pi_\theta$1. RoRecomp then uses rollout lookahead to choose an adapted action at the current belief:

$\pi_\theta$2

For deeper rollout, the paper uses a lookahead horizon $\pi_\theta$3 and terminal evaluation by $\pi_\theta$4 rollout simulations of length $\pi_\theta$5.

The theoretical guarantees are explicit. The aggregation error is bounded by

$\pi_\theta$6

with $\pi_\theta$7 defined by the maximal variation of $\pi_\theta$8 within aggregation cells. Under identity aggregation and resolution $\pi_\theta$9, the approximation converges to the optimal cost. The paper also states a rollout improvement result: if the terminal cost evaluation is exact, then the rollout policy improves or matches the base policy, $^2$00, together with a suboptimality bound

$^2$01

Empirically, the method is evaluated on a real testbed and on CAGE-2. On the testbed, the reported best cost is approximately 20.92 with $^2$02, compared with 49.71 for retrained PPO, and the adaptation completion measure reaches 0.98 in roughly 19 seconds, whereas PPO retraining takes more than 30 minutes. On CAGE-2 scenario 1, the method is on par with state-of-the-art baselines around cost 13.23–13.32; on scenario 2, it achieves best or statistically equivalent cost around 37.89–39.05 with adaptation around 15 seconds. In this domain, RoRecomp refers to recomposing containment, throttling, deception, reset, and analysis actions online from a base policy using current-model rollout (Hammar et al., 21 Jul 2025).

7. Shared structure, distinctions, and open issues

Taken together, these papers suggest that RoRecomp is best understood as a recurring design pattern rather than a single algorithm. The rollout stream is transformed before it drives updates or final decisions. In R$^2$03PO, the transformation is architectural: rollouts come from $^2$04 while inference uses $^2$05 (Wang et al., 17 Jan 2026). In the RLVR efficiency paper, it is dataset-level: short-correct and long-incorrect responses are over-represented and buffered through compensation batches (Li et al., 30 Sep 2025). In RSE, it is synthesis-level: fragments from multiple rollouts are accumulated and stitched into a final answer (Wang et al., 29 Jan 2026). In Octopus, it is segment-level: $^2$06 and $^2$07 are recombined across rollouts to densify self-correction supervision (Ding et al., 9 Feb 2026). In adaptive security, it is policy-level: online decisions are recomposed from an aggregated offline policy through rollout lookahead (Hammar et al., 21 Jul 2025).

The differences are as important as the commonality. Some versions of RoRecomp alter the policy class; others leave the policy unchanged and only alter training batches; others are training-free and act only at inference; and the security formulation belongs to POMDP control rather than LLM reasoning. A direct equation-level correspondence therefore does not exist across all papers. What is shared is a commitment to extracting more structured signal from rollout data than would be available from naive on-policy consumption or independent sampling.

The literature also identifies clear limits. In R$^2$08PO, behavior beyond 3B and 8B scales, especially 70B+, is unknown (Wang et al., 17 Jan 2026). In efficiency-oriented RLVR, tasks that genuinely require longer chains may suffer if short-correct responses are overemphasized, and the method depends on verifiable rewards and calibrated $^2$09 and compensation schedules (Li et al., 30 Sep 2025). In RSE, hallucinated intermediate conclusions, incorrect pruning, and noisy domains can degrade the Experience Bank, requiring validation, contradiction checks, or fallback to independent sampling (Wang et al., 29 Jan 2026). In Octopus, naive recomposition can induce distribution shift or reward hacking, so recomposition is tightly constrained by response format, masking, and KL regularization (Ding et al., 9 Feb 2026). In network security, stale simulators, overly coarse aggregation, or insufficient particles can degrade rollout quality (Hammar et al., 21 Jul 2025).

A plausible implication is that “recomposition” has become a useful research abstraction for handling a recurring failure mode of rollout-based systems: valuable information exists in trajectories, but the default pipeline either entangles it with conflicting objectives, dilutes it with noisy samples, or discards it after one use. RoRecomp methods intervene precisely at that bottleneck, whether by perturbing logits, selecting extremes, storing reusable claims, swapping response segments, or rolling out from an aggregated base policy.