RLAnything: Forge Environment, Policy, and Reward Model in Completely Dynamic RL System

Abstract: We propose RLAnything, a reinforcement learning framework that dynamically forges environment, policy, and reward models through closed-loop optimization, amplifying learning signals and strengthening the overall RL system for any LLM or agentic scenarios. Specifically, the policy is trained with integrated feedback from step-wise and outcome signals, while the reward model is jointly optimized via consistency feedback, which in turn further improves policy training. Moreover, our theory-motivated automatic environment adaptation improves training for both the reward and policy models by leveraging critic feedback from each, enabling learning from experience. Empirically, each added component consistently improves the overall system, and RLAnything yields substantial gains across various representative LLM and agentic tasks, boosting Qwen3-VL-8B-Thinking by 9.1% on OSWorld and Qwen2.5-7B-Instruct by 18.7% and 11.9% on AlfWorld and LiveBench, respectively. We also that optimized reward-model signals outperform outcomes that rely on human labels. Code: https://github.com/Gen-Verse/Open-AgentRL

• The paper presents a novel framework that jointly optimizes policy, reward model, and environment synthesis via closed-loop feedback to enhance RL agent performance.
• It integrates step-wise process rewards with outcome rewards using consistency feedback, achieving accuracy improvements up to 18.7% across multiple domains.
• The approach dynamically adjusts environment task difficulty through critic feedback, paving the way for self-evolving and scalable reinforcement learning systems.

RLAnything: A Closed-Loop Framework for Dynamic Reinforcement Learning Systems

Introduction and Motivation

The paper "RLAnything: Forge Environment, Policy, and Reward Model in Completely Dynamic RL System" (2602.02488) introduces RLAnything, a dynamic RL framework where the policy, environment, and reward model are forged in closed-loop optimization. The framework is motivated by the limitations of current RL approaches in LLM-based agents, wherein binary outcome rewards and static environments provide insufficient learning signals, especially as agentic tasks grow in complexity and require long-horizon reasoning. RLAnything aims to address these issues by integrating step-wise process rewards with outcome rewards, jointly optimizing the reward model and policy via outcome and consistency feedback, and actively adapting environment task difficulty according to policy capability.

Figure 1: The RLAnything framework motivates integration of step-wise and outcome rewards, joint optimization of the reward model and policy, and adaptive environment tasks driven by critic feedback.

The RLAnything Framework

RLAnything tightly couples the policy model $\pi_\theta$, the reward model $r_\phi$, and a suite of environment tasks $Q$ via closed-loop feedback mechanisms:

• Integrated Feedback for Policy Optimization: The policy receives a composite reward signal at each step, $$R_{\tau_i} = O_\tau + \frac{\lambda}{m} \sum_{j=1}^m S_{\tau_i, j}$$, combining verifiable outcome reward $O_\tau$ and step-wise quality signals $S_{\tau_i, j}$ from the reward model.
• Consistency Feedback for Reward Model: The reward model is optimized using $R_{S_{\tau_i, j}} = R_{\tau_i} \cdot S_{\tau_i, j}$, directly incentivizing agreement with both process-level and outcome-level supervision.
• Environment Adaptation via Critic Feedback: Environment task difficulty is dynamically adjusted using critic feedback, extracted from the reward model’s evaluations on failed steps. Threshold-based mechanisms determine when tasks are too hard or easy, prompting targeted rewrites to match policy capability.

  Figure 2: Task adaptation examples using critic feedback show dynamic environment forging across agentic domains.

The theoretical analysis demonstrates that balancing task difficulty is critical for unbiased reward estimation: if trajectory outcomes are skewed, the estimation of $p_+$ and $p_-$ in $\mu = p_+ + p_-$ becomes unbalanced, damaging reward precision. Thus, environment adaptation benefits not only policy generalization but also reward model accuracy.

Empirical Results

Comprehensive evaluations are performed on three representative agentic domains:

• GUI Agents (OSWorld): Using Qwen3-VL-8B-Thinking for policy and reward model, RLAnything achieves a 9.1% accuracy improvement on OSWorld, with a notable increase on OOD tasks.
• LLM Agents (AlfWorld): Qwen2.5-7B-Instruct (policy) and Qwen2.5-14B-Instruct (reward) show 18.7% and 11.9% accuracy gains on AlfWorld and LiveBench, respectively.
• Code Generation: Joint code and unit-test evolution driving substantial improvements in code correctness and test quality.

Figure 3: Summarized experimental results from RLAnything: consistent improvement in policy and reward metrics across domains, with linear scaling of new environment tasks and reward signal strength.

Figure 4: Each dynamic component of RLAnything (policy, reward, environment) yields monotonic improvement in policy optimization curves.

RLAnything consistently outperforms static and partial variants; all dynamic components contribute to final gains, especially in OOD generalization. Reward model efficacy (process and outcome accuracy) improves jointly with policy advancement.

Reward Model Supervision Without Human Labels

A critical empirical finding is that optimized reward-model signals, derived from closed-loop self-consistency, outperform outcome labels based on human-written evaluation scripts. This contradicts typical assumptions in RLVR, enabling scalable learning without bottlenecks in human supervision.

Figure 5: (a) Integrated reward model signals surpass outcome-only supervision. (b) Scaling learning trajectories via composite rewards for long-horizon tasks.

Environment Scaling and Active Learning

Environment adaptation through critic feedback yields a nearly linear increase in accepted tasks during training, supporting scalable synthesis and exploration. Rigorous error pattern summarization enables precise and targeted modifications, avoiding drift in task semantics.

Figure 6: (a) Linear growth in accepted tasks and stabilization of policy accuracy on new tasks. (b) Adaptive co-evolution of response length and chain-of-thought reasoning in AlfWorld.

RLAnything demonstrates active learning from experience: difficult tasks are simplified and easy tasks are perturbed for increased complexity, ensuring a curriculum matched to the current policy and reward model competency.

State-of-the-Art Results on Multimodal Agents and Coding

RLAnything establishes new state-of-the-art on OSWorld, surpassing competitive baselines including UI-TARS1.5-7B and OpenCUA-7B.

Figure 7: RLAnything-optimized agent surpasses prior state-of-the-art models on OSWorld tasks across all categories.

In coding tasks, agentic methods leveraging RLAnything-optimized coders and unit testers show substantial improvements in both code solution correctness and the discriminative power of generated test suites.

Figure 8: RLAnything drives scaling of training for GUI agents and unlocks strong agentic coding results using LiveBench evaluations.

Theoretical Implications and Future Directions

RLAnything unifies several recent advances in RL for reasoning agents, integrating concepts from process reward modeling, curriculum learning, and scalable environment synthesis. The closed-loop co-evolution of policy, reward, and environment steps beyond prior RLVR systems that lack step-wise signals or dynamic environment adaptation.

Implications include:

• Reward Model Design: The demonstrated superiority of process-based model supervision over outcome-based rewards indicates new directions for scalable RL where supervised labels (human or evaluator-based) may be replaced by strong reasoning verifiers.
• Dynamic Environment Synthesis: Automated adaptation of environment tasks, grounded in error-driven critic feedback, is crucial for scalable, robust RL agent development, facilitating both policy and reward optimization.
• Self-Evolving Agents: RLAnything lays the groundwork for fully self-improving agentic systems, wherein agents can bootstrap their own learning signals and tasks, potentially enabling open-ended generalization.

Future research may explore scaling RLAnything to multi-agent systems, further automation in environment synthesis (including programmatic and multimodal domains), and deeper analysis of reward channel integrity and specification gaming risks.

Conclusion

RLAnything introduces a formally motivated framework for joint, closed-loop optimization of environment, policy, and reward models in complex RL systems. Integrating step-wise and outcome signals, leveraging consistency feedback, and enabling environment adaptation via critic feedback result in monotonic improvements in performance and reward precision. The empirical and theoretical results confirm the necessity of dynamic, feedback-driven RL systems for robust agentic learning, with broad implications for the future of scalable and self-evolving AI agents.