RedRFT Benchmark for Reinforcement Fine-Tuning

• RedRFT Benchmark is a lightweight, modular framework for reinforcement fine-tuning red teaming, combining ClearRL and Tianshou paradigms for standardized PPO evaluation.
• It integrates a monolithic implementation with modular components to enable rapid prototyping of novel intrinsic reward schemes and constraint enforcement mechanisms.
• Empirical studies show that LoRA, KL regularization, and cross-entropy Lagrange multiplier updates enhance stability and performance in managing toxic versus valid prompt generation.

RedRFT is a lightweight, modular benchmark designed for reinforcement fine-tuning (RFT)-based red teaming of LLMs with a focus on reproducibility, stability, and implementation clarity. By integrating the compactness of single-file CleanRL implementations with the extensibility of Tianshou's modular framework, RedRFT standardizes the evaluation of policy-gradient-driven red teaming methods and enables rapid prototyping of novel intrinsic reward schemes and constraint enforcement mechanisms (Zheng et al., 4 Jun 2025).

1. Architectural Principles

RedRFT features two primary implementation paradigms. The first adopts a CleanRL-style monolithic script that encompasses all critical stages—rollout generation, advantage estimation via Generalized Advantage Estimation (GAE), buffer handling, policy/value architecture, and the full Proximal Policy Optimization (PPO) update sequence—within a single file. In each iteration, the policy $\theta$ and reference $\theta_{\text{ref}}$ generate on-policy trajectories $(s_t, a_t, r_t^E, r_t^I, s_{t+1})$, which are buffered for subsequent batched PPO updates involving the clipped surrogate, entropy, and KL divergence penalties.

The second paradigm embraces Tianshou-like modularity, decomposing core roles into specialized components: Collector, ReplayBuffer, Policy/Critic Network, IntrinsicRewardModule (for plug-and-play diversity scoring), and Learner. The data flow proceeds from trajectory generation to buffering, through modular intrinsic-reward computation, culminating in policy parameter updates. PPO is instantiated as a Learner that leverages an independent GAE module and supports flexible loss definitions, thus enabling rapid experimentation with new reward or constraint types without system-wide refactoring.

2. Core Mathematical Foundations

RedRFT formalizes RFT-based red teaming using PPO with several enhancements. The central PPO objective employs a clipped surrogate plus regularization: $$\max_{\theta} \; \sum_t \min \bigl\{ \rho_t(\theta) A_t, \mathrm{clip}(\rho_t(\theta),1-\epsilon,1+\epsilon) A_t \bigr\} - \lambda^{\mathrm{ent}} \mathbb{E}[\ln \pi'_\theta(a_t|s_t)] - \lambda^{\mathrm{KL}} D_{\mathrm{KL}}(\pi'_\theta(\cdot|s_t) \,\|\, \pi_{\mathrm{ref}}(\cdot|s_t))$$ where $$\rho_t(\theta) = \frac{\pi'_\theta(a_t|s_t)}{\pi_\theta(a_t|s_t)}$$, $A_t$ is the advantage, and $\epsilon$ is the clipping parameter.

Generalized Advantage Estimation (GAE) computes advantages as

$$\delta_t = r_t + \gamma V_\phi(s_{t+1}) - V_\phi(s_t), \quad A_t = \sum_{\ell=0}^{\infty} (\gamma \lambda)^\ell \delta_{t+\ell}$$

with total reward $r_t = r_t^E + r_t^I$ (extrinsic + intrinsic).

Intrinsic-reward metrics are modular and encompass:

• Prompt-level (sentence) diversity: The cosine-novelty bonus, computed at the final step,

$$r_{T-1}^{\mathrm{Cos}} = \sum_{\tau' \sim \text{buffer}} -\phi(\tau) \cdot \phi(\tau')$$

• State-level (token) diversity via policy cover:

$$r_t^{\mathrm{PC}} = \bigl( \rho_{\rm buffer}(s_{t+1}) d_\pi(s_{t+1}) \bigr)^{-1}$$

• Density-based diversity (evaluation-only):

$$r^{\mathrm{div}}(\tau) = -\ln(\widehat\rho_T(\tau)), \quad \widehat\rho_T(\tau) \approx \frac{k}{\kappa^d}, \; \kappa = \|\phi(\tau) - \phi^*(\tau)\|$$

A KL penalty controls deviation from the reference policy, while additional constraints, e.g., bounding gibberish cost $J_{\text{gib}} \leq \tau_{\text{gib}}$, are enforced via Lagrangian dual variables. The resulting mixed advantage for PPO becomes: $$A_t^{\mathrm{mix}} = \lambda^{\mathrm{tox}} A_t^{\mathrm{tox}} + \lambda^{\mathrm{I}} A_t^{\mathrm{I}} - \lambda^{\mathrm{gib}} A_t^{\mathrm{gib}}$$ Lagrange multipliers $\lambda$ are updated via either standard SGD or a cross-entropy loss,

$$\min_{\lambda^{\mathrm{gib}} > 0} \mathbb{E}\Bigl[-(1-y)\ln(1-\lambda^{\mathrm{gib}}) - y\ln(\lambda^{\mathrm{gib}})\Bigr], \quad y = \mathbf{1}(J_\mathrm{gib} > \tau_{\mathrm{gib}})$$

for stable constraint enforcement.

3. Intrinsic Reward Computation Pipeline

At every generation step, RedRFT orchestrates an extrinsic reward $r_t^E$ (e.g., toxicity detector output) and a cost $c_t$ (e.g., gibberish score). Complete trajectories are batched; the IntrinsicRewardModule samples buffer trajectories and computes either prompt- or state-level intrinsic rewards, which are then summed with $r_t^E$ to yield total rewards for learning.

On the learner side, GAE-recomputed advantages, the weighted mix of extrinsic and intrinsic terms, and any enabled KL/Lagrange penalties are passed into the PPO objective. This modular reward architecture facilitates plug-and-play experimentation with novel intrinsic objectives, directly supporting reproducibility and fair comparison.

4. Empirical Ablation Studies

Several ablation studies clarify the impact of core design choices:

• LoRA and KL-regularization: Comparing (a) LoRA + KL, (b) LoRA w/o KL, and (c) full fine-tune without KL, on a toxic continuation task, final cumulative toxicity × novelty scores were ≈0.40, ≈0.35, and ≈0.20, respectively. Both LoRA and KL substantially increase stable exploration.
• Lagrange multiplier update: Using standard SGD vs. the cross-entropy loss to adjust $\lambda_{\text{gib}}$, the cross-entropy approach stabilizes constraint satisfaction (reduced $J_{\text{gib}}$ oscillations), maintains gibberish below threshold, and boosts tox-diversity by ≈10–15%.
• Batch-size effects: On-policy batch sizes of 256 with PPO minibatches of 16 yield the best observed tradeoff between stability and computational efficiency.

5. Implementation Recommendations and Best Practices

RedRFT's comprehensive evaluation yields several best practices:

• State-level intrinsic rewards (policy cover) match prompt-level bonuses in efficacy while providing denser feedback.
• Lagrangian dual-based constrained policy optimization consistently improves tradeoff between producing toxic yet valid (non-gibberish) prompts compared to unconstrained PPO.
• Larger batch sizes (≥256) combined with smaller updates (minibatch=16) stabilize gradients.
• LoRA-based parameterization, when paired with modest KL regularization, prevents policy collapse and is resource-efficient.
• Cross-entropy-based λ updates avoid instability in constraint enforcement.
• Modularization (distinct Collector, Buffer, RewardModules, Learner, Lagrange updater) accelerates research iteration and enables principled benchmarking of algorithmic variants.

6. Context and Impact

RedRFT addresses a critical reproducibility and standardization gap in RFT-based red teaming: prior evaluations suffered from implementation-specific details in PPO pipelines that confounded comparison. By unifying the strengths of CleanRL and Tianshou into a single, well-documented codebase—with modular, explicit reward and constraint signal handling—RedRFT supports rigorous empirical analysis and rapid, controlled development of new red teaming primitives, drivers, and evaluation metrics. It is positioned to become a reference point for both methodological studies and fast, high-fidelity prototyping in reinforcement-driven adversarial evaluation of LLMs (Zheng et al., 4 Jun 2025).