F-TIS: Harnessing Diverse Models in Collaborative GRPO

Abstract: Reinforcement learning methods such as GRPO have seen great popularity in LLM post-training. In GRPO, models produce completions to a set of prompts, which are rewarded, and the policy is updated towards the relatively high reward completions. Due to the auto-regressive nature of models, the generation phase of such style of training can be extremely time consuming. As a solution, prior work has sought to distribute the inference step across many nodes, working parallel. These works assume primarily homogeneous models in the training in order to keep samples as close to on-policy as possible. This assumption may be impractical in decentralized systems, where parties with various computes and preferences may wish to collaborate on the same task. Thus, decentralized training requires an approach that can handle heterogeneous models - different models collaborating on the same tasks. However, this leads to highly off-policy samples presented during training, which prior work has identified that off-policy samples can hurt GRPO convergence. To enable heterogeneity, we propose Filtered Truncated Importance Sampling (F-TIS) - a GRPO-style training paradigm that can use off-policy samples to improve local model's learning. Our framework allows various models to collaborate in the same RL training run while being communication efficient. We extensively evaluate F-TIS in various heterogeneous setups and we show that it exhibits identical final model convergence to purely on-sample training. Furthermore, we observe in some setups better generalization on out-of-distribution tasks than on-policy training, increasing model's performance by up to 12\%.

• The paper introduces F-TIS, which combines truncated importance sampling with sample filtering to stabilize off-policy RL training for heterogeneous LLMs.
• Experiments show that F-TIS achieves convergence comparable to on-policy baselines across varying model sizes, expertise, and trainable parameters.
• F-TIS enhances out-of-distribution generalization, notably improving weaker model performance in collaborative decentralized RL scenarios.

F-TIS: Filtered Truncated Importance Sampling for Collaborative GRPO with Diverse Models

Context and Motivation

Post-training of LLMs via RL methods, particularly GRPO, has become standard for enhancing reasoning and alignment. However, deployment in decentralized contexts exposes substantial barriers: the assumption of homogeneous model architectures and parameterizations. Variations in model size, expertise, or trainable parameters across collaborating nodes render standard GRPO training highly off-policy, leading to degraded convergence and performance. The bottleneck of auto-regressive generation and the necessity to distribute inference steps further complicates practical decentralized RL setups, motivating the need for algorithms tolerant to heterogeneity and off-policy sampling.

Technical Contributions

GRPO and Off-Policy Heterogeneity

Traditional GRPO requires contemporaneous generation and policy update, implicitly enforcing on-policy sampling. Model drift and heterogeneity (model size, expertise, parameter subset) introduce substantial divergence between generator and trainer policies—making off-policy sampled completions unavoidable (Figure 1).

Figure 1: Validation curve for the 1.5B model in a vertical decentralized RL setup, showing degraded performance when NoIS (no importance sampling) is used for heterogeneous collaboration.

This heterogeneity leads to model collapse or poor convergence if not addressed. The paper rigorously demonstrates this through baseline experiments with models (Qwen2.5-1.5B and Qwen2.5-3B) collaborating in vertical distribution, where naive approaches yield significantly worse performance compared to isolated training.

Importance Sampling Approaches

Two main variants of importance sampling are considered:

• VIS (Vanilla Importance Sampling): Token-level importance weights communicated for every generation.
• TIS (Truncated Importance Sampling): Importance term is moved outside the token sum, with clipping applied (constant $C$). Empirically, TIS provides superior stability, especially for larger models (Figure 2).

Figure 2: Comparative validation curves for the 1.5B model under NoIS, VIS, and TIS, with TIS showing improved convergence.

Both methods incur negligible communication overhead (4 bytes/token). However, neither sufficiently prevents model collapse or unstable updates in highly off-policy regimes.

Sample Filtering

Sample filtering drops completions with negative advantage ($\hat{A}_i < 0$) or KL divergence above a threshold ($g$), restricting updates to informative samples. Experiments demonstrate that even this basic filter substantially stabilizes heterogeneous RL, achieving performance close to on-policy baselines (Figure 3).

Figure 3: Validation curve for the 1.5B model with filtering in NoIS, evidencing improved stability over pure NoIS.

F-TIS: Filtered Truncated Importance Sampling

The core contribution is the synthesis of filtering and TIS into F-TIS: only completions with $\hat{A}_i > 0$ or below KL threshold $g$ are used for update, with TIS applied to importance weights and clipping. This achieves both the numerical stability of filtering and the policy robustness of truncated sampling. The loss function combines per-token advantages, filtering logic, and clipped importance ratios.

Experiments span multiple axes of heterogeneity:

Model Size Heterogeneity

Collaborative training of Qwen2.5-1.5B and 3B (base/coder) models yields convergence indistinguishable from on-policy baselines, with slower initial progress but matched or sometimes improved final performance (Figure 4, Figure 5).

Figure 4: Validation curve for the 1.5B model in collaborative size-heterogeneous F-TIS training.

Figure 5: Validation curve for the 3B model in collaborative size-heterogeneous F-TIS training.

Model Expertise Heterogeneity

Collaborative runs with models of identical architecture but distinct expertise (base/coder) also display stable convergence, confirming F-TIS’s applicability across model parameterizations (Figure 6, Figure 7).

Figure 6: Validation curve for the 1.5B Coder model in collaborative expertise-heterogeneous F-TIS training.

Figure 7: Validation curve for the 1.5B model in collaborative expertise-heterogeneous F-TIS training.

Trainable Parameters Heterogeneity

Joint training with LoRA-adapted models confirms that F-TIS enables mutually beneficial parameter sharing, with LoRA models showing better convergence and generalization compared with isolated training (Figure 8).

Figure 8: Validation curve for the 1.5B model with trainable parameter heterogeneity in F-TIS.

Out-of-Distribution Generalization

On the MATH-500 dataset (out-of-distribution from training), F-TIS-trained models—especially the smaller 1.5B variants—exhibit substantial performance gains (up to 12% in some cases). Larger models occasionally benefit, but collaborative partners that previously underperformed in isolation consistently improve. In particular, 3B Coder models paired with smaller coders outperform their alone baseline by more than 12%, suggesting cross-model complementarities in the learning signal.

Algorithmic Analysis

Ablations over the KL threshold $g$ reveal nuanced impacts: smaller values accelerate early training for smaller models; larger values support better exploration for larger architectures (Figure 9).

Figure 9: Validation curve for the 1.5B model under different KL thresholds $g$, demonstrating impact on convergence.

Additionally, comparisons between F-TIS and F-VIS illustrate superior late-stage generalization for F-TIS, even if F-VIS converges slightly faster initially (Figure 10).

Figure 10: Validation curve for the 1.5B model comparing F-VIS and F-TIS approaches.

Practical and Theoretical Implications

F-TIS delivers practical impact by enabling efficient, communication-light, collaborative RL post-training for LLMs with heterogeneity in architecture, expertise, and trainable parameters. The framework requires only token/logprob communication, making it scalable for resource-diverse decentralized settings, and achieves final performance matching on-policy baselines. The observed improvements on out-of-distribution datasets suggest that heterogeneous collaboration via F-TIS may enhance generalization, possibly through implicit robustness or shared exploration.

Theoretically, F-TIS unifies stability-ensuring sample selection and importance quantification, accounting for the observed off-policy collapse in standard GRPO. This formulation is expected to motivate further exploration of distributed RL algorithms for LLM post-training, as well as more principled study of advantage filtering and importance clipping.

Limitations and Future Directions

F-TIS shows slower initial convergence relative to purely on-policy setups and its improvements on generalization are partner-dependent: the weaker model almost always benefits, but the stronger partner occasionally regresses. There is a degradation of performance in the horizontal collaboration regime, where advantage calculation across the swarm’s mean yields unwanted bias, most notably for the larger model (Figure 11).

Figure 11: Degradation in validation curve for the 1.5B model under horizontal collaborative F-TIS training.

Further research is needed to optimize group advantage calculation and bias mitigation in horizontal settings, as well as to extend F-TIS for federated and privacy-preserving scenarios. There is also room to generalize beyond group advantage to more complex reward functional forms, and integrate adaptive filtering or dynamic importance clipping schedules.

Conclusion

F-TIS introduces a principled, communication-efficient algorithm for collaborative GRPO-style RL training with heterogeneous LLMs. Across model size, expertise, and parameter subset heterogeneity, F-TIS achieves convergence matching on-policy training, enables improved out-of-distribution generalization for weaker models, and demonstrates robustness to off-policy sampling. These findings establish F-TIS as a foundational paradigm for distributed, collaborative post-training of LLMs, opening the door to scalable multi-party RL in real-world decentralized environments (2605.22537).