Filtered Truncated Importance Sampling (F-TIS)

• Filtered Truncated Importance Sampling (F-TIS) is a decentralized RL method that combines truncated importance sampling with advantage sign and KL divergence filtering to manage heterogeneous off-policy data.
• The approach leverages GRPO-style LLM post-training by applying token-level importance ratios, truncation, and asymmetric filtering to maintain convergence similar to on-policy training.
• Empirical studies show that F-TIS matches standalone model performance and can boost out-of-distribution accuracy by up to 12% in collaborative, heterogeneous environments.

Searching arXiv for the cited paper and closely related work on truncated/filtered importance sampling in RL and Monte Carlo. Filtered Truncated Importance Sampling (F-TIS) is a GRPO-style reinforcement learning paradigm for LLM post-training that is designed to make heterogeneous, decentralized collaboration feasible when different models generate off-policy completions for one another. Introduced in "F-TIS: Harnessing Diverse Models in Collaborative GRPO" (Blagoev et al., 21 May 2026), it combines truncated importance sampling with filtering based on advantage sign and KL divergence, so that local models can safely learn from completions produced by other, possibly very different, models while keeping communication efficient. In the reported experiments, F-TIS exhibits identical final model convergence to purely on-sample training across a range of heterogeneous settings, and in some setups improves out-of-distribution performance by up to 12% (Blagoev et al., 21 May 2026).

1. Problem setting and motivation

F-TIS is formulated in the context of Group Relative Policy Optimization (GRPO), a reinforcement learning method used for LLM post-training. In GRPO, a policy $\pi_\theta$ generates a group of $G$ completions for each prompt $p$, each completion receives a scalar reward, and the policy is updated toward relatively high-reward completions using group-relative advantages rather than a value function. The generation phase is particularly expensive because autoregressive inference produces tokens sequentially, so distributed generation across many nodes is attractive (Blagoev et al., 21 May 2026).

The difficulty addressed by F-TIS arises when distributed or decentralized training uses heterogeneous models rather than homogeneous workers. The paper identifies three forms of heterogeneity: model size, model expertise, and trainable parameters. Concretely, collaborating models may differ in parameter count, domain specialization, or finetuning regime, such as LoRA adapters versus full finetuning. In this setting, samples generated by one model can be highly off-policy for another model, and the paper reports that prior work and its own experiments show that treating such data as on-policy degrades GRPO convergence and can cause collapse (Blagoev et al., 21 May 2026).

The motivating failure case is explicit. When Qwen2.5-1.5B and Qwen2.5-3B are trained collaboratively on GSM8k with NoIS, meaning that off-policy samples are used without importance sampling correction, both models perform substantially worse than if trained alone. This motivates a mechanism that can preserve the throughput benefits of collaborative generation while protecting convergence and final performance. F-TIS is proposed precisely for that role: it lets diverse models share experience by communicating only tokens and behavior-policy log-probabilities, approximately 8 bytes per token, rather than gradients or parameter snapshots (Blagoev et al., 21 May 2026).

2. Formal construction

The starting point is standard GRPO notation. For a prompt $p$, the model generates completions $a_i$, $i=1,\dots,G$, with rewards $r_i$. GRPO uses the group-relative advantage

$\hat A_i = \frac{r_i - \mu_r}{\sigma_r},$

where $\mu_r$ and $\sigma_r$ are the reward mean and standard deviation within the group. The behavior policy that generated a completion is denoted $G$0, which need not equal the current local policy $G$1 (Blagoev et al., 21 May 2026).

F-TIS builds on truncated importance sampling by introducing a truncated per-token importance ratio,

$G$2

where $G$3 is the truncation constant. This caps extreme target-to-behavior ratios and thereby mitigates the variance explosion associated with highly off-policy trajectories. The loss then multiplies this truncated correction by a PPO-style clipped inner term, as in the TIS formulation reviewed by the paper (Blagoev et al., 21 May 2026).

The distinctive element is filtering. F-TIS defines a filtered per-token advantage

$G$4

where $G$5 is a KL filtering threshold. The rule is asymmetric. Positive-advantage samples are always retained, regardless of how off-policy they are. Non-positive-advantage samples are retained only when the KL divergence between the current policy and the behavior policy is below the threshold; otherwise their advantage is set to zero, which removes their gradient contribution, although the paper notes that such samples may still be used in computing group statistics (Blagoev et al., 21 May 2026).

This asymmetry is central to the method’s interpretation. Good trajectories from other models are always allowed to influence learning, whereas low-reward trajectories from far-away policies are filtered out. The resulting objective therefore combines three mechanisms at once: per-token importance weighting, truncation at $G$6, and KL/advantage-based filtering. The paper presents this combination as the formal definition of Filtered Truncated Importance Sampling (Blagoev et al., 21 May 2026).

3. Decentralized collaborative GRPO workflow

The paper focuses on vertical decentralized RL. In this topology, each node owns a subset of prompts and generates the full completion group for its assigned prompts. At each iteration, each node samples prompts, uses its local policy $G$7 to generate $G$8 completions per prompt, logs the generated tokens and their behavior-policy log-probabilities, and computes rewards through a rule-based reward mechanism. These completions and log-probabilities are then shared with collaborating nodes. The communicated payload is deliberately small: tokens plus per-token log-probabilities, approximately 8 bytes per token (Blagoev et al., 21 May 2026).

Advantage computation proceeds at the prompt level. For each group, the node computes $G$9, $p$0, and the standardized group-relative advantages $p$1. During the local update phase, each node evaluates its current-policy log-probabilities on both self-generated and received completions, computes the KL divergence between the current policy and the behavior policy on the trajectory, forms truncated importance ratios, applies the filtering rule to the advantage, and updates $p$2 using the resulting F-TIS-weighted GRPO loss (Blagoev et al., 21 May 2026).

This workflow differs from on-policy GRPO in a specific way. In on-policy GRPO, $p$3, so importance ratios are effectively 1 and no off-policy correction is needed. F-TIS relaxes that assumption: completions may originate from any source model, their influence is adjusted by target-to-behavior ratios, those ratios are truncated, and some negative-advantage off-policy samples are filtered away. The method is therefore a decentralized off-policy correction scheme specialized to GRPO-style language-model training (Blagoev et al., 21 May 2026).

The paper also compares vertical and horizontal collaboration. In horizontal collaboration, all nodes contribute completions to each group, so advantages are computed relative to the swarm’s mean performance rather than a single model’s own performance. The reported result is noticeable degradation, especially for the 3B model. The paper interprets this as evidence that vertical collaboration is preferable because the advantage is computed relative to one model’s own performance distribution rather than the swarm’s mixed distribution (Blagoev et al., 21 May 2026).

4. Stability mechanisms and baseline comparisons

The main baseline taxonomy in the paper distinguishes six variants: NoIS, VIS, TIS, F-NoIS, F-VIS, and F-TIS. NoIS uses off-policy samples as if they were on-policy. VIS applies vanilla importance sampling inside the GRPO loss. TIS uses truncated importance sampling. F-NoIS adds filtering without importance sampling. F-VIS combines filtering with vanilla importance sampling. F-TIS combines filtering with truncated importance sampling (Blagoev et al., 21 May 2026).

The failure mode of naive off-policy collaboration is stark. In heterogeneous training with Qwen2.5-1.5B and Qwen2.5-3B on GSM8k, NoIS leads to significant underperformance of both models relative to training alone. The paper treats this as confirmation that even small divergence between generation and training policies hurts GRPO. Filtering alone already changes the picture: F-NoIS with $p$4 shows clear improvement over NoIS, stabilizes training, and achieves performance close to the on-policy baseline (Blagoev et al., 21 May 2026).

The comparison between vanilla and truncated importance sampling is also asymmetric. In a heterogeneous setting with $p$5, TIS and VIS behave similarly for the smaller model, but TIS is clearly superior for the larger model. When filtering is added, F-VIS shows faster early convergence but does not generalize well in later iterations, whereas F-TIS converges more slowly initially, generalizes better, and yields higher final performance. The paper therefore presents truncation as the preferable importance-weighting base even after filtering is introduced (Blagoev et al., 21 May 2026).

The rationale for filtering is stated directly in the paper’s qualitative analysis of unstable off-policy learning. Off-policy negative-advantage samples can “end up amplifying less probable tokens, which the model would not have produced, resulting often times in gibberish completions.” F-TIS addresses that problem by allowing a local model to benefit from others’ high-reward trajectories while protecting it from far-off-policy failures. This suggests that the method’s robustness derives less from unbiasedness in a strict Monte Carlo sense than from a deliberately chosen bias-variance trade-off adapted to autoregressive policy optimization (Blagoev et al., 21 May 2026).

5. Empirical behavior across heterogeneous settings

The empirical study uses Qwen2.5 family models and systematically varies three heterogeneity axes: size, expertise, and trainable parameters. The size experiments pair Qwen2.5-1.5B with Qwen2.5-3B, and likewise Qwen2.5-Coder-1.5B with Qwen2.5-Coder-3B. The expertise experiments pair base and coder models of the same size. The trainable-parameter experiments pair base models with and without LoRA, using LoRA dimension 128 on query and value matrices. Training is performed on GSM8k with rule-based reward $p$6 for correct answer and format and $p$7 otherwise, using vertical decentralized GRPO with group size $p$8, batch sizes around $p$9 to $p$0, $p$1 iterations, learning rate $p$2, $p$3, and default $p$4, $p$5. Evaluation uses greedy-decoding pass@1 on GSM8k validation and MATH-500 pass@1 for out-of-distribution evaluation (Blagoev et al., 21 May 2026).

Across model-size heterogeneity experiments, F-TIS collaborative training yields validation curves that nearly coincide with the “alone” baselines, reaching near identical final performance. Initial convergence is slower, especially for the smaller model, but the final curves match. The same pattern is reported for model-expertise heterogeneity: for both 1.5B and 3B, collaborative curves closely follow the respective alone baselines on GSM8k (Blagoev et al., 21 May 2026).

The out-of-distribution results on MATH-500 are more differentiated:

Models	Alone	F-TIS collaborative
Base 1.5B	0.406	0.47
Base 3B	0.575	0.54
Coder 1.5B	0.41	0.47
Coder 3B	0.478	0.59

These values show that collaborative F-TIS often improves the weaker participant and can substantially improve the stronger participant in some coder settings, including the reported increase up to 12% on out-of-distribution tasks (Blagoev et al., 21 May 2026).

The trainable-parameters experiments show a related effect. F-TIS is used to train full finetuning and LoRA variants together, and the GSM8k validation curves indicate that, for 3B, the LoRA model benefits substantially when trained collaboratively with the base model. On MATH-500, 3B PEFT alone scores $p$6, whereas 3B PEFT + 3B Base via F-TIS scores $p$7, a 7% improvement. The paper interprets this as evidence that LoRA GRPO training can be improved with off-policy samples from a non-PEFT model via F-TIS (Blagoev et al., 21 May 2026).

The ablation on the filtering threshold $p$8 further clarifies the role of the filter. The paper tests $p$9. For 1.5B, smaller $a_i$0, such as $a_i$1, gives better initial performance because fewer off-policy negative samples are used. For 3B, higher $a_i$2 gives better final performance, and the paper attributes this to the larger model benefiting from more low-reward off-policy completions as an exploration effect. The default $a_i$3 is chosen as a compromise. This suggests that the filter is not merely a safety device; it also modulates how much off-policy exploration a model can exploit (Blagoev et al., 21 May 2026).

6. Broader context, limitations, and open directions

F-TIS is empirical rather than theorem-driven. The paper explicitly states that it does not present formal theorems or convergence proofs. Its theoretical story is heuristic: bounded importance weights reduce variance, filtering limits the effect of heavily off-policy low-value data, and the combined mechanism approximates the behavior of on-policy GRPO when policies remain close and truncation rarely activates. This suggests that F-TIS should be understood as a practically stabilized off-policy correction method rather than a formally characterized estimator (Blagoev et al., 21 May 2026).

Within the broader importance-sampling literature, F-TIS belongs to a family of methods that deliberately transform or cap weights to control degeneracy. "Importance sampling with transformed weights" studies clipping of large importance weights as a way to accept bias in exchange for substantial variance reduction and lower mean squared error (Vázquez et al., 2017). "Pareto Smoothed Importance Sampling" replaces the largest raw ratios by expected order statistics from a fitted generalized Pareto tail and uses $a_i$4 as a diagnostic of tail heaviness and estimator reliability (Vehtari et al., 2015). These methods operate in Monte Carlo estimation rather than GRPO, but they articulate the same design principle: heavy-tailed or extreme importance weights are often more harmful than a controlled amount of bias (Vázquez et al., 2017, Vehtari et al., 2015).

A more directly related RL comparison appears in "GIPO: Gaussian Importance Sampling Policy Optimization," which also frames policy optimization under stale or off-policy data as a truncated-importance-sampling problem, but replaces hard clipping with a log-ratio-based Gaussian trust weight that softly dampens extreme ratios while maintaining non-zero gradients (Lu et al., 4 Mar 2026). F-TIS differs in two important respects: it is built around GRPO for LLM post-training, and its filter depends on advantage sign and KL divergence rather than a smooth ratio-only attenuation (Blagoev et al., 21 May 2026, Lu et al., 4 Mar 2026). A plausible implication is that F-TIS is tailored to the specific pathology of heterogeneous collaborative sequence generation, where low-reward trajectories from distant policies are especially damaging.

The limitations reported for F-TIS are concrete. It introduces extra computation per token because KL divergence and importance ratios must be computed for each completion. The overhead is mostly computational rather than communicational, since only tokens and behavior-policy log-probabilities are exchanged. Horizontal collaboration remains problematic even with F-TIS, and no formal convergence proof is provided. The paper identifies several open directions: extending the method beyond vertical collaboration, formalizing theoretical guarantees for GRPO-style off-policy training, and exploring more advanced reward models or multi-task post-training settings beyond math reasoning (Blagoev et al., 21 May 2026).

In that sense, F-TIS occupies a specific position in the current landscape of off-policy correction. It is not a generic Monte Carlo truncation rule, nor a purely trust-region surrogate, nor a domain-adaptation filtering mechanism. It is a GRPO-specific procedure that uses truncated importance ratios together with asymmetric KL/advantage filtering to let heterogeneous models collaborate without sacrificing the convergence behavior associated with on-policy training, and sometimes with measurable gains in out-of-distribution reasoning (Blagoev et al., 21 May 2026).