---
title: Filtered Truncated Importance Sampling (F-TIS)
url: https://www.emergentmind.com/topics/filtered-truncated-importance-sampling-f-tis
type: topic
---

# Filtered Truncated Importance Sampling (F-TIS)

Searching arXiv for the cited paper and closely related work on truncated/filtered importance sampling in RL and Monte Carlo.
Filtered Truncated Importance Sampling (F-TIS) is a GRPO-style reinforcement learning paradigm for LLM post-training that is designed to make heterogeneous, decentralized collaboration feasible when different models generate off-policy completions for one another. Introduced in "F-TIS: Harnessing Diverse Models in Collaborative GRPO" [2605.22537], it combines truncated importance sampling with filtering based on advantage sign and KL divergence, so that local models can safely learn from completions produced by other, possibly very different, models while keeping communication efficient. In the reported experiments, F-TIS exhibits identical final model convergence to purely on-sample training across a range of heterogeneous settings, and in some setups improves out-of-distribution performance by up to 12% [2605.22537].

## 1. Problem setting and motivation

F-TIS is formulated in the context of Group Relative Policy Optimization (GRPO), a reinforcement learning method used for LLM post-training. In GRPO, a policy \(\pi_\theta\) generates a group of \(G\) completions for each prompt \(p\), each completion receives a scalar reward, and the policy is updated toward relatively high-reward completions using group-relative advantages rather than a value function. The generation phase is particularly expensive because autoregressive inference produces tokens sequentially, so distributed generation across many nodes is attractive [2605.22537].

The difficulty addressed by F-TIS arises when distributed or decentralized training uses heterogeneous models rather than homogeneous workers. The paper identifies three forms of heterogeneity: model size, model expertise, and trainable parameters. Concretely, collaborating models may differ in parameter count, domain specialization, or finetuning regime, such as LoRA adapters versus full finetuning. In this setting, samples generated by one model can be highly off-policy for another model, and the paper reports that prior work and its own experiments show that treating such data as on-policy degrades GRPO convergence and can cause collapse [2605.22537].

The motivating failure case is explicit. When Qwen2.5-1.5B and Qwen2.5-3B are trained collaboratively on GSM8k with NoIS, meaning that off-policy samples are used without importance sampling correction, both models perform substantially worse than if trained alone. This motivates a mechanism that can preserve the throughput benefits of collaborative generation while protecting convergence and final performance. F-TIS is proposed precisely for that role: it lets diverse models share experience by communicating only tokens and behavior-policy log-probabilities, approximately 8 bytes per token, rather than gradients or parameter snapshots [2605.22537].

## 2. Formal construction

The starting point is standard GRPO notation. For a prompt \(p\), the model generates completions \(a_i\), \(i=1,\dots,G\), with rewards \(r_i\). GRPO uses the group-relative advantage
\[
\hat A_i = \frac{r_i - \mu_r}{\sigma_r},
\]
where \(\mu_r\) and \(\sigma_r\) are the reward mean and standard deviation within the group. The behavior policy that generated a completion is denoted \(\pi_{\theta_{\text{gen}}}\), which need not equal the current local policy \(\pi_\theta\) [2605.22537].

F-TIS builds on truncated importance sampling by introducing a truncated per-token importance ratio,
\[
\tilde r_{i,t} =
\min\!\left(
\frac{\pi_\theta(a_{i,t}\mid p\circ a_{i,<t})}
{\pi_{\theta_{\text{gen}}}(a_{i,t}\mid p\circ a_{i,<t})},
\, C
\right),
\]
where \(C>0\) is the truncation constant. This caps extreme target-to-behavior ratios and thereby mitigates the variance explosion associated with highly off-policy trajectories. The loss then multiplies this truncated correction by a PPO-style clipped inner term, as in the TIS formulation reviewed by the paper [2605.22537].

The distinctive element is filtering. F-TIS defines a filtered per-token advantage
\[
\hat A_{t,i} =
\begin{cases}
\hat A_i, & \text{if } \hat A_i > 0 \text{ or } \mathcal D_{\text{KL}}(\pi_\theta \parallel \pi_{\theta_{\text{gen}}}) < g,\\
0, & \text{else,}
\end{cases}
\]
where \(g>0\) is a KL filtering threshold. The rule is asymmetric. Positive-advantage samples are always retained, regardless of how off-policy they are. Non-positive-advantage samples are retained only when the KL divergence between the current policy and the behavior policy is below the threshold; otherwise their advantage is set to zero, which removes their gradient contribution, although the paper notes that such samples may still be used in computing group statistics [2605.22537].

This asymmetry is central to the method’s interpretation. Good trajectories from other models are always allowed to influence learning, whereas low-reward trajectories from far-away policies are filtered out. The resulting objective therefore combines three mechanisms at once: per-token importance weighting, truncation at \(C\), and KL/advantage-based filtering. The paper presents this combination as the formal definition of Filtered Truncated Importance Sampling [2605.22537].

## 3. Decentralized collaborative GRPO workflow

The paper focuses on vertical decentralized RL. In this topology, each node owns a subset of prompts and generates the full completion group for its assigned prompts. At each iteration, each node samples prompts, uses its local policy \(\pi_{\theta_{\text{gen}}}\) to generate \(G\) completions per prompt, logs the generated tokens and their behavior-policy log-probabilities, and computes rewards through a rule-based reward mechanism. These completions and log-probabilities are then shared with collaborating nodes. The communicated payload is deliberately small: tokens plus per-token log-probabilities, approximately 8 bytes per token [2605.22537].

Advantage computation proceeds at the prompt level. For each group, the node computes \(\mu_r\), \(\sigma_r\), and the standardized group-relative advantages \(\hat A_i\). During the local update phase, each node evaluates its current-policy log-probabilities on both self-generated and received completions, computes the KL divergence between the current policy and the behavior policy on the trajectory, forms truncated importance ratios, applies the filtering rule to the advantage, and updates \(\theta\) using the resulting F-TIS-weighted GRPO loss [2605.22537].

This workflow differs from on-policy GRPO in a specific way. In on-policy GRPO, \(\pi_{\theta_{\text{gen}}}=\pi_\theta\), so importance ratios are effectively 1 and no off-policy correction is needed. F-TIS relaxes that assumption: completions may originate from any source model, their influence is adjusted by target-to-behavior ratios, those ratios are truncated, and some negative-advantage off-policy samples are filtered away. The method is therefore a decentralized off-policy correction scheme specialized to GRPO-style language-model training [2605.22537].

The paper also compares vertical and horizontal collaboration. In horizontal collaboration, all nodes contribute completions to each group, so advantages are computed relative to the swarm’s mean performance rather than a single model’s own performance. The reported result is noticeable degradation, especially for the 3B model. The paper interprets this as evidence that vertical collaboration is preferable because the advantage is computed relative to one model’s own performance distribution rather than the swarm’s mixed distribution [2605.22537].

## 4. Stability mechanisms and baseline comparisons

The main baseline taxonomy in the paper distinguishes six variants: NoIS, VIS, TIS, F-NoIS, F-VIS, and F-TIS. NoIS uses off-policy samples as if they were on-policy. VIS applies vanilla importance sampling inside the GRPO loss. TIS uses truncated importance sampling. F-NoIS adds filtering without importance sampling. F-VIS combines filtering with vanilla importance sampling. F-TIS combines filtering with truncated importance sampling [2605.22537].

The failure mode of naive off-policy collaboration is stark. In heterogeneous training with Qwen2.5-1.5B and Qwen2.5-3B on GSM8k, NoIS leads to significant underperformance of both models relative to training alone. The paper treats this as confirmation that even small divergence between generation and training policies hurts GRPO. Filtering alone already changes the picture: F-NoIS with \(g=50\) shows clear improvement over NoIS, stabilizes training, and achieves performance close to the on-policy baseline [2605.22537].

The comparison between vanilla and truncated importance sampling is also asymmetric. In a heterogeneous setting with \(C=2\), TIS and VIS behave similarly for the smaller model, but TIS is clearly superior for the larger model. When filtering is added, F-VIS shows faster early convergence but does not generalize well in later iterations, whereas F-TIS converges more slowly initially, generalizes better, and yields higher final performance. The paper therefore presents truncation as the preferable importance-weighting base even after filtering is introduced [2605.22537].

The rationale for filtering is stated directly in the paper’s qualitative analysis of unstable off-policy learning. Off-policy negative-advantage samples can “end up amplifying less probable tokens, which the model would not have produced, resulting often times in gibberish completions.” F-TIS addresses that problem by allowing a local model to benefit from others’ high-reward trajectories while protecting it from far-off-policy failures. This suggests that the method’s robustness derives less from unbiasedness in a strict Monte Carlo sense than from a deliberately chosen bias-variance trade-off adapted to autoregressive policy optimization [2605.22537].

## 5. Empirical behavior across heterogeneous settings

The empirical study uses Qwen2.5 family models and systematically varies three heterogeneity axes: size, expertise, and trainable parameters. The size experiments pair Qwen2.5-1.5B with Qwen2.5-3B, and likewise Qwen2.5-Coder-1.5B with Qwen2.5-Coder-3B. The expertise experiments pair base and coder models of the same size. The trainable-parameter experiments pair base models with and without LoRA, using LoRA dimension 128 on query and value matrices. Training is performed on GSM8k with rule-based reward \(1\) for correct answer and format and \(0\) otherwise, using vertical decentralized GRPO with group size \(12\), batch sizes around \(16\) to \(24\), \(50\) iterations, learning rate \(1\times 10^{-6}\), \(\epsilon=0.2\), and default \(C=2\), \(g=50\). Evaluation uses greedy-decoding pass@1 on GSM8k validation and MATH-500 pass@1 for out-of-distribution evaluation [2605.22537].

Across model-size heterogeneity experiments, F-TIS collaborative training yields validation curves that nearly coincide with the “alone” baselines, reaching near identical final performance. Initial convergence is slower, especially for the smaller model, but the final curves match. The same pattern is reported for model-expertise heterogeneity: for both 1.5B and 3B, collaborative curves closely follow the respective alone baselines on GSM8k [2605.22537].

The out-of-distribution results on MATH-500 are more differentiated:

| Models | Alone | F-TIS collaborative |
|---|---:|---:|
| Base 1.5B | 0.406 | 0.47 |
| Base 3B | 0.575 | 0.54 |
| Coder 1.5B | 0.41 | 0.47 |
| Coder 3B | 0.478 | 0.59 |

These values show that collaborative F-TIS often improves the weaker participant and can substantially improve the stronger participant in some coder settings, including the reported increase up to 12% on out-of-distribution tasks [2605.22537].

The trainable-parameters experiments show a related effect. F-TIS is used to train full finetuning and LoRA variants together, and the GSM8k validation curves indicate that, for 3B, the LoRA model benefits substantially when trained collaboratively with the base model. On MATH-500, 3B PEFT alone scores \(0.5\), whereas 3B PEFT + 3B Base via F-TIS scores \(0.56\), a 7% improvement. The paper interprets this as evidence that LoRA GRPO training can be improved with off-policy samples from a non-PEFT model via F-TIS [2605.22537].

The ablation on the filtering threshold \(g\) further clarifies the role of the filter. The paper tests \(g\in\{5,10,50,100\}\). For 1.5B, smaller \(g\), such as \(5\), gives better initial performance because fewer off-policy negative samples are used. For 3B, higher \(g=50\) gives better final performance, and the paper attributes this to the larger model benefiting from more low-reward off-policy completions as an exploration effect. The default \(g=50\) is chosen as a compromise. This suggests that the filter is not merely a safety device; it also modulates how much off-policy exploration a model can exploit [2605.22537].

## 6. Broader context, limitations, and open directions

F-TIS is empirical rather than theorem-driven. The paper explicitly states that it does not present formal theorems or convergence proofs. Its theoretical story is heuristic: bounded importance weights reduce variance, filtering limits the effect of heavily off-policy low-value data, and the combined mechanism approximates the behavior of on-policy GRPO when policies remain close and truncation rarely activates. This suggests that F-TIS should be understood as a practically stabilized off-policy correction method rather than a formally characterized estimator [2605.22537].

Within the broader importance-sampling literature, F-TIS belongs to a family of methods that deliberately transform or cap weights to control degeneracy. "Importance sampling with transformed weights" studies clipping of large importance weights as a way to accept bias in exchange for substantial variance reduction and lower mean squared error [1702.01987]. "Pareto Smoothed Importance Sampling" replaces the largest raw ratios by expected order statistics from a fitted generalized Pareto tail and uses \(\hat{k}\) as a diagnostic of tail heaviness and estimator reliability [1507.02646]. These methods operate in Monte Carlo estimation rather than GRPO, but they articulate the same design principle: heavy-tailed or extreme importance weights are often more harmful than a controlled amount of bias [1702.01987; 1507.02646].

A more directly related RL comparison appears in "GIPO: Gaussian Importance Sampling Policy Optimization," which also frames policy optimization under stale or off-policy data as a truncated-importance-sampling problem, but replaces hard clipping with a log-ratio-based Gaussian trust weight that softly dampens extreme ratios while maintaining non-zero gradients [2603.03955]. F-TIS differs in two important respects: it is built around GRPO for LLM post-training, and its filter depends on advantage sign and KL divergence rather than a smooth ratio-only attenuation [2605.22537; 2603.03955]. A plausible implication is that F-TIS is tailored to the specific pathology of heterogeneous collaborative sequence generation, where low-reward trajectories from distant policies are especially damaging.

The limitations reported for F-TIS are concrete. It introduces extra computation per token because KL divergence and importance ratios must be computed for each completion. The overhead is mostly computational rather than communicational, since only tokens and behavior-policy log-probabilities are exchanged. Horizontal collaboration remains problematic even with F-TIS, and no formal convergence proof is provided. The paper identifies several open directions: extending the method beyond vertical collaboration, formalizing theoretical guarantees for GRPO-style off-policy training, and exploring more advanced reward models or multi-task post-training settings beyond math reasoning [2605.22537].

In that sense, F-TIS occupies a specific position in the current landscape of off-policy correction. It is not a generic Monte Carlo truncation rule, nor a purely trust-region surrogate, nor a domain-adaptation filtering mechanism. It is a GRPO-specific procedure that uses truncated importance ratios together with asymmetric KL/advantage filtering to let heterogeneous models collaborate without sacrificing the convergence behavior associated with on-policy training, and sometimes with measurable gains in out-of-distribution reasoning [2605.22537].

Source: https://www.emergentmind.com/topics/filtered-truncated-importance-sampling-f-tis