Selective Importance Sampling (SIS)

• Selective Importance Sampling (SIS) is a technique that prioritizes informative units via token-level rejection to reduce estimator variance in Monte Carlo and reinforcement learning settings.
• It integrates rejection sampling into token-level corrections for large language models, yielding improved on-policy gradient approximations and stability.
• Empirical studies show that SIS enhances performance and robustness with minimal overhead, making it effective across dense, MoE, and off-policy training regimes.

Searching arXiv for papers on Selective Importance Sampling and closely related SIS usages. to=arxiv_search.search 趣赢കം diýenjson {"^^^^^^^^11query11 Importance Sampling^^^^^^^^11\11 OR SIS importance sampling11^^^^^^^^", "^^^^^^^^11max_results11 ^^^^11\11query11^^^^, "^^^^^^^^11sort_by11 "^^^^^^^^11submittedDate11 "^^^^^^^^11sort_order11 "^^^^^^^^11descending11 to=arxiv_search.search ’winiകം 体育彩票天天json {"^^^^^^^^11query11 Importance Sampling^^^^^^^^11\11 OR abs:^^^^^^^^11\11 Importance Sampling^^^^^^^^11\11 "^^^^^^^^11max_results11 ^^^^11\11query11^^^^, "^^^^^^^^11sort_by11 "^^^^^^^^11submittedDate11 "^^^^^^^^11sort_order11 "^^^^^^^^11descending11 Selective Importance Sampling (SIS) denotes a class of importance-sampling constructions in which sampling effort or correction strength is concentrated on informative units rather than applied uniformly. In a recent explicit formulation for reinforcement-learning post-training of LLMs, SIS treats the behavior policy as a proposal distribution and performs a token-level rejection test: accepted tokens are regarded as on-policy and receive unit importance weight, while rejected tokens retain the standard importance-sampling correction (&&&^^^^11query11^^^^&&&). More broadly, closely related selective schemes appear in distributed stochastic optimization, weighted subset selection, and other task-specific estimators, although the classical importance-sampling literature does not treat “Selective Importance Sampling” as a standard standalone category; instead, it organizes the field around importance sampling, multiple importance sampling, and adaptive importance sampling (&&&^^^^11\11^^^^&&&).

^^^^11\11^^^^. Definition and conceptual scope

In the standard importance-sampling setting, a target expectation under a distribution PRESERVED_PLACEHOLDER_^^^^11query11^^^^ is evaluated using samples from a proposal PRESERVED_PLACEHOLDER_^^^^11\11^^^^, with correction by the likelihood ratio PRESERVED_PLACEHOLDER_^^^^11 OR SIS importance sampling11^^^^. The general purpose is variance reduction or feasibility when direct sampling from the target is difficult. The review literature emphasizes that efficiency depends critically on proposal choice, that self-normalized estimators are generally biased for finite PRESERVED_PLACEHOLDER_^^^^11max_results11^^^^, and that multiple- and adaptive-proposal constructions are major modern extensions (&&&^^^^11\11^^^^&&&).

Within that broader framework, the adjective “selective” identifies a more specific design principle: rather than applying the same correction mechanism to every sampled object, the method preferentially promotes, retains, or samples objects whose contribution to the target estimator is expected to be large. In the LLM post-training setting, this selectivity is literal and tokenwise. In dataset-selection settings, it appears as nonuniform example acquisition followed by inverse-probability weighting. In trajectory-prediction settings, it appears as learned omission of low-importance neighbors. These uses share the same operational motif—estimate importance, emphasize the informative subset, and compensate for the induced sampling bias when unbiasedness is required—but they do not all instantiate the same formal estimator (&&&^^^^11query11^^^^&&&, &&&^^^^11sort_by11^^^^&&&, &&&^^^^11submittedDate11^^^^&&&).

A recurrent source of confusion is terminological. The acronym “SIS” has long denoted Sequential Importance Sampling in statistics and Monte Carlo, including contingency tables, genealogical inference, Ising models, blackout simulation, and counting linear extensions &&&^^^^11sort_order11^^^^&&&); (&&&^^^^11descending11^^^^&&&); (&&&^^^^11query11^^^^&&&); (&&&^^^^11title:\11^^^^&&&); (&&&^^^^11\11query11^^^^&&&)]. More recently, “SIS” also denotes Stratified Importance Sampling in post-deployment model monitoring (&&&^^^^11\11\11^^^^&&&). Selective Importance Sampling is therefore best understood as a specific modern usage rather than the historically dominant expansion of the acronym.

^^^^11 OR SIS importance sampling11^^^^. Token-level SIS in LLM alignment

The most explicit recent formulation of Selective Importance Sampling arises in off-policy RL for autoregressive LLMs. The starting point is the standard sequence factorization

PRESERVED_PLACEHOLDER_^^^^11sort_by11^^^^

which implies the sequence-level importance ratio

PRESERVED_PLACEHOLDER_^^^^11submittedDate11^^^^

The exact off-policy gradient is

PRESERVED_PLACEHOLDER_^^^^11sort_order11^^^^

The central difficulty is that the sequence-level ratio is a product over token-level mismatches, so variance can explode or collapse on long reasoning trajectories (&&&^^^^11query11^^^^&&&).

SIS addresses this by importing the logic of rejection sampling into tokenwise off-policy correction. For each position PRESERVED_PLACEHOLDER_^^^^11descending11^^^^, it defines

PRESERVED_PLACEHOLDER_^^^^11query11^^^^

Given the sampled token PRESERVED_PLACEHOLDER_^^^^11title:\11^^^^, SIS draws an acceptance indicator

PRESERVED_PLACEHOLDER_^^^^11\11query11^^^^

If PRESERVED_PLACEHOLDER_^^^^11\11\11^^^^, the token is accepted and treated as on-policy; if PRESERVED_PLACEHOLDER_^^^^11\11 OR SIS importance sampling11^^^^, it remains off-policy. The paper states the key distributional identity

PRESERVED_PLACEHOLDER_^^^^11\11max_results11^^^^

so accepted tokens are exactly distributed as target-policy samples (&&&^^^^11query11^^^^&&&).

The resulting SIS-modified token ratio is

PRESERVED_PLACEHOLDER_^^^^11\11sort_by11^^^^

This makes SIS a mixed correction rule: accepted tokens contribute with unit weight, while rejected tokens preserve the original importance correction. Operationally, the method is a plug-in replacement for the ratio term inside tokenwise or sequencewise policy-gradient losses. For GRPO, the only substantive change is

PRESERVED_PLACEHOLDER_^^^^11\11submittedDate11^^^^

The same substitution is stated to apply to DAPO, GSPO, and other related objectives (&&&^^^^11query11^^^^&&&).

Because exact computation of PRESERVED_PLACEHOLDER_^^^^11\11sort_order11^^^^ requires a vocabulary-wide maximization, the method also introduces a top-PRESERVED_PLACEHOLDER_^^^^11\11descending11^^^^ envelope approximation: PRESERVED_PLACEHOLDER_^^^^11\11query11^^^^

PRESERVED_PLACEHOLDER_^^^^11\11title:\11^^^^

Tokens outside PRESERVED_PLACEHOLDER_^^^^11 OR SIS importance sampling11query11^^^^ remain off-policy. The accepted-token distribution then matches the target policy conditioned on membership in the top-PRESERVED_PLACEHOLDER_^^^^11 OR SIS importance sampling11\11^^^^ set, with deviation controlled by the target mass outside that set: PRESERVED_PLACEHOLDER_^^^^11 OR SIS importance sampling11 OR SIS importance sampling11^^^^

^^^^11max_results11^^^^. Theoretical properties

The principal theoretical claim for token-level SIS is not merely variance moderation in a generic sense, but a reduction in the discrepancy between tokenwise and sequencewise off-policy gradients. Let

PRESERVED_PLACEHOLDER_^^^^11 OR SIS importance sampling11max_results11^^^^

The stated bound is

PRESERVED_PLACEHOLDER_^^^^11 OR SIS importance sampling11sort_by11^^^^

where

PRESERVED_PLACEHOLDER_^^^^11 OR SIS importance sampling11submittedDate11^^^^

This identifies the cumulative log-importance deviation PRESERVED_PLACEHOLDER_^^^^11 OR SIS importance sampling11sort_order11^^^^ as the control quantity for the token/sequence approximation gap (&&&^^^^11query11^^^^&&&).

SIS contracts that deviation because accepted tokens contribute zero log-ratio. The corresponding quantity becomes

PRESERVED_PLACEHOLDER_^^^^11 OR SIS importance sampling11descending11^^^^

Whenever at least one token is accepted, the bound is tightened strictly. In this sense, SIS differs from simple ratio shrinkage: it does not merely attenuate all ratios; it converts a subset of off-policy tokens into exactly on-policy tokens under the rejection test (&&&^^^^11query11^^^^&&&).

The broader importance-sampling literature supplies a useful backdrop for this claim. In distributed SGD, the variance of the importance-sampling estimator is minimized when the proposal is proportional to the PRESERVED_PLACEHOLDER_^^^^11 OR SIS importance sampling11query11^^^^-norm of the gradient, yielding an unbiased gradient estimator with minimum covariance trace (&&&^^^^11\11descending11^^^^&&&). That result does not define SIS in the modern token-rejection sense, but it supports the same general principle: if sampling probability tracks per-sample influence, unbiased correction and variance reduction can coexist. A plausible implication is that Selective Importance Sampling occupies the intersection of proposal adaptation and estimator stabilization rather than constituting a wholly separate Monte Carlo paradigm.

^^^^11sort_by11^^^^. Empirical behavior and implementation profile

In LLM RL post-training, SIS is described as an algorithm-agnostic plug-in with negligible systems overhead. The implementation reuses old-policy logits from rollout and current-policy logits from training, requires no extra model forward pass, and adds only per-token top-PRESERVED_PLACEHOLDER_^^^^11 OR SIS importance sampling11title:\11^^^^ selection and acceptance-probability computation. The reported cost is around ^^^^11\11^^^^% wall-clock overhead per training step (&&&^^^^11query11^^^^&&&).

The empirical study covers dense and mixture-of-experts models across math and agent benchmarks. The evaluated dense models are Qwen^^^^11max_results11^^^^-^^^^11query11 and Qwen^^^^11max_results11^^^^-^^^^11\11sort_by11; the evaluated MoE model is Qwen^^^^11max_results11^^^^-^^^^11max_results11query11; Llama-^^^^11max_results11^^^^.^^^^11 OR SIS importance sampling11^^^^-^^^^11max_results11^^^^B-Instruct appears in the appendix. Benchmarks include MATH^^^^11submittedDate11query11query11^^^^, AMC^^^^11 OR SIS importance sampling11max_results11^^^^, AIME^^^^11 OR SIS importance sampling11sort_by11^^^^, AIME^^^^11 OR SIS importance sampling11submittedDate11^^^^, NQ, TriviaQA, PopQA, HotpotQA, Musique, and Bamboogle (&&&^^^^11query11^^^^&&&).

The reported gains are consistent across objectives and model families. On Qwen^^^^11max_results11^^^^-^^^^11query11, SIS improves GRPO average math accuracy from ^^^^11sort_by11title:\11^^^^.^^^^11query11submittedDate11 to ^^^^11submittedDate11 OR SIS importance sampling11^^^^.^^^^11submittedDate11title:\11^^^^ and agent average from ^^^^11sort_by11submittedDate11^^^^.^^^^11query11submittedDate11 to ^^^^11sort_by11descending11^^^^.^^^^11query11max_results11. On Qwen^^^^11max_results11^^^^-^^^^11\11sort_by11, GRPO math average goes from ^^^^11submittedDate11\11^^^^.^^^^11 OR SIS importance sampling11title:\11^^^^ to ^^^^11submittedDate11descending11^^^^.^^^^11sort_order11sort_order11. The largest reported math gain is +^^^^11sort_order11^^^^.^^^^11max_results11descending11 average points, and the strongest agent gains are around +^^^^11 OR SIS importance sampling11^^^^.^^^^11descending11^^^^ (&&&^^^^11query11^^^^&&&).

The paper also stresses robustness under difficult off-policy regimes. Three stressors are highlighted: stale rollouts reused for more gradient updates, MoE routing mismatch, and no clipping at all. Baseline methods degrade as staleness increases, whereas SIS remains better than the baseline at every staleness level. Under MoE mismatch, SIS helps prevent entropy collapse and gradient spikes; SIS + R^^^^11max_results11^^^^ is reported as strongest under routing mismatch; and SIS still functions without clipping, which is presented as evidence that the method is itself a stabilization mechanism (&&&^^^^11query11^^^^&&&).

Acceptance rates provide a direct measure of how much of the trajectory can be reclassified as effectively on-policy. They are often around ^^^^11query11^^^^.^^^^11query11 on math tasks and frequently above ^^^^11query11^^^^.^^^^11sort_order11 on agent tasks. This suggests that, in the tested regimes, a substantial fraction of tokens satisfy the token-level rejection criterion and therefore contribute without importance correction (&&&^^^^11query11^^^^&&&).

^^^^11submittedDate11^^^^. Related selective paradigms

Selective Importance Sampling has close conceptual relatives outside LLM alignment. In distributed deep learning, one set of workers can search for informative examples while a master performs parameter updates on examples chosen by importance sampling. The resulting estimator remains unbiased, and the variance-minimizing proposal is proportional to the PRESERVED_PLACEHOLDER_^^^^11max_results11query11^^^^-norm of the per-example gradient. This framework is described as essentially a practical, distributed version of Selective Importance Sampling because it selects informative examples more often rather than treating all examples equally (&&&^^^^11\11descending11^^^^&&&).

In batch subset selection, Importance Weighted Subset Selection (IWeS) chooses examples with nonuniform sampling probabilities derived from entropy or disagreement and assigns inverse-probability weights PRESERVED_PLACEHOLDER_^^^^11max_results11\11^^^^ to selected points. The weighted empirical loss is explicitly described as an unbiased sampling technique, and the paper provides generalization and sampling-rate bounds. This is best viewed as a machine-learning analogue of selective importance sampling: examples are sampled preferentially because they are informative for model improvement, but the training objective is corrected by importance weights (&&&^^^^11sort_by11^^^^&&&).

A different task-specific analogue appears in human trajectory prediction. The method called Selective Social-Interaction via Individual Importance introduces an Importance Estimator to score neighboring people and uses Gumbel Softmax so that discrete neighbor selection remains trainable. The goal is computational pruning rather than unbiased Monte Carlo estimation, and the paper explicitly distinguishes itself from classic SIS on that basis. On JRDB, the method changes ADE from PRESERVED_PLACEHOLDER_^^^^11max_results11 OR SIS importance sampling11^^^^ to PRESERVED_PLACEHOLDER_^^^^11max_results11max_results11^^^^, FDE from PRESERVED_PLACEHOLDER_^^^^11max_results11sort_by11^^^^ to PRESERVED_PLACEHOLDER_^^^^11max_results11submittedDate11^^^^, and reduces computation from about PRESERVED_PLACEHOLDER_^^^^11max_results11sort_order11^^^^G FLOPs to PRESERVED_PLACEHOLDER_^^^^11max_results11descending11^^^^G FLOPs, an ^^^^11query11^^^^.^^^^11\11 reduction, when the variance loss is used (&&&^^^^11submittedDate11^^^^&&&).

These related works indicate that “selective” importance mechanisms now appear in several ML subfields. What varies across them is the object being selected—tokens, training examples, or neighboring agents—and the statistical role of the correction. In some cases the correction preserves unbiasedness exactly; in others, selection is purely a learned computational pruning device.

^^^^11sort_order11^^^^. Terminology, distinctions, and recurrent misconceptions

The strongest historical misconception is to equate Selective Importance Sampling with the much older Sequential Importance Sampling literature. That older usage dominates statistics and combinatorial Monte Carlo, including multiway tables, zero-one contingency tables, varying-population genealogies, Ising models, power-system cascading outages, and linear-extension counting &&&^^^^11sort_order11^^^^&&&); (&&&^^^^11 OR SIS importance sampling11descending11^^^^&&&); (&&&^^^^11descending11^^^^&&&); (&&&^^^^11query11^^^^&&&); (&&&^^^^11title:\11^^^^&&&); (&&&^^^^11\11query11^^^^&&&)]. In those works, “SIS” refers to sampling stage by stage through a sequence of conditional proposals; it does not denote selective reclassification of sampled units as on-policy.

A second source of ambiguity is the newer Stratified Importance Sampling literature in model monitoring, where SIS denotes a hybrid of stratification and within-stratum importance weighting. There, the estimator is

PRESERVED_PLACEHOLDER_^^^^11max_results11query11^^^^

and the paper proves unbiasedness, consistency, and finite-sample MSE improvements over either importance sampling or stratified random sampling under stated conditions (&&&^^^^11\11\11^^^^&&&). This is a distinct construction with a different objective and a different acronym expansion.

Expansion of SIS	Domain	Representative paper
Selective Importance Sampling	Off-policy RL for LLMs	(&&&^^^^11query11^^^^&&&)
Sequential Importance Sampling	Monte Carlo over sequential state constructions	(&&&^^^^11max_results11sort_by11^^^^&&&)
Stratified Importance Sampling	Label-efficient model monitoring	(&&&^^^^11\11\11^^^^&&&)

A third misconception is to treat Selective Importance Sampling as synonymous with clipping or weight truncation. Classical IS reviews discuss truncated importance sampling, nonlinear importance sampling, and Pareto-smoothed importance sampling as ways to control extreme weights (&&&^^^^11\11^^^^&&&). By contrast, the LLM version of SIS uses a rejection-sampling argument so that accepted tokens are distributed according to the target policy and therefore receive weight PRESERVED_PLACEHOLDER_^^^^11max_results11title:\11^^^^, not merely a reduced ratio (&&&^^^^11query11^^^^&&&).

Finally, not every “importance-based selection” method is an unbiased SIS estimator in the strict Monte Carlo sense. The subset-selection work and distributed SGD preserve unbiased weighted objectives, whereas the human-trajectory method uses learned selection to reduce inference cost and explicitly does not target unbiased integral estimation (&&&^^^^11sort_by11^^^^&&&, &&&^^^^11submittedDate11^^^^&&&). This suggests that Selective Importance Sampling is best regarded as a family resemblance concept centered on selective allocation of sampling mass and correction effort, with one fully explicit modern instantiation in off-policy LLM alignment.