SGRS: Saliency-Guided Rejection Sampling

• The paper introduces SGRS, a novel method that leverages per-token gradient-attention saliency to dynamically reject ungrounded tokens in LVLMs.
• It computes a saliency score by fusing attention weights with input gradients and applies adaptive thresholding to filter out hallucinations.
• Empirical results demonstrate reduced hallucination rates and improved factual accuracy on benchmarks when SGRS is integrated in the decoding process.

Saliency-Guided Rejection Sampling (SGRS) is an inference-time filtering framework developed for large vision-LLMs (LVLMs) to mitigate hallucinations during autoregressive generation. SGRS leverages a per-token gradient–attention saliency metric to dynamically reject candidate tokens that are weakly grounded in the model’s recent context, thereby filtering out predictions with a heightened risk of factual incoherence or hallucination. The method is formulated in the context of the LVLMs-Saliency framework, which quantifies the visual grounding strength of each output token by fusing self-attention weights with their input gradients (Zhang et al., 28 Jan 2026).

1. Saliency Score Definition and Computation

At each autoregressive decoding step $P$ in a pretrained LVLM $M$, SGRS computes a scalar saliency score for every candidate next token $c$. The process involves:

• Extraction of self-attention weight matrices $A^{(l,h)} \in [0,1]^{P \times P}$ for each layer $l = 1 \ldots L$ and attention head $h = 1 \ldots H$.
• Computation of the cross-entropy loss $\ell(y, t)$ for a one-hot label vector $y$ corresponding to token $t$ over the softmax logits $o(s^{(P)})$.
• Backpropagation of this loss to obtain gradients $M$0.
• Construction of saliency matrices as $M$1, where $M$2 denotes the Hadamard product and $M$3 applies causal masking, so only previous positions can contribute.
• Aggregation and $M$4 normalization across heads to yield $M$5 for each layer.
• Pooling $M$6 over a set of target layers $M$7 (typically the middle-to-deep layers) and all previous output positions $M$8 to yield a scalar saliency score per candidate:

$M$9

This saliency score quantifies the extent to which the next token $c$0 is grounded in preceding outputs via the current model gradients.

2. Context-Adaptive Thresholding

Instead of a static threshold, SGRS determines an adaptive acceptance criterion for token grounding at each step. For decoding position $c$1, the acceptance threshold $c$2 is given by:

$c$3

where $c$4 is a window covering the $c$5 most recent output tokens and $c$6 is a scaling factor. This context-adaptive threshold reflects recent model behavior and requires the saliency of any new candidate $c$7 to exceed a fraction $c$8 of the local average saliency history.

3. Rejection Sampling Procedure

The rejection sampling procedure of SGRS operates as follows:

1. Compute logits $c$9 and select the top-$A^{(l,h)} \in [0,1]^{P \times P}$0 most probable candidate tokens $A^{(l,h)} \in [0,1]^{P \times P}$1.
2. For up to $A^{(l,h)} \in [0,1]^{P \times P}$2 trials, sample a candidate $A^{(l,h)} \in [0,1]^{P \times P}$3 from $A^{(l,h)} \in [0,1]^{P \times P}$4 proportional to their softmax probabilities. Compute the candidate’s saliency score $A^{(l,h)} \in [0,1]^{P \times P}$5 and compare to the context threshold $A^{(l,h)} \in [0,1]^{P \times P}$6.
3. If $A^{(l,h)} \in [0,1]^{P \times P}$7, accept $A^{(l,h)} \in [0,1]^{P \times P}$8 as the next output token $A^{(l,h)} \in [0,1]^{P \times P}$9. Otherwise, remove $l = 1 \ldots L$0 from $l = 1 \ldots L$1 and repeat.
4. If no token is accepted after $l = 1 \ldots L$2 trials, select $l = 1 \ldots L$3 as a fallback.
5. Emit $l = 1 \ldots L$4 and increment $l = 1 \ldots L$5.

The method includes stabilization strategies such as exponential moving average smoothing for $l = 1 \ldots L$6, a minimal floor $l = 1 \ldots L$7 to prevent over-rejection, and bounding rejection trials to avoid inference stagnation (Zhang et al., 28 Jan 2026).

Hyperparameter	Default Value	Purpose
$l = 1 \ldots L$8	20	Top-K sampling size
$l = 1 \ldots L$9	5	Max rejection trials
$h = 1 \ldots H$0	5	Saliency history window
$h = 1 \ldots H$1	0.6	Threshold scaling factor
$h = 1 \ldots H$2	$h = 1 \ldots H$3	Target (middle/deep) layers
$h = 1 \ldots H$4	0.05	Minimal acceptance floor

4. Theoretical and Empirical Motivation

Empirical analysis demonstrates a pronounced relationship between saliency scores and output factuality. Specifically:

• Mean saliency of correct tokens $h = 1 \ldots H$5–$h = 1 \ldots H$6 versus hallucinated tokens $h = 1 \ldots H$7–$h = 1 \ldots H$8 across LLaVA-1.5-7B, Qwen2-VL-7B, and Intern-VL-7B.
• Hallucination probability decreases monotonically with increasing token saliency.
• Artificial reduction of the saliency signal ($h = 1 \ldots H$9) elevates the hallucination rate (Zhang et al., 28 Jan 2026).

While no formal theorem asserts that $\ell(y, t)$0 implies hallucination with probability 1, negative correlation between low saliency and hallucination provides an actionable operational filter.

5. Practical Implementation and Considerations

SGRS requires one backward pass per candidate per decoding step, increasing per-token latency by 30–40% compared to greedy decoding. For efficiency, SGRS can be selectively applied—restricted to tokens with high factuality risk or in conjunction with Local Coherence Reinforcement (LocoRE) for limited computational overhead ($\ell(y, t)$1 when used alone) (Zhang et al., 28 Jan 2026). The temperature used in softmax sampling is kept at 1.0 throughout.

Variants include:

• Smoothing $\ell(y, t)$2 via exponential moving average,
• Setting a floor $\ell(y, t)$3 for acceptance,
• Bounded rejection trials,
• Fallback mode on rejection exhaustion.

Practical hyperparameter values were tuned using held-out splits of the CHAIR and POPE hallucination detection benchmarks.

6. Experimental Results and Comparative Effectiveness

SGRS, combined with LocoRE, was assessed on standard benchmarks, comparing standard top-K and nucleus sampling methods and eight additional plug-and-play hallucination mitigation techniques. For LLaVA-1.5-7B:

• CHAIR hallucination rate: $\ell(y, t)$4 (SGRS+LocoRE) versus $\ell(y, t)$5 (baseline).
• POPE-F1 score: $\ell(y, t)$6 (SGRS+LocoRE) versus $\ell(y, t)$7 (baseline).

Comparable improvements were obtained across Qwen2-VL and Intern-VL families. On the MME general benchmark, SGRS+LocoRE improved the “Existence” and “Position” subtasks by 5–7 points compared to greedy sampling. Qualitative examples show SGRS’s ability to reject visually ungrounded (hallucinatory) tokens, such as “blue,” when saliency maps collapse, retaining candidates (“gray,” “watch”) exhibiting strong context dependency (Zhang et al., 28 Jan 2026).

7. Significance and Broader Implications

SGRS offers an interpretable, gradient-linked mechanism for online filtering of weakly grounded tokens in LVLMs, reducing hallucination while preserving fluency and downstream task performance. Its granularity and context sensitivity derive from leveraging intrinsic model dynamics—attention and gradient propagation—rather than relying solely on heuristics or external post-hoc filters. Implementation remains computationally more intensive than greedy decoding, but the method remains practical given its substantial factuality gains, particularly under controlled token selection or in conjunction with LocoRE. This suggests avenues for future research in combining gradient-aware and structural saliency signals for robust LVLM decoding (Zhang et al., 28 Jan 2026).