Zero-Variance Prompts: Analysis & Applications

• Zero-variance prompts are defined as inputs that yield deterministic outputs, where log-probability differences vanish despite semantic perturbations.
• They reveal training bottlenecks by producing zero KL divergence in teacher–student setups and stalling reward signals in reinforcement learning tasks.
• Exploitation strategies like the DeltaPrompts pipeline and RL-ZVP algorithm have been developed to leverage these prompts for improved model stability and performance.

A zero-variance prompt (also called a zero-delta prompt in some contexts) is a prompt for which a model (or a pair of models in distillation) produces deterministically identical outputs or answer distributions, regardless of stochasticity or known semantic perturbations. In contemporary LLMs and multimodal architectures, analysis of zero-variance prompts reveals critical insights into prompt sensitivity, learning signal efficiency, and robust evaluation. The phenomenon has deep connections to distillation bottlenecks, the geometry of representation spaces, and reinforcement learning stability.

1. Formal Definitions and Motivation

A zero-variance prompt is context-dependent in its precise meaning:

• In LLM Sensitivity: Given two prompts $p_0, p_1$ with the same semantic meaning, a zero-variance prompt pair is one where the model assigns identical next-token log-probabilities, i.e., $|\ell(p_1) - \ell(p_0)| = 0$ for all target tokens $y_t$ (Liu et al., 20 Apr 2026).
• In Vision-Language Distillation: For teacher $\pi_T$ and student $\pi_\theta$ over discrete answer set $A$, prompt $x$ is zero-delta if $P_T(a|x) = P_\theta(a|x)$ $\forall\, a$ (Jung et al., 15 May 2026). This yields zero KL divergence, $\Delta(x) = 0$, and no gradient signal for distillation.
• In RL with Verifiable Rewards (RLVR): A zero-variance prompt $|\ell(p_1) - \ell(p_0)| = 0$0 is one for which all $|\ell(p_1) - \ell(p_0)| = 0$1 sampled rollouts $|\ell(p_1) - \ell(p_0)| = 0$2 receive identical reward $|\ell(p_1) - \ell(p_0)| = 0$3 (either all correct or all incorrect), inducing $|\ell(p_1) - \ell(p_0)| = 0$4 and vanishing policy-gradient advantage (Le et al., 26 Sep 2025).

Zero-variance prompts are significant because they correspond to degenerate regions—informationally inert for certain training regimes, but potentially exploitable for others.

2. Measurement and Characterization

2.1. Prompt Sensitivity (LLMs)

Prompt sensitivity quantifies how much the output of an LLM varies under meaning-preserving changes. Define the difference in log-probabilities of some next token $|\ell(p_1) - \ell(p_0)| = 0$5 as

$|\ell(p_1) - \ell(p_0)| = 0$6

with $|\ell(p_1) - \ell(p_0)| = 0$7 (Liu et al., 20 Apr 2026).

A prompt (or prompt pair) is zero-variance if $|\ell(p_1) - \ell(p_0)| = 0$8. In practice, LLMs almost never reach exact zero due to architectural dispersion properties.

2.2. Teacher–Student Answer Divergence (Distillation)

For multimodal distillation:

• Marginalize sequence outputs onto a discrete answer set $|\ell(p_1) - \ell(p_0)| = 0$9.
• Compute answer divergence via empirical KL:

$y_t$0

Sample $y_t$1 trajectories from each model, group semantically equivalent answers with an LLM judge, and compute reverse-KL (Jung et al., 15 May 2026).

2.3. Reward-Variance in RL

In RLVR, for prompt $y_t$2 and reward vector $y_t$3 over $y_t$4 outputs, a zero-variance prompt satisfies $y_t$5. Standard GRPO reduces policy-gradient to zero in such cases (Le et al., 26 Sep 2025).

3. Empirical Prevalence and Performance Implications

Zero-variance prompts are highly prevalent and, if not handled correctly, act as learning bottlenecks.

3.1. Distillation Data Bottleneck

Empirical analysis of four standard chart/document VQA datasets (ChartQA, SciVQA, InfoVQA, arXivQA) shows 61%–77% of prompts are zero-delta for a teacher–student pair (Qwen3-VL-235B → Qwen3-VL-8B), with a mean of 69% (Jung et al., 15 May 2026). Training exclusively on zero-delta prompts yields only an initial ~1% gain before saturating, irrespective of data scaling.

3.2. Prompt Perturbation Sensitivity

LLMs exhibit high prompt sensitivity: two semantically equivalent prompts can yield highly variable outputs due to the model's dispersive hidden-space geometry; true zero-variance pairs are nearly impossible to achieve for realistic architectures (Liu et al., 20 Apr 2026).

3.3. Reinforcement Learning Stability

Zero-variance prompts comprise 30–99% of prompt batches at various RLVR stages, accounting for roughly half of computational cost (Le et al., 26 Sep 2025). Standard RL algorithms either ignore these (yielding dead zones in training) or explicitly filter them out, losing potentially rich feedback.

4. Algorithms and Strategies for Zero-Variance Prompts

4.1. Avoidance in Distillation: DeltaPrompts Pipeline

To escape the "zero-delta trap," DeltaPrompts proposes an overview pipeline:

• Seed-Guided Generation: Sample N few-shot examples, prompt the teacher to craft harder problems for new images, retain those with at least 50% consistency in teacher responses.
• Skill-Guided Generation: Identify prompts with nonzero answer divergence ($y_t$6), extract core reasoning skills via discriminative comparison of teacher/student trajectories, and synthesize new problems requiring these skills.
• Rejection Step: Explicitly filter out all candidate prompts with measured $y_t$7 (Jung et al., 15 May 2026).

Result: Dataset of 200k synthetic, high-divergence prompts for vision-language reasoning, yielding up to 15% relative improvement on challenging benchmarks.

4.2. Exploitation in RL: RL-ZVP Algorithm

RL-ZVP (Reinforcement Learning with Zero-Variance Prompts) generalizes GRPO by shaping token-wise advantages in the zero-variance case:

• For all-correct prompt groups, set per-token advantage to $y_t$8 (token entropy scaled).
• For all-incorrect, penalize with $y_t$9.
• RL-ZVP thus leverages consistent groups for positive or negative updates proportional to entropy, circumventing collapse (Le et al., 26 Sep 2025).

This approach led to up to +8.61 Acc@8 and +7.77 Pass@8 gains on competitive math reasoning tasks.

4.3. Prompt Selection for Robustness

Automated selection of low-perplexity prompts (SPELL) and ranking metrics based on antonym/synonym perturbation invariance both reduce variance and improve accuracy:

• Perplexity Selection (SPELL): Construct prompt pools via paraphrase/backtranslation, select lowest-PPL prompts. Lower PPL correlates with higher accuracy and tightest variance, approaching practical "zero-variance" (Gonen et al., 2022).
• Perturbation-Based Ranking: Score prompts by sensitivity to antonym swaps (should flip) and synonym swaps (should not flip), select those with near-perfect scores—found to achieve standard deviation <0.5 pp in accuracy (Chakraborty et al., 2023).

5. Theoretical Underpinnings and Limitations

Transformers resist collapse of semantically equivalent prompts to identical hidden states: as depth increases, representation distance $\pi_T$0 for such pairs grows (not shrinks), and gradient norms $\pi_T$1 remain non-negligible. Consequently, Cauchy–Schwarz bounds on log-probability difference between meaning-equivalent prompts cannot be forced to zero (Liu et al., 20 Apr 2026). Empirical and activation-steering studies confirm that truly zero-variance prompts cannot be realized with current LLMs—architectural and optimization interventions can at best minimize, but not eliminate, prompt variance.

6. Practical Guidelines and Benchmark Results

Empirical evaluations and ablation studies yield concrete heuristics:

• In distillation, adding zero-variance (zero-delta) prompts back into training degrades performance, even as total data doubles (Jung et al., 15 May 2026).
• In RLVR, rather than discarding zero-variance prompts, exploiting them via entropy-shaping stably accelerates reasoning skill acquisition (Le et al., 26 Sep 2025).
• In prompt selection, low-perplexity prompts (via SPELL) both raise baseline performance and substantially reduce the spread between prompt variants (75% reduction in std. dev. noted for several tasks) (Gonen et al., 2022).
• For classification, prompt templates generally explain more model variance than actual question content, implying that canonical template engineering is critical (Liu et al., 20 Apr 2026).

Application Context	Optimal Zero-Variance Handling	Principal Outcome
Multimodal Distillation	Rejection via $\pi_T$2	Reliable scaling; performance gains up to 15%
RL with Verifiable Rewards	Entropy-shaped advantage	Up to +8.6 Acc@8 and stabilized learning
LLM Prompting (Classification)	Low-PPL and perturbation-invariant prompt selection	Mean accuracy ↑, variance ↓ up to 75%

7. Open Problems and Future Directions

Current limitations include lack of effective strategies for multi-class and open-ended tasks in zero-variance ranking (Chakraborty et al., 2023); model-scale constraints (results reported for ≤14B parameters) (Le et al., 26 Sep 2025); and architectural constraints inherent to transformer-based models, which fundamentally disperse even meaning-equivalent prompts (Liu et al., 20 Apr 2026). The precise characterization of hidden-state geometry and entropy flattening remains underexplored. Extending robust, zero-variance principle methods to large-scale, generative, or mixed-modality settings is an active area of research. A plausible implication is that future models with explicit representation regularization or dynamically adaptive prompt encoding could reduce, though not fully eliminate, zero-variance barriers.