Local Perplexity Ranking for Robust LLM Inference

• Local Perplexity Ranking is an in-context denoising strategy that replaces noisy candidate demonstrations with lower-perplexity, semantically similar examples.
• It decomposes sequence perplexity into inherent and matching components to effectively identify label mismatches in text generation tasks.
• Empirical evaluations show significant performance gains on benchmarks like SCIQ while offering efficiency improvements over global ranking methods.

Local Perplexity Ranking (LPR) is an in-context denoising strategy for LLM inference that improves robustness to noisy demonstrations in text generation tasks. LPR systematically replaces candidate demonstrations that are likely mislabeled with semantically similar neighbors exhibiting lower perplexity, thereby reducing the prevalence of mismatched input–output pairs in prompt construction. This approach is grounded in a local decomposition of perplexity into inherent and matching components, utilizes semantic neighborhood retrieval, and achieves significant empirical gains on a range of noisy text generation benchmarks (Gao et al., 2024).

1. Perplexity Decomposition and Motivation

In the context of sequence modeling, LPR adopts the standard sequence perplexity definition for a concatenated input–output pair $\bm{z} = (\bm{x}, \bm{y})$ with $|\bm{z}|$ tokens under model $p_\theta$: $$\mathrm{Perplexity}(\bm{z}) = \exp\left( -\frac{1}{|\bm{z}|} \sum_{i=1}^{|\bm{z}|} \log p_\theta(z_i \mid z_{<i}) \right).$$ Empirically, $\mathrm{Perplexity}(\bm{z})$ is elevated for demonstrations containing incorrect (noisy) output $\tilde y$ for input $\bm{x}$. To explain the informativeness of this signal, perplexity is conceptually decomposed as

$$\mathrm{Perplexity} = \mathrm{Inherent~Perplexity} + \mathrm{Matching~Perplexity},$$

where the inherent component captures the base “difficulty” of a given input–output pair (approximately constant for semantically similar queries), and the matching component reflects the penalty for label–input mismatch. Local comparison within semantic neighborhoods thus enables the detection of anomalously high perplexity that is specifically attributable to noisy labels.

2. LPR Algorithm: Formal Procedure

Given a test input $\bm{x}_{\text{test}}$, a noisy candidate pool $$\widetilde{\bm{C}} = \{(\bm{x}_j, \tilde y_j)\}_{j=1}^M$$, and a base retrieval strategy $|\bm{z}|$0 (e.g., Random, TopK by similarity, DPP for diversity), LPR augments selection as follows:

1. Select initial $|\bm{z}|$1 candidates $|\bm{z}|$2.
2. For every candidate $|\bm{z}|$3, compute the text embedding $|\bm{z}|$4 and sequence perplexity.
3. For each $|\bm{z}|$5 in $|\bm{z}|$6:
   • Compute cosine similarity to all other candidates.
   • Identify the $|\bm{z}|$7 nearest neighbors $|\bm{z}|$8 in embedding space (excluding $|\bm{z}|$9).
   • Form a local cluster $p_\theta$0.
   • Sort $p_\theta$1 by perplexity, resulting in ranking $p_\theta$2 for each member.
   • If the normalized rank $p_\theta$3 (candidate is among the bottom $p_\theta$4 fraction by perplexity), replace $p_\theta$5 with the top neighbor $p_\theta$6 among $p_\theta$7 for which $p_\theta$8.
4. Return the cleaned demonstration set $p_\theta$9.

Default hyperparameters are $$\mathrm{Perplexity}(\bm{z}) = \exp\left( -\frac{1}{|\bm{z}|} \sum_{i=1}^{|\bm{z}|} \log p_\theta(z_i \mid z_{<i}) \right).$$0 and $$\mathrm{Perplexity}(\bm{z}) = \exp\left( -\frac{1}{|\bm{z}|} \sum_{i=1}^{|\bm{z}|} \log p_\theta(z_i \mid z_{<i}) \right).$$1, i.e., bottom 50% is eligible for replacement.

3. Semantic Similarity and Neighbor Retrieval

LPR relies on semantic clustering in the joint input–output space. Each example $$\mathrm{Perplexity}(\bm{z}) = \exp\left( -\frac{1}{|\bm{z}|} \sum_{i=1}^{|\bm{z}|} \log p_\theta(z_i \mid z_{<i}) \right).$$2 is mapped to a fixed-size text embedding $$\mathrm{Perplexity}(\bm{z}) = \exp\left( -\frac{1}{|\bm{z}|} \sum_{i=1}^{|\bm{z}|} \log p_\theta(z_i \mid z_{<i}) \right).$$3 via a sentence encoder (BERT-base in the implementation; SBERT-compatible). The cosine similarity metric,

$$\mathrm{Perplexity}(\bm{z}) = \exp\left( -\frac{1}{|\bm{z}|} \sum_{i=1}^{|\bm{z}|} \log p_\theta(z_i \mid z_{<i}) \right).$$4

is used to compute nearest neighbors. LPR performs ranking and substitution exclusively among the local semantic cluster, facilitating effective denoising without altering the global distributional properties of retrieved demonstrations.

4. Hyperparameters, Selection, and Ranking Criteria

The main parameters for LPR are the neighborhood size $$\mathrm{Perplexity}(\bm{z}) = \exp\left( -\frac{1}{|\bm{z}|} \sum_{i=1}^{|\bm{z}|} \log p_\theta(z_i \mid z_{<i}) \right).$$5 and replacement threshold $$\mathrm{Perplexity}(\bm{z}) = \exp\left( -\frac{1}{|\bm{z}|} \sum_{i=1}^{|\bm{z}|} \log p_\theta(z_i \mid z_{<i}) \right).$$6. Within each cluster of $$\mathrm{Perplexity}(\bm{z}) = \exp\left( -\frac{1}{|\bm{z}|} \sum_{i=1}^{|\bm{z}|} \log p_\theta(z_i \mid z_{<i}) \right).$$7 elements, demonstrations are ranked by ascending perplexity; the relative position (normalized rank) determines replacement eligibility. The clustering and ranking are independent of the specific choice of initial retrieval method, and cached computations enable efficient scaling over candidate pools.

Hyperparameter	Default Value	Functional Role
$$\mathrm{Perplexity}(\bm{z}) = \exp\left( -\frac{1}{|\bm{z}|} \sum_{i=1}^{|\bm{z}|} \log p_\theta(z_i \mid z_{<i}) \right).$$8	4	Size of local neighbor set
$$\mathrm{Perplexity}(\bm{z}) = \exp\left( -\frac{1}{|\bm{z}|} \sum_{i=1}^{|\bm{z}|} \log p_\theta(z_i \mid z_{<i}) \right).$$9	0.5	Replacement threshold (bottom fraction)
Encoder	BERT-base-uncased	Semantic retrieval in $\mathrm{Perplexity}(\bm{z})$0
Retrieval	(e.g.) TopK, DPP	Base method for initial candidate selection

5. Experimental Protocol and Evaluation

LPR was evaluated on six prominent text generation datasets: NQ (Natural Questions), WebQ (WebQuestions), SQuAD (reading comprehension), SCIQ (science question answering), GeoQuery (SQL code generation), NL2Bash (shell command generation). Noise was injected during training using two regimes:

• Irrelevant Noise: With probability $\mathrm{Perplexity}(\bm{z})$1, replace the gold answer with a random answer from another instance.
• Relevant Noise: With probability $\mathrm{Perplexity}(\bm{z})$2, substitute the correct answer with a contextually plausible but incorrect response generated by ChatGPT-4. Noise rates $\mathrm{Perplexity}(\bm{z})$3 were considered.

Experiments utilized greedy and nucleus sampling from the Llama2-7B LLM, with additional evaluations on Llama2-13B, Mistral-7B, and OPT-6.7B. Baselines included Random, standard TopK (cosine similarity on input embeddings), and DPP. Evaluation metrics were Exact Match (EM) for question answering and BLEU for code generation, using bert-base-uncased for retrieval embedding.

6. Quantitative Results and Ablation Studies

LPR demonstrates substantial empirical benefit over baseline retrieval methods under noisy annotation conditions. For example, on SCIQ with 60% irrelevant noise, the TopK baseline yields 29.31% EM versus 48.06% EM for TopK+LPR—an 18.75 percentage-point improvement. Across datasets, LPR consistently delivers $\mathrm{Perplexity}(\bm{z})$4–$\mathrm{Perplexity}(\bm{z})$5 EM gain in high-noise settings.

Ablation on the replacement threshold $\mathrm{Perplexity}(\bm{z})$6 (averaged at 60% noise) found:

• $\mathrm{Perplexity}(\bm{z})$7: +14.2 EM
• $\mathrm{Perplexity}(\bm{z})$8: +13.8 EM (default)
• $\mathrm{Perplexity}(\bm{z})$9: +12.9 EM

Ablation on neighbor count $\tilde y$0:

• $\tilde y$1: +11.5 EM
• $\tilde y$2: +13.8 EM (default)
• $\tilde y$3: +14.0 EM

Comparison with Global Perplexity Ranking (selecting globally lowest-perplexity demonstrations from 100 candidates) shows LPR performs better in clean and low-noise regimes and is approximately $\tilde y$4 faster due to the locality of comparisons. Transfer experiments with Llama2-13B, Mistral-7B, and OPT-6.7B yield consistent gains of $\tilde y$5–$\tilde y$6 EM on 60% noise. LPR also recovers $\tilde y$7–$\tilde y$8 accuracy on text classification tasks (SST-2, AGNews) with label noise, relative to TopK/DPP.

7. Theoretical Insights, Constraints, and Potential Extensions

The principal insight behind LPR is that while noisy labels inflate perplexity, absolute perplexity conflates example difficulty and noise. By restricting comparison to a neighborhood of semantically similar examples—where inherent perplexity is approximately constant—deviations in perplexity more precisely flag label mismatches. Local replacement thus eliminates poor demonstrations while retaining semantic diversity.

Limitations include the assumption that most examples within each semantic neighborhood are correctly labeled; performance degrades as local noise rates exceed 80%. LPR’s empirical effectiveness is not accompanied by a formal theoretical guarantee. Computational overhead is moderate, since most LLM and embedding computations can be cached.

Extensions proposed include adaptive $\tilde y$9 (thresholding based on perplexity gap), soft denoising via weighted neighbor voting, incorporation of alternative “matching” signals such as textual entailment, and theoretical analysis linking matching perplexity to the in-context log-prob gradient (Gao et al., 2024).