The Silent Vote: Improving Zero-Shot LLM Reliability by Aggregating Semantic Neighborhoods

Abstract: LLMs are increasingly used as zero-shot classifiers in complex reasoning tasks. However, standard constrained decoding suffers from a phenomenon we define as Renormalization Bias. When a model is restricted to a small set of target labels, the standard softmax operation discards the probability mass assigned to semantic synonyms in the original distribution. This loss of information, which we call the Silent Vote, results in artificial overconfidence and poor calibration. We propose Semantic Softmax, an inference-time layer that recovers this lost information by aggregating the scores of the semantic neighborhood surrounding each target label. We evaluate this approach on Qwen-3 and Phi-4-mini models using GoEmotions and Civil Comments datasets. Our results demonstrate consistent improvements across all evaluation metrics: Semantic Softmax substantially reduces Expected Calibration Error (ECE) and Brier Score, while simultaneously enhancing discriminative performance in terms of AUROC and Macro-F1. By accounting for linguistic nuances, our method provides a more calibrated and accurate alternative for zero-shot classification.

• The paper identifies the Renormalization Bias in zero-shot LLM decoding and proposes Semantic Softmax to aggregate semantic neighborhood probabilities effectively.
• The methodology uses unconstrained top-K extraction and cosine similarity weighting to redistribute probability mass, achieving an over 8× reduction in Expected Calibration Error.
• Experimental results on emotion and toxicity datasets confirm improved calibration and highlight the technique’s potential for high-stakes, ambiguous classification tasks.

Improving Zero-Shot LLM Calibration through Semantic Neighborhood Aggregation: An Analysis of "The Silent Vote"

Introduction

The reliability of LLMs as zero-shot classifiers is central to scalable deployment in real-world natural language understanding tasks. "The Silent Vote: Improving Zero-Shot LLM Reliability by Aggregating Semantic Neighborhoods" (2605.09739) identifies and systematically addresses a core deficiency in standard LLM decoding: the phenomenon of Renormalization Bias, which produces substantial overconfidence and miscalibration, particularly when models are forced to select from constrained candidate label sets. The paper introduces Semantic Softmax, an inference-time technique that aggregates logit mass from semantic neighborhoods around candidate labels, thereby recovering probability mass otherwise lost due to the silent vote effect. The following essay offers an in-depth, technical summary and critical analysis.

Renormalization Bias in Constrained LLM Decoding

Zero-shot text classification with LLMs typically relies on constrained decoding, where label probabilities are calculated by applying a softmax over a restricted vocabulary corresponding to the candidate labels. This procedure discards the probability mass assigned to semantic synonyms and related concepts, inherently distributed in the model's output space due to its pretraining and task adaptation. The discarded mass—termed the "Silent Vote"—is not arbitrary; it represents meaningful evidence within the semantic manifold that the label representations inhabit.

This renormalization produces a sharp "Synonym Trap": the resulting class probability distribution becomes artificially peaked, yielding systematic overconfidence. Unlike true confidence, this artifact arises not from the input evidence but from the operational constraints imposed on the output space. The overconfidence is particularly egregious in settings rife with semantic ambiguity or human annotator disagreement, such as fine-grained emotion classification or toxicity detection. As a result, standard pipelines obscure the aleatoric uncertainty that robust human-level annotation would reflect.

Semantic Softmax: Methodology

Semantic Softmax is introduced as a lightweight, training-free layer, implemented entirely at inference. The algorithm proceeds as follows:

1. Unconstrained Top-K Extraction: For a given prompt, the logit vector over the full vocabulary is extracted. The top $K$ tokens (by probability mass) are selected.
2. Semantic Kernel Construction: Output embeddings for both the candidate label tokens and the selected top-K tokens are projected into the embedding space. For each candidate label, a semantic kernel computes weights as thresholded cosine similarities between the candidate and each top-K token.
3. Mass Aggregation and Weighted Scoring: For each label, the semantic softmax aggregates the corresponding weighted probabilities from its semantic neighborhood, normalized across all candidates.

By re-incorporating semantically aligned neighbors, Semantic Softmax dynamically redistributes mass otherwise ignored by constrained softmax, thereby recovering uncertainty and resolving the synonym trap. The sole hyperparameters—neighborhood size $K$ and threshold $\tau$—are shown by ablation to be robust across reasonable values in practice.

Experimental Validation

The method is extensively evaluated on Qwen-3-1.7B and Phi-4-mini (3.8B) models, ensuring generality across architectures. Two datasets are employed:

• GoEmotions: Captures fine-grained, overlapping emotion categories and is a stress-test for synonym recovery.
• Civil Comments (Ambiguous Subset): Includes human mean scores derived from annotator disagreement, enabling the assessment of recovered aleatoric uncertainty.

Key evaluation metrics include Expected Calibration Error (ECE), Brier Score, AUROC, and Macro-F1.

Numerical Results:

• ECE Reduction: For Qwen-3, ECE drops from 0.574 to 0.069 on GoEmotions; for Phi-4-mini, from 0.421 to 0.065. The ECE is reduced by factors exceeding 8×, implying a significant calibration improvement.
• Brier Score: Marked reductions (e.g., Qwen-3: 0.842 to 0.591) indicate improved probability alignment.
• Discriminative Power: AUROC and Macro-F1 are consistently improved or maintained, demonstrating that calibration gains do not trade off against accuracy; for example, AUROC for Qwen-3 on Civil Comments improves from 0.784 to 0.882.

Reliability and Uncertainty Quantification

The efficacy of Semantic Softmax in mitigating overconfidence and reflecting observed data uncertainty is depicted in the calibration and confidence distribution plots.

Figure 1: (a) Standard constrained decoding is overconfident (below diagonal), while Semantic Softmax aligns closely with optimal calibration. (b) Semantic Softmax distributes probability mass in ambiguous sequences to match human annotator consensus, unlike the forced extremes of standard decoding.

As shown in Figure 1, the calibration curve of Semantic Softmax adheres closely to the ideal diagonal, accurately quantifying predictive reliability. For ambiguous inputs, instead of producing artificially extreme probabilities, Semantic Softmax reflects spectrum-valued uncertainty commensurate with human judgment. This capacity to restore meaningful probability under ambiguity is critical for high-stakes domains and aligns machine predictions with inherent task uncertainty.

Limitations

The study confines its scope to the LLM-as-classifier paradigm, leaving open the scalability and applicability of Semantic Softmax for generative or multi-label structured prediction. The technique depends on the quality of the embedding geometry; models with poorly organized semantic spaces may degrade neighbor aggregation efficacy. Furthermore, the incremental inference-time cost—though minimal for single-token classification—precludes deployment in high-throughput or long-form generation without further optimization. Multilingual generalizability remains an open question.

Implications and Future Directions

By identifying and operationally solving Renormalization Bias, this work informs both the theoretical understanding of LLM uncertainty and the practical deployment of large-scale zero-shot classifiers. Adopting Semantic Softmax in moderation, behavioral analysis, or sentiment systems rectifies confidence-accuracy misalignment without retraining or architectural adjustments.

Potential future developments include:

• Scaling to larger LLMs: Systematic studies on foundation models; assessing whether gains persist or scale.
• Application to open-set and generative tasks: Extending semantic aggregation for generative or open-vocabulary scenarios where class boundaries are softer.
• Cross-lingual and low-resource adaptation: Investigating embedding geometry in multilingual settings, especially for languages with sparse synonym representation.
• Dynamic kernel adaptation: Learning or adapting semantic kernel functions in a task-specific manner to optimize calibration-accuracy tradeoffs.

Conclusion

"The Silent Vote" rigorously exposes a fundamental flaw inherent to LLM zero-shot classification pipelines. By formalizing Renormalization Bias and introducing Semantic Softmax, the authors furnish a precise, operational fix that produces substantial improvement in calibration (over 8× ECE reduction) and discriminative fidelity across domains reflecting both hard and ambiguous semantic choices. The technique stands as a robust inference-layer intervention for enterprise and high-stakes applications, transforming the reliability and trustworthiness of LLM-based classifiers without expensive retraining or infrastructure disruption.