Reranking Laws for Language Generation: A Communication-Theoretic Perspective

Abstract: To ensure LLMs are used safely, one must reduce their propensity to hallucinate or to generate unacceptable answers. A simple and often used strategy is to first let the LLM generate multiple hypotheses and then employ a reranker to choose the best one. In this paper, we draw a parallel between this strategy and the use of redundancy to decrease the error rate in noisy communication channels. We conceptualize the generator as a sender transmitting multiple descriptions of a message through parallel noisy channels. The receiver decodes the message by ranking the (potentially corrupted) descriptions and selecting the one found to be most reliable. We provide conditions under which this protocol is asymptotically error-free (i.e., yields an acceptable answer almost surely) even in scenarios where the reranker is imperfect (governed by Mallows or Zipf-Mandelbrot models) and the channel distributions are statistically dependent. We use our framework to obtain reranking laws which we validate empirically on two real-world tasks using LLMs: text-to-code generation with DeepSeek-Coder 7B and machine translation of medical data with TowerInstruct 13B.

• The paper demonstrates the generator-reranker protocol achieves an asymptotically error-free system even with imperfect rerankers.
• It adapts communication theory principles to show that error probability decays exponentially for independent hypotheses and as a power law for dependent ones.
• Empirical studies on text-to-code generation and medical translation validate the framework's efficacy and robustness in practical applications.

Reranking Laws for Language Generation: A Communication-Theoretic Perspective

The paper "Reranking Laws for Language Generation: A Communication-Theoretic Perspective" by António Farinhas, Haau-Sing Li, and André F. T. Martins presents a conceptual framework to mitigate errors in LLMs by drawing analogies from communication theory. The primary focus is on reducing hallucinations and unacceptable outputs in LLMs by employing a reranker-based approach, akin to redundancy strategies used in error correction for noisy communication channels.

Summary

The authors introduce a generator-reranker system, where the LLM first generates multiple hypotheses, and a reranker then selects the most appropriate one. This parallels sending multiple message descriptions through noisy channels in communication systems. The generator is viewed as a sender and the reranker as a receiver, aiming to decode the correct message by selecting the most reliable hypothesis.

Key Contributions

1. Asymptotically Error-Free Protocol: The paper demonstrates that this generator-reranker protocol can yield an asymptotically error-free system. This holds even in scenarios where the reranker is imperfect and the hypotheses generated are dependent.
2. Independent Hypotheses:
   • For independent hypotheses and a perfect reranker, the error probability decays exponentially with the number of hypotheses, $N$.
   • With imperfect rerankers modeled by Mallows or Zipf-Mandelbrot distributions, the error probability still asymptotically approaches zero, albeit at different rates.
3. Dependent Hypotheses:
   • For dependent hypotheses coupled by a Beta prior, the protocol remains asymptotically error-free, with the error probability decaying as a power law when the reranker is perfect.
4. Empirical Validation:
   • The theoretical reranking laws are validated empirically on text-to-code generation (using DeepSeek-Coder 7B) and machine translation of medical data (using TowerInstruct 13B). The empirical results support the theoretical predictions, demonstrating the validity of the reranking laws.

Implications and Future Directions

Practical Implications

1. Application in High-Stakes Domains: The findings are particularly relevant to applications in high-stakes domains such as medical or legal fields, where the cost of errors is substantial.
2. Enhancing Robustness: Integrating rerankers in LLMs can significantly improve their robustness, enabling safer deployment in diverse applications.
3. Scaling Laws for Decoding: The derived reranking laws offer insights into the efficient use of hypotheses during decoding, guiding practitioners on the number of hypotheses needed to achieve desired error thresholds.

Theoretical Implications

1. Interdisciplinary Insights: The paper's approach bridges natural language processing and communication theory, suggesting that other sophisticated error-correction codes might inspire even more efficient decoding strategies.
2. Foundation for Further Research: The conceptual framework lays the groundwork for future studies to explore more complex dependency structures among hypotheses and different types of rerankers.

Conclusion

In conclusion, the paper introduces a novel perspective on improving the reliability of LLMs by leveraging principles from communication theory. The generator-reranker protocol shows promising results in reducing error rates, with robust theoretical backing and empirical validation. This work holds significant potential for both practical applications and advancing theoretical understanding in the domain of language generation.