Understanding Ranking LLMs: A Mechanistic Analysis for Information Retrieval

Abstract: Transformer networks, particularly those achieving performance comparable to GPT models, are well known for their robust feature extraction abilities. However, the nature of these extracted features and their alignment with human-engineered ones remain unexplored. In this work, we investigate the internal mechanisms of state-of-the-art, fine-tuned LLMs for passage reranking. We employ a probing-based analysis to examine neuron activations in ranking LLMs, identifying the presence of known human-engineered and semantic features. Our study spans a broad range of feature categories, including lexical signals, document structure, query-document interactions, and complex semantic representations, to uncover underlying patterns influencing ranking decisions. Through experiments on four different ranking LLMs, we identify statistical IR features that are prominently encoded in LLM activations, as well as others that are notably missing. Furthermore, we analyze how these models respond to out-of-distribution queries and documents, revealing distinct generalization behaviors. By dissecting the latent representations within LLM activations, we aim to improve both the interpretability and effectiveness of ranking models. Our findings offer crucial insights for developing more transparent and reliable retrieval systems, and we release all necessary scripts and code to support further exploration.

• The paper demonstrates strong correlations between specific human-engineered features and neuron activations in ranking LLMs.
• It employs ridge regression-based probing to analyze how transformer layers represent lexical and semantic features, revealing grouped feature representations.
• The study reveals overfitting tendencies with out-of-distribution data, offering practical insights for refining ranking models.

Mechanistic Interpretability in Ranking LLMs: An Examination of Feature Representation in Transformer Networks

The paper "Probing Ranking LLMs: Mechanistic Interpretability in Information Retrieval" explores the internal workings of ranking LLMs, specifically focusing on the representation and significance of human-engineered features from datasets like MSLR within these models. The study utilizes probing techniques to analyze fine-tuning-based passage-reranking transformer networks, employing models such as RankLlama, a LoRa fine-tuned variant of Llama-2.

Approach and Methodology

The paper employs a probing-based, layer-by-layer analysis of neurons within the LLMs to understand the representation of various lexical and semantic features. The authors deploy ridge regression-based probes to map activations extracted from LLM layers to known features, investigating the captured information in the MLP units of each transformer block. Key features from the MSLR dataset are scrutinized, including both lexical parameters and query-document interaction features.

The probing is performed using datasets like MS MARCO and analyzed with respect to statistical information retrieval metrics, BERT/T5 scores, and various query-document similarity measures. The study explores activations across all layers of the RankLlama models and aggregates them token-wise for computation efficiency.

Key Findings

1. Feature Representation: The study identifies strong correlations between certain human-engineered features—such as covered query term number, min of term frequency, and mean of stream length normalized term frequency—and neuron activations in the LLMs. Conversely, features like BM25 and sum of term frequency are notably absent.
2. Feature Combinations: The paper finds that specific combinations and transformations of features are well-represented, suggesting that LLMs may interpret features in grouped or transformed contexts rather than in isolation.
3. OOD Variability: The investigation reveals that the RankLlama 13b model displays differing feature extraction behaviors for in-distribution versus out-of-distribution data, indicating potential overfitting during fine-tuning.
4. Numerical Results: The research demonstrates that certain features achieve coefficients of determination ($R^2$) greater than 0.85, signifying robust feature representation within LLMs.

Theoretical and Practical Implications

The study makes contributions towards bridging the gap between statistical and neural approaches to information retrieval by elucidating the nuanced mechanisms LLMs use for decision-making. It offers researchers insights into the feature extraction capabilities of LLMs, which can enhance interpretability and provide a roadmap for refining ranking algorithms.

Practically, these findings hold promise for model refinement, such as modifications to existing statistical features to better align with LLM activations or enhancing statistical ranking models by leveraging identified LLM features.

Future Directions

The paper suggests avenues for potential future research, including non-linear probes to better understand composite feature representations. The long-term vision involves cataloging all active features within LLMs and utilizing this knowledge to improve both model performance and interpretability.

In summary, "Probing Ranking LLMs: Mechanistic Interpretability in Information Retrieval" delivers valuable insights into the internal dynamics of ranking transformer networks, delineating which traditional features LLMs deem important and setting the stage for further research into LLM mechanisms and their application in robust, interpretable ranking systems.