ChunkGroupSHAP: Listwise SHapley for Dense Retrieval

• The paper introduces ChunkGroupSHAP, a method that clusters semantically similar text chunks to provide listwise post-hoc explanations for dense ranking models.
• It computes group Shapley values by perturbing shared cross-document features, aligning explanation granularity with the encoder’s contextual representations.
• Empirical findings show that query-local grouping with ChunkGroupSHAP improves fidelity over word-level methods, especially in heterogeneous retrieval settings.

ChunkGroupSHAP is a listwise SHapley method for explaining embedding-based rankings via semantic chunk grouping. It was introduced for dense retrieval settings in which documents are scored through contextual sentence- and passage-level representations, while many earlier listwise explanation methods still attribute rankings to isolated words. The method clusters semantically related chunks into shared cross-document features, masks a group by perturbing all documents with related evidence, and attributes rankings at a granularity closer to dense representations while preserving the listwise setup. Its central empirical claim is that the best explanation unit is setting-dependent: word features for lexical BM25, corpus-level groups for dense rankers, and query-local grouping for heterogeneous web retrieval (Kim et al., 26 Jun 2026).

1. Problem setting and motivation

ChunkGroupSHAP studies post-hoc explanation of rankings rather than isolated query-document scores. Given a query $q$, a retrieval model $R$ returns a ranked candidate list

$D_q = (d_1,\ldots,d_n),$

sorted by model scores $s_R(q,d)$. In the dense setting considered by the method, the score is

$s_R(q,d) = \cos(E(q), E(d)),$

where $E(\cdot)$ is an encoder producing normalized vectors (Kim et al., 26 Jun 2026).

The method starts from a feature-unit mismatch. RankSHAP uses stemmed word features, and RankingSHAP is compared in a word-feature setup. That is natural for BM25-like lexical rankers, because BM25 is directly driven by term matches and term statistics. For dense rankers, however, word-level explanation units are too fine-grained relative to the passage-level evidence the model encodes; perturbing isolated words may not correspond to how a dense encoder actually represents relevance; and the resulting explanations are often fragmented and less inspectable (Kim et al., 26 Jun 2026).

Raw chunks provide the opposite trade-off. A chunk is a text span obtained by segmenting documents with a tokenizer-based sliding window, and chunks are intended to correspond more closely to the sentence/passage-level units used by dense retrieval systems. But a raw chunk is usually document-specific. If a feature occurs in only one document, then masking it behaves more like a pointwise perturbation than a listwise one. ChunkGroupSHAP is proposed to resolve this tension by treating groups of semantically similar chunks as the features of the cooperative game, so that a single perturbation can alter many documents at once while still operating at passage-level granularity (Kim et al., 26 Jun 2026).

2. Formal framework

Let $\mathcal{U}$ be the document universe used to build groups. Documents are segmented into chunks by

$$\{c_1^{(d)}, c_2^{(d)}, \ldots\} = Chunk(d;\, w, s) \quad \forall\, d \in \mathcal{U},$$

where $w$ is the chunk window size and $s$ is the stride. The full set of chunks is

$R$0

Each chunk is embedded using a grouping encoder $R$1,

$R$2

and $R$3-means clustering yields a group assignment

$R$4

with feature set

$R$5

A group is therefore a shared cross-document semantic feature: different chunks from different documents can belong to the same group if they express similar meaning (Kim et al., 26 Jun 2026).

A coalition is a subset $R$6. For a coalition $R$7, document $R$8 is reconstructed by retaining only chunks whose assigned groups are active: $R$9 The surviving chunks are concatenated in original order, and overlapping tokens are deduplicated when adjacent retained chunks are joined. If no chunk remains, the document is treated as empty and assigned score zero. The original ranker is then run again on reconstructed documents. For dense rankers,

$D_q = (d_1,\ldots,d_n),$0

For BM25, the reconstructed text is tokenized and rescored using the original BM25 parameters (Kim et al., 26 Jun 2026).

The cooperative-game value is listwise and self-referential. Let $D_q = (d_1,\ldots,d_n),$1 be the original ranker scores on $D_q = (d_1,\ldots,d_n),$2. Given cutoff $D_q = (d_1,\ldots,d_n),$3, the coalition value is

$D_q = (d_1,\ldots,d_n),$4

where $D_q = (d_1,\ldots,d_n),$5 is the ranking induced by coalition scores. The method sets $D_q = (d_1,\ldots,d_n),$6. It also sets $D_q = (d_1,\ldots,d_n),$7, explicitly overriding the empty-coalition behavior because empty documents would tie, and stable sorting could otherwise preserve the original order and yield an artificially high NDCG (Kim et al., 26 Jun 2026).

The grouped feature attribution for group $D_q = (d_1,\ldots,d_n),$8 is its Shapley value. A faithful reconstruction from the paper’s Shapley framework is

$D_q = (d_1,\ldots,d_n),$9

The paper estimates these attributions using KernelSHAP. With base value $s_R(q,d)$0, the approximation satisfies

$s_R(q,d)$1

because here $s_R(q,d)$2 and $s_R(q,d)$3 (Kim et al., 26 Jun 2026).

3. Semantic chunk grouping, masking semantics, and computation

ChunkGroupSHAP uses two grouping scopes. In corpus-level grouping, groups are built once over a large universe $s_R(q,d)$4, such as the whole corpus or a large candidate union, yielding reusable group IDs across queries. In query-local grouping, groups are built only for the current query’s retrieved set, $s_R(q,d)$5. Corpus-level groups are reusable and global; query-local groups are specialized to the current ranking and often improve local faithfulness, especially in heterogeneous retrieval settings such as MS MARCO (Kim et al., 26 Jun 2026).

The main chunk settings are $s_R(q,d)$6, with additional diagnostics including larger windows such as $s_R(q,d)$7, and for MS MARCO Passage a small setting $s_R(q,d)$8. The grouping hyperparameter is $s_R(q,d)$9 in the main benchmark, with larger sweeps up to $s_R(q,d) = \cos(E(q), E(d)),$0 in diagnostics. The implementation uses scikit-learn $s_R(q,d) = \cos(E(q), E(d)),$1-means with fixed random seed $s_R(q,d) = \cos(E(q), E(d)),$2; for large chunk matrices with at least 50,000 rows it uses mini-batch $s_R(q,d) = \cos(E(q), E(d)),$3-means with batch size $s_R(q,d) = \cos(E(q), E(d)),$4 and three initializations; otherwise it uses standard $s_R(q,d) = \cos(E(q), E(d)),$5-means. For dense rankers, chunk embeddings are computed using the target dense ranker itself; for BM25, which has no embedding space, groups are built using E5-base embeddings (Kim et al., 26 Jun 2026).

The perturbation semantics are deletion-based reconstruction rather than embedding substitution. For ChunkGroupSHAP, masking removes raw-text chunk spans from reconstructed documents. For dense rankers, masking removes matching raw-text token spans while preserving remaining punctuation, casing, and whitespace; for BM25, masking removes matching token identities before sparse scoring. The method does not use learned mask embeddings, neutral replacement tokens, blank tokens inserted into the encoder, or direct embedding replacement. Instead it builds a new perturbed document text and rescors that text with the original ranker (Kim et al., 26 Jun 2026).

A practical advantage of the method is that the feature space is fixed at $s_R(q,d) = \cos(E(q), E(d)),$6, whereas word-feature counts vary with query and retrieved documents, and raw chunk counts vary with document number and length. This makes the SHAP regression dimension more controlled and the sampling budget easier to allocate. The dominant cost, however, is repeatedly re-encoding perturbed top-$s_R(q,d) = \cos(E(q), E(d)),$7 documents for dense rankers. In the main E5-small top-50 experiments, observed per-query wall-clock time is substantial: MS MARCO $s_R(q,d) = \cos(E(q), E(d)),$8 sec/query, FinanceBench $s_R(q,d) = \cos(E(q), E(d)),$9 sec/query, and AILACaseDocs $E(\cdot)$0 sec/query (Kim et al., 26 Jun 2026).

4. Empirical behavior across rankers and corpora

The experiments cover MS MARCO, FinanceBench, AILACaseDocs, FinQA, and an auxiliary MS MARCO Passage diagnostic. The rankers are E5-small, E5-base, E5-large, and BM25. Dense rankers use normalized inner-product retrieval, equivalent to cosine similarity over $E(\cdot)$1-normalized embeddings; BM25 uses $E(\cdot)$2 and $E(\cdot)$3. The main explanation setting uses top-50 candidate lists, 5,000 KernelSHAP samples, and up to 100 queries per dataset, except AILACaseDocs with 50 total queries. The main evaluation metric is Fidelity, computed by keeping only the top-7 attributed features, scoring each document by the sum of selected attributions it contains, sorting by that sum to get an attribution-induced ranking, and comparing that ranking with the original ranker output using Kendall’s $E(\cdot)$4, reported as $E(\cdot)$5 (Kim et al., 26 Jun 2026).

The central empirical pattern is summarized by the paper’s own thesis that the best explanation unit is setting-dependent:

Setting	Best explanation unit	Reported tendency
Lexical rankers	Word features	BM25 often favors word-level methods
Dense rankers on focused corpora	Corpus-level chunk groups	ChunkGroupSHAP usually outperforms words
Heterogeneous web retrieval	Query-local grouping	Local grouping improves sharply

On FinanceBench, AILACaseDocs, and FinQA with E5 rankers, corpus-level ChunkGroupSHAP usually outperforms word-based methods. Representative values include FinanceBench with E5-base: RankSHAP $E(\cdot)$6, RankingSHAP $E(\cdot)$7, and ChunkGroupSHAP up to $E(\cdot)$8; AILACaseDocs with E5-large: RankSHAP $E(\cdot)$9, RankingSHAP $\mathcal{U}$0, and ChunkGroupSHAP up to $\mathcal{U}$1; and FinQA with E5-large: RankSHAP $\mathcal{U}$2, RankingSHAP $\mathcal{U}$3, and ChunkGroupSHAP up to $\mathcal{U}$4 (Kim et al., 26 Jun 2026).

BM25 behaves differently. Word-level methods remain strongest on several datasets: on MS MARCO, BM25 RankSHAP is $\mathcal{U}$5, RankingSHAP is $\mathcal{U}$6, and ChunkGroupSHAP is much lower; on FinanceBench, BM25 RankSHAP $\mathcal{U}$7 exceeds chunk-group variants; and on FinQA, BM25 RankingSHAP $\mathcal{U}$8 exceeds chunk groups. The notable exception is AILACaseDocs, where chunk groups outperform word-level methods even for BM25: RankSHAP $\mathcal{U}$9, RankingSHAP $$\{c_1^{(d)}, c_2^{(d)}, \ldots\} = Chunk(d;\, w, s) \quad \forall\, d \in \mathcal{U},$$0, and ChunkGroupSHAP up to $$\{c_1^{(d)}, c_2^{(d)}, \ldots\} = Chunk(d;\, w, s) \quad \forall\, d \in \mathcal{U},$$1. This supports the paper’s claim that document length and corpus structure also matter, not only lexical versus dense scoring (Kim et al., 26 Jun 2026).

MS MARCO is the main exception for dense retrieval. Corpus-level groups underperform words, but query-local grouping improves sharply. For E5-small on MS MARCO, word methods are around $$\{c_1^{(d)}, c_2^{(d)}, \ldots\} = Chunk(d;\, w, s) \quad \forall\, d \in \mathcal{U},$$2, corpus-level ChunkGroupSHAP is $$\{c_1^{(d)}, c_2^{(d)}, \ldots\} = Chunk(d;\, w, s) \quad \forall\, d \in \mathcal{U},$$3–$$\{c_1^{(d)}, c_2^{(d)}, \ldots\} = Chunk(d;\, w, s) \quad \forall\, d \in \mathcal{U},$$4, and query-local grouping reaches $$\{c_1^{(d)}, c_2^{(d)}, \ldots\} = Chunk(d;\, w, s) \quad \forall\, d \in \mathcal{U},$$5; for E5-base, query-local grouping reaches $$\{c_1^{(d)}, c_2^{(d)}, \ldots\} = Chunk(d;\, w, s) \quad \forall\, d \in \mathcal{U},$$6. In diagnostics, query-local grouping improves $$\{c_1^{(d)}, c_2^{(d)}, \ldots\} = Chunk(d;\, w, s) \quad \forall\, d \in \mathcal{U},$$7 on all tested datasets: AILACaseDocs $$\{c_1^{(d)}, c_2^{(d)}, \ldots\} = Chunk(d;\, w, s) \quad \forall\, d \in \mathcal{U},$$8, FinanceBench $$\{c_1^{(d)}, c_2^{(d)}, \ldots\} = Chunk(d;\, w, s) \quad \forall\, d \in \mathcal{U},$$9, and FinQA $w$0 (Kim et al., 26 Jun 2026).

Several ablations clarify what is being gained. A raw-chunk baseline, “RankSHAP w/ Chunk,” often improves over words for dense rankers, confirming that chunk-level evidence is a better explanation unit. On MS MARCO Passage with E5-small, RankSHAP is $w$1, RankSHAP w/ Chunk is $w$2, and ChunkGroupSHAP (query-local) is $w$3. On FinanceBench with E5-small, RankSHAP is $w$4, RankSHAP w/ Chunk is $w$5, and ChunkGroupSHAP is $w$6. This indicates that much of the gain comes from moving from words to chunk-level semantic evidence, while grouping then restores listwise-compatible shared features and a fixed feature dimension (Kim et al., 26 Jun 2026).

Granularity also matters. Larger $w$7 generally helps reconstruction fidelity: for E5-small on AILACaseDocs, $w$8 rises from $w$9 at $s$0 to $s$1 at $s$2; on MS MARCO, it rises from $s$3 to $s$4. Larger windows help on long, domain-specific corpora, with AILACaseDocs and FinanceBench performing best around $s$5. Increasing KernelSHAP budget helps, especially from 1k to 3k samples, but does not alter the central conclusion. Random chunk groups can sometimes match or exceed semantic $s$6-means groups in automatic fidelity—for example, FinanceBench semantic $s$7 versus random $s$8—which shows that $s$9 alone is not enough to validate semantic quality (Kim et al., 26 Jun 2026).

The human evaluation is consistent with the chunk-level presentation claim. On MS MARCO Passage, seven participants reranked five passages for 30 queries using explanation displays, and mean NDCG@5 against the target model ranking was $R$00 for RankSHAP, $R$01 for ChunkGroupSHAP, and $R$02 for ChunkGroupSHAP (Random) (Kim et al., 26 Jun 2026).

5. Position within grouped Shapley research

ChunkGroupSHAP belongs to a broader family of methods that redefine the player space of Shapley attribution rather than leaving players as isolated atomic features. In tabular settings, groupShapley defines a new cooperative game where the players are feature groups rather than individual features, with grouped coalition value

$R$03

and group-level Shapley values computed directly on the partition rather than by summing feature-wise SHAP after the fact (Jullum et al., 2021). ChunkGroupSHAP is analogous in that it computes Shapley values over chunk groups as atomic players, but its player construction is semantic and cross-document, and its value function is listwise NDCG preservation rather than a pointwise prediction explanation (Kim et al., 26 Jun 2026).

The same pattern appears in other structured domains. GroupSegment-SHAP introduces group-segment players for multivariate time series, with explanatory units defined as feature-group $R$04 time-segment blocks derived from HSIC-based feature grouping and MMD-based temporal segmentation (Kim et al., 3 Jan 2026). O-Shap uses the Owen value under a hierarchy of semantically meaningful segments and emphasizes that hierarchy construction is part of the explanation problem itself (Zhou et al., 19 Feb 2026). “Succinct Interaction-Aware Explanations” learns disjoint interacting feature chunks and then computes Shapley values on the induced chunked game, giving a partition-based grouped explanation rather than singleton SHAP or full subset attribution (Xu et al., 2024). By contrast, Generalized SHAP and the Unified Shapley Framework to Explain Prediction Drift mainly generalize the explained target to group-of-observations, comparative cohort, or drift functionals, while keeping players as original features or feature–datapoint groups over dataset slices (Bowen et al., 2020, Shanbhag et al., 2021). GRASP, in turn, is a grouped-feature selection pipeline that aggregates per-feature SHAP magnitudes within predefined groups rather than computing exact Shapley values over groups as players (Luo et al., 11 Feb 2026).

This comparison suggests that ChunkGroupSHAP is best understood as a listwise retrieval-specific instance of grouped-feature Shapley explanation: the “chunk” concept maps to chunked text spans, the “group” concept maps to semantically clustered cross-document features, and the explanandum is the ranked list rather than a single prediction (Kim et al., 26 Jun 2026).

6. Limitations, interpretation, and implications

ChunkGroupSHAP explains behavior within a fixed candidate list. It does not model end-to-end full-corpus retrieval changes, and it does not provide causal guarantees. Its automatic evaluation measures faithfulness to the original ranker, not human interpretability. The random-grouping diagnostic makes this limitation explicit: high $R$05 does not guarantee semantic interpretability, because random chunk groups can sometimes match or exceed semantic groups on the automatic metric alone (Kim et al., 26 Jun 2026).

The method is also sensitive to design choices. Window size, stride, number of groups $R$06, grouping scope, and grouping encoder all affect the resulting features. The paper does not provide an automatic way to choose these for a new corpus. Query-local grouping is often more faithful, but it requires per-query clustering and adds computational overhead. KernelSHAP attributions remain approximate, and finer groupings can increase variance (Kim et al., 26 Jun 2026).

The broader implication is methodological rather than merely empirical: explanation units should follow both the ranker’s representational granularity and the structure of the retrieved corpus. For lexical BM25, isolated words remain the most appropriate unit in several settings. For dense rankers on focused corpora, corpus-level semantic chunk groups align better with the contextual evidence actually used by the model. For heterogeneous retrieval, fixed corpus-level grouping can be too coarse or can mix unrelated evidence, so query-local grouping is often preferable (Kim et al., 26 Jun 2026).

A further implication is suggested by recent work on aggregate SHAP values. That work shows that small aggregate SHAP values on the original data support do not justify feature removal, and proposes evaluation over the extended support, namely the product of marginals, as the theoretically sound setting for safe discard claims (Bhattacharjee et al., 29 Mar 2025). This suggests that if ChunkGroupSHAP scores were to be aggregated for chunk pruning or corpus-level feature removal, the same caution would apply: low aggregate chunk attribution on the observed retrieved lists alone would not, by itself, establish chunk irrelevance.