Mutual Indexing in Document Retrieval

• Mutual Indexing is a framework that formalizes document indexing as a trade-off between compressing representations and preserving information for accurate query prediction.
• It leverages mutual information measures and rate-distortion theory to derive information-optimal codes that support generative document retrieval.
• The approach extends to various applications including cross-lingual, multi-modal, and recommendation systems by ensuring compact yet informative document codes.

Mutual Indexing is a methodological framework for generative document retrieval (GDR) grounded in information theory. It formalizes the construction of document indexes as an explicit trade-off between compressing document representations and retaining information necessary for accurate answer prediction. By leveraging mutual information measures and rate-distortion theory, Mutual Indexing recasts indexing as a well-posed bottleneck problem, enabling the derivation of information-optimal indexing strategies. This approach generalizes beyond traditional retrieval to scenarios such as cross-lingual, multi-modal, and recommendation systems (Du et al., 2024).

1. Formal Setup and Markov Structure

Let $X \in \mathcal{X}$ denote the document space, $T \in \mathcal{T}$ the index-code space (where each $t$ is a compact ID string), and $Q \in \mathcal{Q}$ the query space. The generative model for the process is

$p(x, q, t) = p(x) p(t|x) p(q|x)$

expressing that for each document $x$, the index $t$ is stochastically generated as $p(t|x)$ and queries are sampled from $p(q|x)$, yielding the Markov chain $T \leftarrow X \rightarrow Q$.

This setup accommodates the GDR process: documents are indexed via codes $T \in \mathcal{T}$0 assigned according to $T \in \mathcal{T}$1, and downstream retrieval (or generation) is modeled as mapping queries $T \in \mathcal{T}$2 to these codes.

2. Mutual Information Quantities and Rate-Distortion Trade-offs

Two mutual-information terms govern the indexing objective:

• Compression cost: $T \in \mathcal{T}$3, quantifying the information retained about documents $T \in \mathcal{T}$4 by their codes $T \in \mathcal{T}$5.
• Predictive power: $T \in \mathcal{T}$6, quantifying how much the codes $T \in \mathcal{T}$7 reveal about queries $T \in \mathcal{T}$8.

Classical rate-distortion theory seeks a mapping $T \in \mathcal{T}$9 to minimize $t$0 (compression rate) under a bounded expected distortion $t$1, i.e.,

$t$2

In the context of Mutual Indexing, the "distortion" is formalized by how much information about $t$3 is lost, i.e., how well $t$4 supports downstream retrieval.

3. Information Bottleneck Reformulation

The core objective is to make $t$5 as compressed as possible, i.e., minimize $t$6, while ensuring that $t$7 retains sufficient information to predict $t$8, i.e., $t$9. This is formalized by

$Q \in \mathcal{Q}$0

Via the method of Lagrange multipliers, this yields the Information Bottleneck (IB) objective:

$Q \in \mathcal{Q}$1

with the scalar $Q \in \mathcal{Q}$2 regulating the trade-off.

This recasts indexing as role of the "bottleneck": $Q \in \mathcal{Q}$3 must capture sufficient mutual information with $Q \in \mathcal{Q}$4 (predictive sufficiency) while remaining a compressed representation of $Q \in \mathcal{Q}$5 (compactness).

4. Practical Estimation and Optimization

Operationalizing Mutual Indexing involves estimating $Q \in \mathcal{Q}$6 with a neural sequence-to-sequence (seq2seq) model (e.g., T5 encoder + PAWA decoder). For each query $Q \in \mathcal{Q}$7, the model outputs a distribution $Q \in \mathcal{Q}$8; supervised maximum likelihood training is used:

$Q \in \mathcal{Q}$9

After training, $p(x, q, t) = p(x) p(t|x) p(q|x)$0 approximates the true $p(x, q, t) = p(x) p(t|x) p(q|x)$1, enabling estimation of $p(x, q, t) = p(x) p(t|x) p(q|x)$2 as $p(x, q, t) = p(x) p(t|x) p(q|x)$3 using cross-entropy computations over sampled queries and their predicted indexes.

$p(x, q, t) = p(x) p(t|x) p(q|x)$4 is more involved, given $p(x, q, t) = p(x) p(t|x) p(q|x)$5 and $p(x, q, t) = p(x) p(t|x) p(q|x)$6 is usually one-hot. To study compression, a truncated code (e.g., the first $p(x, q, t) = p(x) p(t|x) p(q|x)$7 symbols of $p(x, q, t) = p(x) p(t|x) p(q|x)$8) is used, collapsing multiple $p(x, q, t) = p(x) p(t|x) p(q|x)$9 to the same code prefix.

5. Bottleneck-Minimal Indexing (BMI) Algorithm

The construction of bottleneck-minimal codes proceeds in three stages:

1. Query-Based Document Embedding: For each document $x$0, aggregate a query set $x$1 comprising real user queries (RealQ), synthetic queries generated via docT5query (GenQ), and random document passages (DocSeg). Each $x$2 is embedded using a fixed BERT encoder, and their mean $x$3 is taken as the document's "query center" in $x$4.
2. Hierarchical k-means Clustering: The set $x$5 is recursively clustered via $x$6-way k-means at each of $x$7 levels, assigning each $x$8 an $x$9-digit code $t$0. For Gaussian assumptions, the induced $t$1 is IB-optimal.
3. Autoregressive Indexer Training: The seq2seq indexer is fine-tuned on $t$2 pairs, training it to predict the index code from queries, thus realizing the bottleneck $t$3 imposed by clustering.

6. Empirical Results and Comparative Evaluation

Empirical evaluation was conducted on NQ320K (Natural Questions subset) and MS MARCO Lite datasets, using Rec@1/10/100 and MRR as metrics. Baselines include hierarchical random, locality-sensitive hashing (LSH), and hierarchical k-means on document embeddings (HKmI).

Key findings include:

• BMI plots in $t$4 vs. $t$5 trace a bottleneck curve, outperforming all baselines by achieving closer proximity to the lower-left (low $t$6 and low distortion).
• BMI improves upon HKmI by 1–7 points in absolute Rec@1, with larger relative gains for smaller seq2seq models (up to +7 points for T5-mini).
• Combining GenQ, RealQ, and DocSeg queries yields superior document representations ($t$7), as shown by ablation.

Method	Rec@1 Improvement (BMI over HKmI)	Model Sensitivity
HKmI Baseline	0	Baseline comparison
BMI (T5-mini)	+7 pts	Greater gain on small models
BMI (Other)	1–7 pts	Consistent-moderate gain

7. Generalization and Scope

Mutual Indexing provides a unified perspective wherein index design is an explicit Pareto optimization between compression ($t$8) and predictive sufficiency ($t$9 or, in general, $p(t|x)$0 for downstream task variable $p(t|x)$1). Unlike previous GDR approaches that largely controlled only distortion relative to $p(t|x)$2, Bottleneck-Minimal Indexing achieves direct IB objective satisfaction by clustering in query space.

This framework generalizes naturally: any application requiring compact-yet-informative codes (e.g., cross-lingual retrieval, multi-modal search, recommendation) can learn or select $p(t|x)$3 so as to trace the Pareto frontier in the $p(t|x)$4 plane, supporting principled trade-offs among competing system objectives (Du et al., 2024).