Comprehensive Info Bottleneck (CoIBA)

• The paper introduces CoIBA, extending the information bottleneck principle from single- to multi-layer attribution to capture distributed evidence.
• It employs a shared, layer-independent damping ratio to compress features while retaining critical information across layers.
• Empirical results on Vision Transformers show enhanced attribution faithfulness and improved performance on benchmarks like ImageNet.

Searching arXiv for the specified paper and closely related attribution/IB work to ground the article. Comprehensive Information Bottleneck (CoIBA) is a feature-attribution method for interpreting Vision Transformers that extends the information bottleneck principle from a single layer to multiple targeted layers. It is designed to reveal the contribution of input variables to the decision-making process by constructing an attribution map from a shared, layer-independent damping ratio that controls feature compression across layers. The central claim is that attribution restricted to a specific layer can neglect evidence distributed across depth, whereas CoIBA estimates “comprehensive information” by applying information bottlenecks in multiple targeted layers and sharing relevant information among them through a common parametric ratio. The method was introduced in "Comprehensive Information Bottleneck for Unveiling Universal Attribution to Interpret Vision Transformers" (Hong et al., 6 Jul 2025).

1. Conceptual scope and motivation

Feature attribution methods aim to expose the contribution of input variables to a model’s decision. In the formulation underlying CoIBA, the limitation of existing methods grounded on the information bottleneck principle is that they compute information in a specific layer, compress features by injecting noise via a parametric damping ratio, and consequently obtain attributions that can neglect evidence distributed across layers. CoIBA addresses this by discovering the relevant information in each targeted layer and by estimating comprehensive information using a shared damping ratio across those layers (Hong et al., 6 Jul 2025).

The shared ratio is the defining structural element of the method. According to the formulation, it complements over-compressed information and discovers omitted clues of the decision by sharing relevant information across the targeted layers. This suggests that CoIBA is not merely a multi-layer repetition of a single-layer bottleneck, but a coupled attribution mechanism in which compression decisions are coordinated across depth.

The method is presented for pretrained networks and, in its empirical instantiation, for Vision Transformers. Its attribution output is token- or patch-level, which aligns naturally with the tokenized input representation used by ViT architectures.

2. Formal problem formulation

Let $X$ denote the input random variable, $Y$ the label, $R_l$ the unperturbed activation at layer $l$ of a pretrained network $f$, and $Z_l$ a bottlenecked or perturbed version of $R_l$. CoIBA inserts $L$ bottlenecks into a sequence of layers $s, s+1, \dots, e$. At each such layer,

$Z_l = \lambda R'_l + (1-\lambda)\epsilon_l,$

where $Y$0 is the activation immediately before the bottleneck at layer $Y$1, $Y$2 is Gaussian noise with the same first and second moments as the non-perturbed $Y$3, and $Y$4 is a shared, layer-independent damping ratio vector of dimension $Y$5, parameterized as $Y$6 for $Y$7 (Hong et al., 6 Jul 2025).

The objective is to find $Y$8 such that the final bottleneck $Y$9 retains as much information as possible about $R_l$0 while simultaneously removing redundant or unnecessary information at every intermediate stage. The multi-layer information bottleneck objective is

$R_l$1

with $R_l$2 and $R_l$3 denoting layer-specific trade-off hyperparameters.

A key simplification follows from a data-processing argument: the average compression term can be upper-bounded by a single mutual information term,

$R_l$4

which yields the collapsed objective

$R_l$5

This reduction is operationally significant because it replaces multiple $R_l$6 parameters with a single $R_l$7. A plausible implication is that the shared-ratio construction and the single-term compression proxy are intended to make multi-layer attribution tractable without abandoning the information-bottleneck semantics.

3. Variational approximation and training criterion

Both the relevance term and the compression term are intractable in direct form, so CoIBA uses variational bounds. For the compression step, each mutual information term is written as

$R_l$8

The true marginal $R_l$9 is replaced by the fixed prior Gaussian

$l$0

giving the upper bound

$l$1

Under the perturbation model, $l$2, and the KL can be written in closed form (Hong et al., 6 Jul 2025).

For the relevance step, the quantity $l$3 is lower-bounded through the network’s final classification head $l$4, which is trained with cross-entropy loss

$l$5

Using the standard variational bound,

$l$6

The resulting training objective becomes

$l$7

with $l$8.

The role of the variational construction is explicitly characterized as a way to fairly reflect the relevant information of each layer by upper-bounding layer-wise information. In this formulation, the training objective is not optimizing attribution quality directly; instead, attribution emerges from a constrained relevance–compression trade-off.

4. Theoretical interpretation and guarantees

The theoretical rationale given for CoIBA is that the method maximizes the information $l$9 that survives to the end while minimizing the mutual information $f$0 at every intermediate stage, or equivalently $f$1 under the single-term bound. By the data-processing inequality and standard properties of the information bottleneck Lagrangian, the information removed at each layer can be argued not to increase classification risk, provided the retained relevance is maintained (Hong et al., 6 Jul 2025).

The paper’s argument is stated in two parts. First, if $f$2 is held roughly constant, then decreasing $f$3 drives $f$4 toward the minimal sufficient statistic of $f$5 for predicting $f$6. Second, any further reduction would necessarily reduce $f$7 and degrade the cross-entropy loss. In this sense, the residual activations in each layer are described as “provably” the smallest set of features necessary to preserve performance on $f$8.

This guarantee should be read in the precise sense supplied by the formulation: CoIBA guarantees that the discarded activation is unnecessary in every targeted layer to make a decision. It does not imply that all semantically meaningful cues are retained in human-interpretable form; rather, it formalizes necessity relative to preserving predictive performance under the imposed bottleneck objective.

A common misconception would be to interpret CoIBA as simply aggregating independent layer-wise attributions. The formulation instead ties layers together through a shared $f$9, and the theoretical claim concerns coordinated compression across the targeted sequence rather than post hoc combination of separate explanations.

5. Algorithmic realization and attribution mechanism

The algorithm operates on a pretrained network $Z_l$0, a chosen layer range $Z_l$1, a trade-off parameter $Z_l$2, a learning rate $Z_l$3, a batch size $Z_l$4, and an initialization $Z_l$5, which makes $Z_l$6. During training, a mini-batch is sampled; for each sample, the activation $Z_l$7 is extracted, perturbed to form $Z_l$8, and then propagated through successive bottlenecked layers until the final perturbed activation $Z_l$9 is obtained. The per-sample loss is

$R_l$0

after which $R_l$1 is updated by gradient descent and $R_l$2 is recomputed by the sigmoid (Hong et al., 6 Jul 2025).

After optimization, the per-token importance score is defined as

$R_l$3

which measures how much noise was injected at patch $R_l$4. The attribution map is then visualized as a heat map.

Several design choices are explicit in this construction. The damping ratio is shared across layers rather than layer-specific. The noise is Gaussian and matched to the first and second moments of the non-perturbed activation at each layer. The attribution variable is token-level, since $R_l$5 has dimension $R_l$6, the number of tokens or patches. This suggests that CoIBA is particularly well aligned with transformer backbones in which the token dimension is a natural explanatory axis.

6. Empirical behavior in Vision Transformers

CoIBA was applied to Vision Transformers by inserting bottlenecks just before the self-attention blocks of layers $R_l$7 through $R_l$8 of a standard ViT-Base/16. The reported experimental results are described as extensive and as demonstrating enhanced faithfulness of the resulting feature attributions (Hong et al., 6 Jul 2025).

On Insertion/Deletion, where AUC is preferred upward for insertion and downward for deletion, CoIBA consistently outperformed IBA and all propagation-based baselines by 2–3 points on ImageNet-1k, ImageNet-R, and ImageNet-A. On ROAD, measured via MoRF and LeRF, CoIBA yielded lower MoRF and higher LeRF errors, again outperforming prior methods by large margins. On FunnyBirds, a synthetic part-based dataset, CoIBA was the top method for completeness, correctness, and contrastivity.

The reported gains were also difficulty-aware: they held on low-confidence, “hard” samples, whereas IBA’s performance collapsed. The ablations isolate several architectural and objective-level effects. Shared $R_l$9 versus layer-specific damping showed that the universal ratio yields better trade-offs in compression and relevance. The variational upper bound versus separate $L$0 showed that the single-term bound both simplifies hyperparameter tuning and yields higher insertion/deletion scores. Uniform versus per-channel noise showed that uniform noise across channels, while remaining per-token, was best.

Runtime is also reported: generating one attribution map takes approximately $L$1 s on an A6000 GPU, on par with single-layer IBA. Within the scope of the reported experiments, this places CoIBA in the category of attribution methods that seek stronger faithfulness without an obvious runtime penalty relative to the single-layer bottleneck baseline.

The empirical pattern supports the paper’s central thesis that evidence for a Vision Transformer’s decision is distributed across layers and that a shared multi-layer bottleneck can recover omitted clues that single-layer attribution may miss. That conclusion remains an interpretation of the reported results, but it is the interpretation most directly suggested by the method’s design and benchmark behavior.