Attention Lattice Adapter (ALA) for Visual Explanations

Updated 12 July 2026

The paper demonstrates that ALA fuses multi-layer features in a lattice configuration to integrate attention with classification, eliminating manual layer selection.
It leverages LoRA-based partial adaptation to update parameters efficiently, reducing training cost while improving explanation fidelity.
A companion Alternating Epoch Architect (AEA) stabilizes attention maps and prevents over-localization, leading to substantially improved mean IoU metrics.

Searching arXiv for the ALA paper and key background papers on CLIP and LoRA. Attention Lattice Adapter (ALA) is a branch-integration explanation module for visual foundation models that was introduced to generate visual explanations while partially updating model parameters for interpretability enhancement. In the reported formulation, ALA is paired with Alternating Epoch Architect (AEA) and built on a CLIP ViT-B/16 image encoder with LoRA-based partial adaptation. Its defining architectural feature is the sequential integration of multiple intermediate features in a lattice-like structure, which removes the need for manual layer selection that characterizes prior ABN-family systems. The resulting attention map is used directly in the prediction pathway, so explanation generation is not merely post hoc but coupled to classification itself (Hirano et al., 18 Sep 2025).

1. Problem setting and conceptual motivation

ALA was proposed in the setting of visual explanation generation for visual foundation models, where prior explanation techniques were described as insufficiently adaptable to complex architectures (Hirano et al., 18 Sep 2025). The paper organizes earlier approaches into three categories: backpropagation-based methods, perturbation-based methods, and branch integration-based methods. Within that framing, backpropagation-based methods such as LRP, Integrated Gradients, Guided Backpropagation, and Grad-CAM are said to “sometimes focus on irrelevant areas” and can yield “insufficient attention maps with overly small attention regions” (Hirano et al., 18 Sep 2025).

Perturbation-based methods are treated as computationally expensive because they require many forward passes on masked images, and their faithfulness is described as potentially ambiguous because masking may create out-of-distribution inputs (Hirano et al., 18 Sep 2025). Branch integration-based methods are presented as more structurally aligned with explanation generation, but prior ABN-family approaches require manual layer selection, meaning that a human must decide which feature layer should drive explanation. The ALA proposal is specifically motivated by this limitation: it seeks to eliminate manual layer selection and instead use multiple intermediate features from the encoder in a structured integration scheme (Hirano et al., 18 Sep 2025).

A central implication of this design is that ALA treats explanation as an internal architectural function rather than as a separate saliency readout. The paper’s reported motivation therefore combines three objectives: adaptability to complex vision foundation models, tighter coupling between explanation and prediction, and partial parameter updating through LoRA rather than full-model retraining (Hirano et al., 18 Sep 2025, Hu et al., 2021, Radford et al., 2021).

2. Architectural organization

The reported system has three components: the VFM image encoder, the Attention Lattice Adapter, and the Perception Branch (PB) (Hirano et al., 18 Sep 2025). It is instantiated on a CLIP ViT-B/16 image encoder, with the input image denoted

$\bm{x} \in \mathbb{R}^{c \times h \times w}.$

The encoder contains $k$ transformer blocks and yields $\bm{h}_v^{(0)}$ , the input to the first block, together with intermediate features $\bm{h}_v^{(i)}$ from the $i$ -th block (Hirano et al., 18 Sep 2025).

ALA’s distinctive design is the replacement of single-layer explanation extraction with sequential multi-layer integration. In the former part of ALA, the input image is divided into patches of size $w_1 \times w_2$ , linearly projected into visual features $\bm{v}$ , and then processed through the transformer. At each layer $i \in \{1,2,3,4\}$ , the corresponding intermediate feature $\bm{h}_v^{i-1}$ is added to the input before that layer, and the output of the $l$ -th transformer layer is $k$ 0 (Hirano et al., 18 Sep 2025). This is the “lattice” construction: instead of selecting one privileged feature level, the model aggregates several intermediate representations.

The latter part of ALA, denoted $k$ 1, computes both an attention map $k$ 2 and predicted probabilities $k$ 3 (Hirano et al., 18 Sep 2025). The paper separates this into $k$ 4, which generates attention, and $k$ 5, which predicts class probabilities from the attention map for attention-loss calculation. This means ALA serves simultaneously as an explanation generator and as an auxiliary classifier whose prediction regularizes the attention branch.

The interaction with the perception branch is explicit:

$k$ 6

and

$k$ 7

Here $k$ 8 denotes the Hadamard product, $k$ 9 is the final image encoder output, and $\bm{h}_v^{(0)}$ 0 is the attention map (Hirano et al., 18 Sep 2025). In effect, the final classifier consumes encoder features modulated by the explanation map. This coupling is one of the paper’s key distinctions from post hoc saliency procedures.

3. LoRA-based partial adaptation and training objective

To reduce training cost and only partially update the visual foundation model, the method applies LoRA to the $\bm{h}_v^{(0)}$ 1-th layer of the CLIP image encoder (Hirano et al., 18 Sep 2025). The update rule is

$\bm{h}_v^{(0)}$ 2

where $\bm{h}_v^{(0)}$ 3 is the frozen original pre-trained weight, $\bm{h}_v^{(0)}$ 4, and $\bm{h}_v^{(0)}$ 5 (Hirano et al., 18 Sep 2025). This is the standard low-rank adaptation mechanism introduced by LoRA, in which a frozen backbone is supplemented by a learned low-rank correction rather than a full weight update (Hu et al., 2021).

The paper’s loss function combines the PB prediction and the ALA-side prediction:

$\bm{h}_v^{(0)}$ 6

where $\bm{h}_v^{(0)}$ 7 is cross-entropy loss, $\bm{h}_v^{(0)}$ 8 is the ground-truth label, and $\bm{h}_v^{(0)}$ 9 balances the two terms (Hirano et al., 18 Sep 2025). This dual-supervision structure is described as standard ABN-style supervision in the source summary.

The formulation places ALA in a hybrid role. It is not simply an auxiliary attention branch disconnected from the task objective, nor is it merely an interpretability overlay appended after classification. Instead, the attention map participates in the classifier’s effective input via $\bm{h}_v^{(i)}$ 0, while the auxiliary ALA-side prediction enforces label consistency on the explanation pathway (Hirano et al., 18 Sep 2025). A plausible implication is that the method is designed to improve explanation faithfulness by making explanation-bearing activations task-relevant during optimization, rather than by extracting them only after training.

4. Alternating Epoch Architect (AEA)

The companion mechanism, Alternating Epoch Architect, is introduced to address a specific failure mode: attention maps that become too small and overly localized (Hirano et al., 18 Sep 2025). The reported concern is that if ALA is updated continuously, the network may learn a shortcut in which ALA acts as a bypass and the model depends on a narrow region of the image. Because there are no ground-truth attention masks, there is no direct supervisory signal forcing coverage of the full object.

AEA alternates the behavior of the attention branch across epochs according to

$\bm{h}_v^{(i)}$ 1

where $\bm{h}_v^{(i)}$ 2 is an all-ones matrix, $\bm{h}_v^{(i)}$ 3 is the epoch number, and $\bm{h}_v^{(i)}$ 4 is any natural number (Hirano et al., 18 Sep 2025). The paper also states that the parameters of ALA are frozen every other epoch.

The interpretation given in the source is operationally clear. In odd epochs, the model uses the learned attention map $\bm{h}_v^{(i)}$ 5. In even epochs, the attention map is replaced by all ones, so the perception branch effectively sees the whole image feature map (Hirano et al., 18 Sep 2025). This alternation is said to keep the perception branch dominant in the loss every other epoch, thereby discouraging over-reliance on tiny salient regions.

This suggests that AEA is less an optimization heuristic in the abstract than a structural constraint on how interpretability is allowed to influence classification. Its purpose is not to maximize sparsity or selectivity, but to prevent pathological collapse of the explanation map. In the reported ablations, the presence or absence of AEA materially changes localization quality, which the authors interpret as evidence that alternating suppression of the attention pathway stabilizes explanation behavior (Hirano et al., 18 Sep 2025).

5. End-to-end explanation workflow and evaluation protocol

The end-to-end workflow reported for ALA is as follows: input image $\bm{h}_v^{(i)}$ 6; preprocessing by resizing to $\bm{h}_v^{(i)}$ 7 and applying flipping and cropping during training; encoding by CLIP ViT-B/16; extraction of multiple intermediate transformer features; integration of those features in ALA through lattice-like multi-layer fusion; generation of attention map $\bm{h}_v^{(i)}$ 8; combination of $\bm{h}_v^{(i)}$ 9 with final encoder features through $i$ 0; production of the classification output; and upsampling or overlay of the attention map on the input image to produce the visual explanation heatmap (Hirano et al., 18 Sep 2025).

The experiments use two datasets: CUB-200-2011 and ImageNet-S (Hirano et al., 18 Sep 2025). For CUB-200-2011, the reported statistics are 11,788 images, 200 bird species, pixel-level masks, and a split of 5,000 train, 994 validation, and 5,794 test. For ImageNet-S, the paper reports 1,195,741 images, 919 classes, pixel-level masks for 21,609 images, and a split of 1,174,132 train, 9,190 validation, and 12,419 test (Hirano et al., 18 Sep 2025).

The baselines are LRP, Integrated Gradients, Guided Backpropagation, Score-CAM, and Grad-CAM (Hirano et al., 18 Sep 2025). The evaluation metrics are mean IoU, insertion, deletion, and insertion-deletion (ID) score. Mean IoU is defined as

$i$ 1

where $i$ 2 is the number of samples, $i$ 3 is the predicted mask for sample $i$ 4, and $i$ 5 is the ground-truth mask (Hirano et al., 18 Sep 2025).

For insertion and deletion, the attention values $i$ 6 are sorted in descending order, defining

$i$ 7

and then constructing inserted and deleted images according to

$i$ 8

The insertion score is the AUC of the insertion curve, the deletion score is the AUC of the deletion curve, and the ID score is insertion score minus deletion score (Hirano et al., 18 Sep 2025). These metrics collectively evaluate overlap with object masks and the effect of retaining or removing highly attended regions.

6. Reported empirical performance and ablations

The main quantitative results reported in the source show that the method outperformed the baseline methods in terms of mean IoU, insertion score, deletion score, and insertion-deletion score on both CUB-200-2011 and ImageNet-S (Hirano et al., 18 Sep 2025).

On CUB-200-2011, the reported results are:

Method	mIoU	ID
LRP	0.000	0.211
IG	0.038	0.393
Guided BP	0.062	0.244
Score-CAM	0.160	0.514
Grad-CAM	0.161	0.221
Ours	0.693	0.697

On ImageNet-S, the reported results are:

Method	mIoU	ID
LRP	0.007	0.298
IG	0.215	0.091
Guided BP	0.215	0.097
Score-CAM	0.200	0.344
Grad-CAM	0.218	0.126
Ours	0.379	0.459

The paper highlights a 53.2-point absolute improvement in mean IoU on CUB-200-2011, computed from $i$ 9, where Grad-CAM is the strongest baseline by mIoU at 0.161 (Hirano et al., 18 Sep 2025). It also states that the difference on CUB is statistically significant ( $w_1 \times w_2$ 0) across all metrics.

The ablation study isolates three components. The full model, with AEA, LoRA, and ALA all enabled, achieves mIoU $w_1 \times w_2$ 1. Without adding intermediate features, mIoU drops to 0.520. Without LoRA, mIoU drops to 0.477. Without AEA, mIoU drops to 0.495 (Hirano et al., 18 Sep 2025). The qualitative description associated with the AEA ablation is that attention becomes localized to only part of the bird, whereas with AEA attention covers the entire bird more often.

These numbers support three paper-specific conclusions. First, multi-layer feature integration is treated as crucial, since removing intermediate-feature addition lowers mIoU from 0.693 to 0.520 (Hirano et al., 18 Sep 2025). Second, lightweight partial adaptation via LoRA materially contributes to explanation quality. Third, the alternating update schedule is not an incidental training trick but a major factor in preventing attention collapse.

7. Interpretation, limitations, and relation to adjacent methods

The paper characterizes ALA as a method that improves interpretability in the sense of better object localization, less background leakage, and stronger correspondence between prediction and explanation (Hirano et al., 18 Sep 2025). Qualitatively, BP methods are described as producing diffuse or noisy maps, and Grad-CAM and Score-CAM as often highlighting both target and irrelevant background, whereas ALA is reported to produce tighter, more object-centric maps (Hirano et al., 18 Sep 2025). Within that framing, ALA belongs to the branch integration-based lineage rather than the purely gradient-based or perturbation-based lineages, but modifies that tradition by replacing manual layer selection with multi-layer lattice integration.

Its use of a CLIP ViT-B/16 backbone places it in the context of visual foundation models built from contrastive language-image pretraining (Radford et al., 2021). Its use of LoRA situates it within the broader literature on parameter-efficient adaptation (Hu et al., 2021). However, the paper explicitly notes that it only evaluates CLIP as the VFM, and although the authors believe the method should generalize to other VFMs, this is not demonstrated (Hirano et al., 18 Sep 2025). It also identifies a possible future improvement: integrating edge-detection modules to better extract object boundaries and filter less significant objects.

A common misconception would be to treat ALA as simply another saliency algorithm. The reported formulation indicates otherwise: the attention map is generated by a trainable branch integrated into the forward path and used multiplicatively in the perception branch, while AEA regulates when that branch is active during training (Hirano et al., 18 Sep 2025). Another potential misconception would be to regard ALA as equivalent to LoRA-based tuning; the source distinguishes these roles. LoRA partially updates the image encoder, whereas ALA is the explanation module and AEA is the alternating training strategy (Hirano et al., 18 Sep 2025).

Taken together, the reported evidence presents ALA as a branch-integration explanation mechanism for visual foundation models in which multi-layer feature fusion replaces manual layer selection, LoRA provides lightweight encoder adaptation, and AEA counteracts collapse into overly small attention regions. The paper’s principal empirical claim is that this combination yields substantially higher object-faithful visual explanations than the listed baselines on CUB-200-2011 and ImageNet-S, with the clearest numerical result being the 53.2-point absolute mean-IoU improvement on CUB-200-2011 (Hirano et al., 18 Sep 2025).