---
title: Multi-Layer Attention Consistency Score (MACS)
url: https://www.emergentmind.com/topics/multi-layer-attention-consistency-score-macs
type: topic
---

# Multi-Layer Attention Consistency Score (MACS)

Searching arXiv for the primary and related papers to ground the article in current literature.
Multi-Layer Attention Consistency Score (MACS) denotes a family of quantitative constructs that measure how stable or coherent attention-related signals remain across multiple layers of a neural network. The term is used explicitly in the context of decoder-only large language models as a heuristic attribution method based on the consistency of maximal attention across layers [2509.17178], and it can also be formalized from closely related ideas in vision, vision-language-action, and explainability research where cross-layer attention consistency is central but not always named MACS [2603.24941], [1811.07484], [2210.09705]. Across these settings, MACS serves either as an attribution score, a pruning-control signal, an attention regularizer, or an evaluation statistic. The unifying principle is that attention becomes more informative when its multi-layer behavior is analyzed rather than treated as a single-layer magnitude snapshot.

## 1. Terminological scope and conceptual definition

MACS is not a single universally standardized object across the literature. The explicit formulation appears in “Attention Consistency for LLMs Explanation” [2509.17178], where MACS is defined for decoder-based models as a token-importance heuristic derived from the consistency of maximal attention across heads and layers. In other papers, the same underlying concept appears under different names: “inter-layer token ranking consistency” in vision-language-action models [2603.24941], “attention consistency loss” in CNN attention learning [1811.07484], and “attention consistency” as a correlation objective across explanation maps in vision models [2210.09705].

In the decoder-only LLM setting, MACS measures whether an input token repeatedly receives strong attention from the current output query across many layers. This makes it a per-token scalar attribution score rather than a training loss [2509.17178]. In vision-language-action models, the analogous signal is not per-token attribution but a frame-level estimate of whether attention magnitude is trustworthy, obtained from the Kendall rank correlation of token rankings across layers [2603.24941]. In CNN explainability, consistency is instead defined as the fraction of inner-layer attention that falls inside a mask derived from last-layer attention for the same class [1811.07484]. These formulations differ operationally, but all instantiate multi-layer consistency as a criterion for judging attention reliability.

A plausible implication is that MACS is best understood as a research category rather than a single formula: any score that compresses cross-layer agreement of attention, attention-derived rankings, or attention maps into a scalar diagnostic fits the concept, provided its precise form is specified.

## 2. Canonical decoder-only formulation

The clearest formal definition of MACS is given for decoder-only Transformers in [2509.17178]. Let the input tokens be \(X=\{x_1,\dots,x_N\}\), let \(t_1,\dots,t_{k-1}\) be the already generated tokens, and let the generation step be \(k\), with query position \(u=n-1=N+k-1\). For layer \(l\) and head \(h\), the self-attention vector from the current query is
\[
\mathbf{a}^{(n-1,l,h)} \in \mathbb{R}^{n-1}.
\]

This attention is split into attention to input tokens and attention to previous outputs. The method then defines a redistributed attention vector over inputs:
\[
(\mathbf{a}_{R}^{(n-1,l,h)})_i
= a_{I,i}^{(n-1,l,h)}
+ \frac{1}{N} \sum_{p=1}^{k-1} a_{O,p}^{(n-1,l,h)},
\]
which adds a uniform share of attention to previously generated tokens back to each input token [2509.17178].

For each layer, MACS retains only the strongest per-token signal across heads:
\[
(\mathbf{m}'_{l})_i
= \max_{h \in \{1,\dots,H\}}
\left( (\mathbf{a}_{R}^{(n-1,l,h)})_i \right).
\]
This max-pooling step makes MACS a “maximal attention” method rather than a full-path aggregation method [2509.17178].

To avoid multiplicative collapse, a floor vector is introduced:
\[
\mathbf{m}_l = \alpha \mathbf{m}'_l + (1-\alpha)\mathbf{1}^N,
\]
with default \(\alpha=0.8\) [2509.17178]. Multi-layer consistency is then accumulated by Hadamard products:
\[
\mathbf{c}_0 = \mathbf{m}_0,
\qquad
\mathbf{c}_l = \mathbf{m}_l \odot \mathbf{c}_{l-1}, \quad l=1,\dots,L.
\]
The raw MACS score for input token \(i\) is \(C_i=(\mathbf{c}_L)_i\). Finally, per-step scores are standardized with a Z-score:
\[
z_i^{(k)} = \frac{(\mathbf{c}_L^{(k)})_i - \mu^{(k)}}{\sigma^{(k)}}.
\]
These \(z_i^{(k)}\) are the final attribution scores at generation step \(k\) [2509.17178].

The central intuition is that transient attention spikes are suppressed by multiplication across layers, whereas tokens that repeatedly receive strong maximal attention remain salient. This design is explicitly positioned against dense attention aggregation such as Attention Rollout, which multiplies full attention matrices and therefore accumulates many weak or noisy paths [2509.17178].

## 3. Alternative formalizations in adjacent domains

A distinct MACS-style construct is recoverable from “Beyond Attention Magnitude: Leveraging Inter-layer Rank Consistency for Efficient Vision-Language-Action Models” [2603.24941]. There, token importance at layer \(l\) is first defined by attention magnitude:
\[
S_j^{(l)} = \sum_{i=1}^{N} A^{(l)}_{i,j},
\]
where \(A^{(l)}\in\mathbb{R}^{N\times N}\) is the attention matrix. Tokens are then ranked by \(S_j^{(l)}\), and inter-layer consistency is measured with Kendall’s coefficient:
\[
\tau_0 = \frac{P-Q}{\sqrt{(P+Q+T)(P+Q+U)}}.
\]
A frame-level consistency statistic is obtained by averaging across successive layers:
\[
\tau = \frac{1}{L-1}\sum_{l=1}^{L-1}\tau_0\bigl(r^{(l)},r^{(l+1)}\bigr).
\]
Although the paper names this quantity \(\tau\) rather than MACS, the details explicitly identify it as an inter-layer attention rank consistency measure that can be reconstructed as a MACS-style score [2603.24941].

In CNN attention learning, “Sharpen Focus: Learning with Attention Separability and Consistency” defines attention consistency through overlap between an inner-layer attention map and a mask derived from last-layer attention [1811.07484]. If \(\mathcal{A}^{in}\) is the inner-layer attention map and \(Mask\) is the soft region mask derived from last-layer ground-truth attention, then
\[
L_{AC} = \theta -
\frac{\sum_{i,j}\big(\mathcal{A}^{in}_{ij}\cdot Mask_{ij}\big)}
{\sum_{i,j}\mathcal{A}^{in}_{ij}},
\]
with \(\theta=0.8\) empirically [1811.07484]. The normalized overlap term inside this loss is already a layer-pair consistency score in \([0,1]\), and the paper’s details support generalizing it across multiple layers as a MACS-style statistic [1811.07484].

In ATCON, attention consistency is not between layers but between attribution maps or between original and masked-input attention maps, optimized via Pearson correlation [2210.09705]. This is methodologically relevant because it demonstrates that consistency may be defined directly through similarity of attention maps:
\[
L_A(\theta,x) = Pearson(A_{\text{Grad-CAM,1}}, A_{\text{Grad-CAM,2}}).
\]
A plausible implication is that MACS can be instantiated either structurally, through ranking agreement or overlap, or geometrically, through correlation of aligned attention maps [2210.09705].

## 4. Interpretation: what MACS measures

The meaning of a high or low MACS depends on the operational definition. In decoder-only LLMs, a high MACS score means an input token receives strong maximal attention across many layers from the current query, and is therefore treated as important for the current generation step [2509.17178]. In that setting, consistency is positively associated with salience.

In the VLA formulation of inter-layer rank consistency, the interpretation is inverted for pruning control. The paper reports that when the ranking of top-attention tokens remains highly stable across layers, attention can be “locked” onto spurious features. It calls this the “spurious locking hypothesis”: low \(\tau\) corresponds to dynamic refinement and more trustworthy attention, whereas high \(\tau\) corresponds to rigid focus on redundant or misleading features [2603.24941]. The distributions for two frame types are reported as strongly separated, with an AUC of 0.91 when \(\tau\) is used to predict attention reliability [2603.24941].

In CNN attention learning, higher cross-layer consistency means that earlier and later attention maps for the same class concentrate on the same object region, which is treated as desirable because it combines fine-grained inner-layer discrimination with last-layer localization [1811.07484]. In ATCON, higher consistency indicates that different attribution mechanisms or masked/unmasked attention maps point to similar regions, which is used as an unsupervised fine-tuning objective [2210.09705].

These cases show that MACS is not inherently “good” or “bad.” It is a summary of cross-layer regularity. Whether high consistency is beneficial depends on the architecture, task, and decision variable. In LLM explanation, persistence of attention can support attribution [2509.17178]. In token pruning for VLA policies, excessive persistence can indicate spurious fixation [2603.24941]. This is one of the main sources of confusion around the term.

## 5. Algorithmic uses

MACS has been used or can be directly derived for several distinct purposes.

In decoder-only LLMs, MACS is an attribution method. It provides token-level importance scores during generation without requiring backpropagation or perturbation. The reported complexity is roughly \(O(LHN)\) per generation step, linear in layers, heads, and input length [2509.17178]. This enables streaming explanation, and the paper explicitly describes MACS as lightweight and easily deployable [2509.17178].

In vision-language-action models, the MACS-style rank-consistency statistic drives token pruning through TIES, a dynamic framework that balances attention-magnitude selection with diversity-based selection [2603.24941]. TIES first calibrates a reference distribution of \(\tau\) over sampled frames, then at inference time uses the current frame’s consistency to determine how much it should trust magnitude-based top-\(k\) pruning [2603.24941]. In Soft-TIES, the current trust weight \(w_t\) determines
\[
N_{\text{top}} = \lfloor w_t \cdot \rho N \rfloor,
\qquad
N_{\text{uni}} = \rho N - N_{\text{top}},
\]
and the final retained token set is
\[
\mathcal{T}_{\text{final}} = \mathcal{T}_{\text{top}} \cup \mathcal{T}_{\text{uni}}.
\]
Hard-TIES replaces interpolation with a threshold rule on \(\tau\) [2603.24941].

In CNN training, the MACS-like overlap statistic is used as a regularizer via \(L_{AC}\), combined with classification loss and attention separation losses:
\[
L = L_C + L^{in}_{AS} + L^{la}_{AS} + L_{AC}.
\]
This makes cross-layer attention consistency a first-class training objective rather than a post hoc score [1811.07484].

In ATCON, consistency is likewise optimized as a fine-tuning objective, but across attention maps from different explanation methods or different masked views:
\[
L_A = -\sum_{x\in X} h(A_{x,1},...,A_{x,M}),
\]
with Pearson correlation used in the main implementation [2210.09705].

## 6. Empirical results

The strongest explicit empirical results for MACS as a named method come from decoder-only LLM explanation [2509.17178]. On a SQuAD 2.0 subset, MACS attains the highest reported mAUC-PR among the compared methods:
- Random: \(0.113 \pm 0.01\)
- Rollout: \(0.147 \pm 0.02\)
- AttnLRP: \(0.565 \pm 0.03\)
- AtMan: \(0.315 \pm 0.03\)
- MACS: \(0.601 \pm 0.03\) [2509.17178]

Its faithfulness metrics are reported as statistically comparable to AttnLRP while remaining much cheaper computationally [2509.17178]. On CNN/DailyMail summarization, the efficiency table reports:
- Baseline: 16088 MB peak VRAM, 7.46 tok/s
- Rollout: 42649 MB, 3.04 tok/s
- AtMan: 16253 MB, 1.66 tok/s
- AttnLRP: 24112 MB, 3.49 tok/s
- MACS: 17998 MB, 5.69 tok/s [2509.17178]

The same paper states that MACS shows a favorable trade-off with a 22% decrease in VRAM usage and 30% reduction in latency relative to more complex techniques [2509.17178]. It also reports that in 97.68% of SQuAD samples, MACS’s best step occurs before answer tokens are generated, supporting an “anticipatory attention” interpretation [2509.17178].

In VLA pruning, TIES—built around MACS-style inter-layer rank consistency—improves both efficiency and policy success [2603.24941]. On CogACT + SIMPLER under the Visual Matching protocol, Hard-TIES with 56 retained tokens achieves 78.1% average success versus 72.7% for the unpruned 256-token baseline [2603.24941]. The same paper reports that TIES improves average success rates by 6% while reducing token usage by 78%, and that extreme pruning to \(T=28\) still remains above the full-token baseline with an 83.6% FLOPs reduction [2603.24941].

In CNN attention consistency learning, adding the consistency term improves fine-grained classification on CUB-200-2011. The ablation reported in [1811.07484] shows:
- Baseline ResNet-50: 81.7%
- \(+L^{in}_{AS}\): 85.15%
- \(+L^{in}_{AS}+L_{AC}\): 85.77%
- \(+L^{in}_{AS}+L^{la}_{AS}+L_{AC}\): 86.20%

The same work reports gains across CIFAR-100 (+3.33%), Caltech-256 (+1.64%), ILSVRC2012 (+0.92%), CUB-200-2011 (+4.8%), and PASCAL VOC2012 (+5.73%) [1811.07484].

ATCON reports that optimizing attention consistency improves both prediction and attention-map quality on small training sets, including a 6.6-point F1 lift on the authors’ video dataset, a 2.9-point F1 lift on PASCAL, and a 1.8-point mean IoU lift over Grad-CAM for weakly supervised detection on PASCAL [2210.09705].

## 7. Related ideas, misconceptions, and limitations

One common misconception is that MACS always measures “trustworthy attention.” The literature does not support that universal reading. In [2509.17178], persistent strong attention across layers is exploited as an attribution signal. In [2603.24941], persistent rank stability can instead indicate unreliable attention locked onto spurious background features. The sign of the practical interpretation is therefore task-dependent.

A second misconception is that MACS is identical to raw attention magnitude. The VLA results directly challenge that claim: high-attention tokens can degrade performance, and a controlled experiment shows “performance inversion,” where retaining Bottom-45 tokens on the Drawer task yields 77.31% success and surpasses both Top-45 retention and the full 256-token baseline [2603.24941]. The key insight is that consistency and magnitude are distinct signals.

A third misconception is that cross-layer consistency must always be defined by exact spatial overlap. That is true for the ICASC-style formulation in CNNs [1811.07484], but not for LLM MACS, which tracks per-token maximal attention across layers [2509.17178], nor for VLA pruning, which tracks Kendall rank consistency of token importance orderings [2603.24941].

The limitations are similarly domain-specific. The LLM MACS paper emphasizes that it is a heuristic based on attention consistency rather than direct input sensitivity or marginal impact, and that max-pooling may ignore useful distributed information across heads [2509.17178]. The VLA paper states that the link between inter-layer rank consistency and true token importance is empirical rather than theoretically proven, and notes that low consistency can also arise from oscillation among spurious features, while some tasks may require stable focus [2603.24941]. The CNN consistency-loss paper relies on layer resizing and reference-mask construction, which ties its measure to particular architectural and spatial assumptions [1811.07484].

A plausible implication is that future MACS variants will likely be hybrid: combining persistence, ranking stability, spatial overlap, and cross-modal alignment rather than relying on a single statistic.

## 8. Broader significance and research directions

MACS has become a useful organizing idea because it reframes attention from a one-layer visualization object into a dynamical quantity distributed across depth. In language models, this enables lightweight attribution with competitive faithfulness and substantially better efficiency than gradient-heavy methods [2509.17178]. In robotic vision-language-action systems, it yields a deployment-time control signal for token pruning that can improve both latency and success rate without retraining [2603.24941]. In CNN interpretability and weakly supervised localization, it provides regularization targets or unsupervised fine-tuning objectives that improve discriminability and reduce visual confusion [1811.07484], [2210.09705].

Several directions follow directly from the cited work. One is explicit cross-modal MACS, where consistency is measured jointly over vision and language attention pathways rather than within a single modality. The VLA paper explicitly notes that its current method focuses on visual spatial redundancy and does not yet integrate cross-modal alignment or linguistic cues into the consistency metric [2603.24941]. Another is theoretical analysis of why layer-wise attention persistence correlates with importance in some regimes and with spurious locking in others; the LLM MACS paper identifies this as an open problem [2509.17178]. A third is richer per-token or per-region trajectory analysis instead of a single aggregate scalar, an option discussed as future work in the VLA context [2603.24941].

Taken together, the literature supports a precise but plural understanding of MACS. It is not one immutable metric, but a class of multi-layer attention consistency measures whose concrete form depends on whether the target problem is attribution, pruning, regularization, or explanation alignment. What unifies these variants is the claim that attention becomes substantially more informative when its cross-layer behavior is measured directly rather than inferred from single-layer magnitude alone.

Source: https://www.emergentmind.com/topics/multi-layer-attention-consistency-score-macs