Multi-Layer Attention Consistency Score (MACS)

• MACS is a framework that quantifies the stability of attention signals across neural network layers by aggregating maximal per-token scores.
• It is applied in diverse domains—decoder-only LLMs, vision-language-action models, and CNNs—to guide attribution, pruning, and regularization with improved efficiency.
• Empirical studies show that MACS can reduce computational overhead and enhance explanation fidelity compared to traditional dense attention aggregation methods.

Searching arXiv for the primary and related papers to ground the article in current literature. Multi-Layer Attention Consistency Score (MACS) denotes a family of quantitative constructs that measure how stable or coherent attention-related signals remain across multiple layers of a neural network. The term is used explicitly in the context of decoder-only LLMs as a heuristic attribution method based on the consistency of maximal attention across layers (Lan et al., 21 Sep 2025), and it can also be formalized from closely related ideas in vision, vision-language-action, and explainability research where cross-layer attention consistency is central but not always named MACS (Liu et al., 26 Mar 2026, Wang et al., 2018, Mirzazadeh et al., 2022). Across these settings, MACS serves either as an attribution score, a pruning-control signal, an attention regularizer, or an evaluation statistic. The unifying principle is that attention becomes more informative when its multi-layer behavior is analyzed rather than treated as a single-layer magnitude snapshot.

1. Terminological scope and conceptual definition

MACS is not a single universally standardized object across the literature. The explicit formulation appears in “Attention Consistency for LLMs Explanation” (Lan et al., 21 Sep 2025), where MACS is defined for decoder-based models as a token-importance heuristic derived from the consistency of maximal attention across heads and layers. In other papers, the same underlying concept appears under different names: “inter-layer token ranking consistency” in vision-language-action models (Liu et al., 26 Mar 2026), “attention consistency loss” in CNN attention learning (Wang et al., 2018), and “attention consistency” as a correlation objective across explanation maps in vision models (Mirzazadeh et al., 2022).

In the decoder-only LLM setting, MACS measures whether an input token repeatedly receives strong attention from the current output query across many layers. This makes it a per-token scalar attribution score rather than a training loss (Lan et al., 21 Sep 2025). In vision-language-action models, the analogous signal is not per-token attribution but a frame-level estimate of whether attention magnitude is trustworthy, obtained from the Kendall rank correlation of token rankings across layers (Liu et al., 26 Mar 2026). In CNN explainability, consistency is instead defined as the fraction of inner-layer attention that falls inside a mask derived from last-layer attention for the same class (Wang et al., 2018). These formulations differ operationally, but all instantiate multi-layer consistency as a criterion for judging attention reliability.

A plausible implication is that MACS is best understood as a research category rather than a single formula: any score that compresses cross-layer agreement of attention, attention-derived rankings, or attention maps into a scalar diagnostic fits the concept, provided its precise form is specified.

2. Canonical decoder-only formulation

The clearest formal definition of MACS is given for decoder-only Transformers in (Lan et al., 21 Sep 2025). Let the input tokens be $X=\{x_1,\dots,x_N\}$, let $t_1,\dots,t_{k-1}$ be the already generated tokens, and let the generation step be $k$, with query position $u=n-1=N+k-1$. For layer $l$ and head $h$, the self-attention vector from the current query is

$\mathbf{a}^{(n-1,l,h)} \in \mathbb{R}^{n-1}.$

This attention is split into attention to input tokens and attention to previous outputs. The method then defines a redistributed attention vector over inputs: $$(\mathbf{a}_{R}^{(n-1,l,h)})_i = a_{I,i}^{(n-1,l,h)} + \frac{1}{N} \sum_{p=1}^{k-1} a_{O,p}^{(n-1,l,h)},$$ which adds a uniform share of attention to previously generated tokens back to each input token (Lan et al., 21 Sep 2025).

For each layer, MACS retains only the strongest per-token signal across heads: $$(\mathbf{m}'_{l})_i = \max_{h \in \{1,\dots,H\}} \left( (\mathbf{a}_{R}^{(n-1,l,h)})_i \right).$$ This max-pooling step makes MACS a “maximal attention” method rather than a full-path aggregation method (Lan et al., 21 Sep 2025).

To avoid multiplicative collapse, a floor vector is introduced: $$\mathbf{m}_l = \alpha \mathbf{m}'_l + (1-\alpha)\mathbf{1}^N,$$ with default $t_1,\dots,t_{k-1}$0 (Lan et al., 21 Sep 2025). Multi-layer consistency is then accumulated by Hadamard products: $t_1,\dots,t_{k-1}$1 The raw MACS score for input token $t_1,\dots,t_{k-1}$2 is $t_1,\dots,t_{k-1}$3. Finally, per-step scores are standardized with a Z-score: $t_1,\dots,t_{k-1}$4 These $t_1,\dots,t_{k-1}$5 are the final attribution scores at generation step $t_1,\dots,t_{k-1}$6 (Lan et al., 21 Sep 2025).

The central intuition is that transient attention spikes are suppressed by multiplication across layers, whereas tokens that repeatedly receive strong maximal attention remain salient. This design is explicitly positioned against dense attention aggregation such as Attention Rollout, which multiplies full attention matrices and therefore accumulates many weak or noisy paths (Lan et al., 21 Sep 2025).

3. Alternative formalizations in adjacent domains

A distinct MACS-style construct is recoverable from “Beyond Attention Magnitude: Leveraging Inter-layer Rank Consistency for Efficient Vision-Language-Action Models” (Liu et al., 26 Mar 2026). There, token importance at layer $t_1,\dots,t_{k-1}$7 is first defined by attention magnitude: $t_1,\dots,t_{k-1}$8 where $t_1,\dots,t_{k-1}$9 is the attention matrix. Tokens are then ranked by $k$0, and inter-layer consistency is measured with Kendall’s coefficient: $k$1 A frame-level consistency statistic is obtained by averaging across successive layers: $k$2 Although the paper names this quantity $k$3 rather than MACS, the details explicitly identify it as an inter-layer attention rank consistency measure that can be reconstructed as a MACS-style score (Liu et al., 26 Mar 2026).

In CNN attention learning, “Sharpen Focus: Learning with Attention Separability and Consistency” defines attention consistency through overlap between an inner-layer attention map and a mask derived from last-layer attention (Wang et al., 2018). If $k$4 is the inner-layer attention map and $k$5 is the soft region mask derived from last-layer ground-truth attention, then

$k$6

with $k$7 empirically (Wang et al., 2018). The normalized overlap term inside this loss is already a layer-pair consistency score in $k$8, and the paper’s details support generalizing it across multiple layers as a MACS-style statistic (Wang et al., 2018).

In ATCON, attention consistency is not between layers but between attribution maps or between original and masked-input attention maps, optimized via Pearson correlation (Mirzazadeh et al., 2022). This is methodologically relevant because it demonstrates that consistency may be defined directly through similarity of attention maps: $k$9 A plausible implication is that MACS can be instantiated either structurally, through ranking agreement or overlap, or geometrically, through correlation of aligned attention maps (Mirzazadeh et al., 2022).

4. Interpretation: what MACS measures

The meaning of a high or low MACS depends on the operational definition. In decoder-only LLMs, a high MACS score means an input token receives strong maximal attention across many layers from the current query, and is therefore treated as important for the current generation step (Lan et al., 21 Sep 2025). In that setting, consistency is positively associated with salience.

In the VLA formulation of inter-layer rank consistency, the interpretation is inverted for pruning control. The paper reports that when the ranking of top-attention tokens remains highly stable across layers, attention can be “locked” onto spurious features. It calls this the “spurious locking hypothesis”: low $u=n-1=N+k-1$0 corresponds to dynamic refinement and more trustworthy attention, whereas high $u=n-1=N+k-1$1 corresponds to rigid focus on redundant or misleading features (Liu et al., 26 Mar 2026). The distributions for two frame types are reported as strongly separated, with an AUC of 0.91 when $u=n-1=N+k-1$2 is used to predict attention reliability (Liu et al., 26 Mar 2026).

In CNN attention learning, higher cross-layer consistency means that earlier and later attention maps for the same class concentrate on the same object region, which is treated as desirable because it combines fine-grained inner-layer discrimination with last-layer localization (Wang et al., 2018). In ATCON, higher consistency indicates that different attribution mechanisms or masked/unmasked attention maps point to similar regions, which is used as an unsupervised fine-tuning objective (Mirzazadeh et al., 2022).

These cases show that MACS is not inherently “good” or “bad.” It is a summary of cross-layer regularity. Whether high consistency is beneficial depends on the architecture, task, and decision variable. In LLM explanation, persistence of attention can support attribution (Lan et al., 21 Sep 2025). In token pruning for VLA policies, excessive persistence can indicate spurious fixation (Liu et al., 26 Mar 2026). This is one of the main sources of confusion around the term.

5. Algorithmic uses

MACS has been used or can be directly derived for several distinct purposes.

In decoder-only LLMs, MACS is an attribution method. It provides token-level importance scores during generation without requiring backpropagation or perturbation. The reported complexity is roughly $u=n-1=N+k-1$3 per generation step, linear in layers, heads, and input length (Lan et al., 21 Sep 2025). This enables streaming explanation, and the paper explicitly describes MACS as lightweight and easily deployable (Lan et al., 21 Sep 2025).

In vision-language-action models, the MACS-style rank-consistency statistic drives token pruning through TIES, a dynamic framework that balances attention-magnitude selection with diversity-based selection (Liu et al., 26 Mar 2026). TIES first calibrates a reference distribution of $u=n-1=N+k-1$4 over sampled frames, then at inference time uses the current frame’s consistency to determine how much it should trust magnitude-based top-$u=n-1=N+k-1$5 pruning (Liu et al., 26 Mar 2026). In Soft-TIES, the current trust weight $u=n-1=N+k-1$6 determines

$u=n-1=N+k-1$7

and the final retained token set is

$u=n-1=N+k-1$8

Hard-TIES replaces interpolation with a threshold rule on $u=n-1=N+k-1$9 (Liu et al., 26 Mar 2026).

In CNN training, the MACS-like overlap statistic is used as a regularizer via $l$0, combined with classification loss and attention separation losses: $l$1 This makes cross-layer attention consistency a first-class training objective rather than a post hoc score (Wang et al., 2018).

In ATCON, consistency is likewise optimized as a fine-tuning objective, but across attention maps from different explanation methods or different masked views: $l$2 with Pearson correlation used in the main implementation (Mirzazadeh et al., 2022).

6. Empirical results

The strongest explicit empirical results for MACS as a named method come from decoder-only LLM explanation (Lan et al., 21 Sep 2025). On a SQuAD 2.0 subset, MACS attains the highest reported mAUC-PR among the compared methods:

• Random: $l$3
• Rollout: $l$4
• AttnLRP: $l$5
• AtMan: $l$6
• MACS: $l$7 (Lan et al., 21 Sep 2025)

Its faithfulness metrics are reported as statistically comparable to AttnLRP while remaining much cheaper computationally (Lan et al., 21 Sep 2025). On CNN/DailyMail summarization, the efficiency table reports:

• Baseline: 16088 MB peak VRAM, 7.46 tok/s
• Rollout: 42649 MB, 3.04 tok/s
• AtMan: 16253 MB, 1.66 tok/s
• AttnLRP: 24112 MB, 3.49 tok/s
• MACS: 17998 MB, 5.69 tok/s (Lan et al., 21 Sep 2025)

The same paper states that MACS shows a favorable trade-off with a 22% decrease in VRAM usage and 30% reduction in latency relative to more complex techniques (Lan et al., 21 Sep 2025). It also reports that in 97.68% of SQuAD samples, MACS’s best step occurs before answer tokens are generated, supporting an “anticipatory attention” interpretation (Lan et al., 21 Sep 2025).

In VLA pruning, TIES—built around MACS-style inter-layer rank consistency—improves both efficiency and policy success (Liu et al., 26 Mar 2026). On CogACT + SIMPLER under the Visual Matching protocol, Hard-TIES with 56 retained tokens achieves 78.1% average success versus 72.7% for the unpruned 256-token baseline (Liu et al., 26 Mar 2026). The same paper reports that TIES improves average success rates by 6% while reducing token usage by 78%, and that extreme pruning to $l$8 still remains above the full-token baseline with an 83.6% FLOPs reduction (Liu et al., 26 Mar 2026).

In CNN attention consistency learning, adding the consistency term improves fine-grained classification on CUB-200-2011. The ablation reported in (Wang et al., 2018) shows:

• Baseline ResNet-50: 81.7%
• $l$9: 85.15%
• $h$0: 85.77%
• $h$1: 86.20%

The same work reports gains across CIFAR-100 (+3.33%), Caltech-256 (+1.64%), ILSVRC2012 (+0.92%), CUB-200-2011 (+4.8%), and PASCAL VOC2012 (+5.73%) (Wang et al., 2018).

ATCON reports that optimizing attention consistency improves both prediction and attention-map quality on small training sets, including a 6.6-point F1 lift on the authors’ video dataset, a 2.9-point F1 lift on PASCAL, and a 1.8-point mean IoU lift over Grad-CAM for weakly supervised detection on PASCAL (Mirzazadeh et al., 2022).

7. Related ideas, misconceptions, and limitations

One common misconception is that MACS always measures “trustworthy attention.” The literature does not support that universal reading. In (Lan et al., 21 Sep 2025), persistent strong attention across layers is exploited as an attribution signal. In (Liu et al., 26 Mar 2026), persistent rank stability can instead indicate unreliable attention locked onto spurious background features. The sign of the practical interpretation is therefore task-dependent.

A second misconception is that MACS is identical to raw attention magnitude. The VLA results directly challenge that claim: high-attention tokens can degrade performance, and a controlled experiment shows “performance inversion,” where retaining Bottom-45 tokens on the Drawer task yields 77.31% success and surpasses both Top-45 retention and the full 256-token baseline (Liu et al., 26 Mar 2026). The key insight is that consistency and magnitude are distinct signals.

A third misconception is that cross-layer consistency must always be defined by exact spatial overlap. That is true for the ICASC-style formulation in CNNs (Wang et al., 2018), but not for LLM MACS, which tracks per-token maximal attention across layers (Lan et al., 21 Sep 2025), nor for VLA pruning, which tracks Kendall rank consistency of token importance orderings (Liu et al., 26 Mar 2026).

The limitations are similarly domain-specific. The LLM MACS paper emphasizes that it is a heuristic based on attention consistency rather than direct input sensitivity or marginal impact, and that max-pooling may ignore useful distributed information across heads (Lan et al., 21 Sep 2025). The VLA paper states that the link between inter-layer rank consistency and true token importance is empirical rather than theoretically proven, and notes that low consistency can also arise from oscillation among spurious features, while some tasks may require stable focus (Liu et al., 26 Mar 2026). The CNN consistency-loss paper relies on layer resizing and reference-mask construction, which ties its measure to particular architectural and spatial assumptions (Wang et al., 2018).

A plausible implication is that future MACS variants will likely be hybrid: combining persistence, ranking stability, spatial overlap, and cross-modal alignment rather than relying on a single statistic.

8. Broader significance and research directions

MACS has become a useful organizing idea because it reframes attention from a one-layer visualization object into a dynamical quantity distributed across depth. In LLMs, this enables lightweight attribution with competitive faithfulness and substantially better efficiency than gradient-heavy methods (Lan et al., 21 Sep 2025). In robotic vision-language-action systems, it yields a deployment-time control signal for token pruning that can improve both latency and success rate without retraining (Liu et al., 26 Mar 2026). In CNN interpretability and weakly supervised localization, it provides regularization targets or unsupervised fine-tuning objectives that improve discriminability and reduce visual confusion (Wang et al., 2018, Mirzazadeh et al., 2022).

Several directions follow directly from the cited work. One is explicit cross-modal MACS, where consistency is measured jointly over vision and language attention pathways rather than within a single modality. The VLA paper explicitly notes that its current method focuses on visual spatial redundancy and does not yet integrate cross-modal alignment or linguistic cues into the consistency metric (Liu et al., 26 Mar 2026). Another is theoretical analysis of why layer-wise attention persistence correlates with importance in some regimes and with spurious locking in others; the LLM MACS paper identifies this as an open problem (Lan et al., 21 Sep 2025). A third is richer per-token or per-region trajectory analysis instead of a single aggregate scalar, an option discussed as future work in the VLA context (Liu et al., 26 Mar 2026).

Taken together, the literature supports a precise but plural understanding of MACS. It is not one immutable metric, but a class of multi-layer attention consistency measures whose concrete form depends on whether the target problem is attribution, pruning, regularization, or explanation alignment. What unifies these variants is the claim that attention becomes substantially more informative when its cross-layer behavior is measured directly rather than inferred from single-layer magnitude alone.