Explicitly Modeled Attention Maps

• Explicitly Modeled Attention Maps are defined attention representations that are directly supervised or parameterized, replacing implicit features with clear, interpretable maps.
• They leverage diverse methodologies—such as geometric kernels, probabilistic latent variables, and human-guided losses—to enforce spatial, temporal, or structured dependencies.
• Applications in fixation prediction, object recognition, video action detection, and tracking have shown measurable performance gains and improved model transparency.

Explicitly modeled attention maps are attention representations whose form, semantics, or supervision are made first-class components of a model rather than left as implicit by-products of generic feature learning. Across the literature, this explicitness appears in several recurrent forms: spatial maps analytically derived from CNN activations for fixation prediction and scanpath generation (Zanca et al., 2018); motion maps constructed from frame differencing and injected into tracking networks (Raj et al., 2024); single-channel attention maps directly edited and supervised by humans in an Attention Branch Network (Mitsuhara et al., 2019); actor-conditioned spatio-temporal gating tensors in video action detection (Ulutan et al., 2018); probabilistic and quantized spatial attention variables learned under an Information Bottleneck objective (Lai et al., 2021); probabilistic top-down maps based on Marginal Winning Probabilities in Excitation Backprop (Zhang et al., 2016); structured spatial masks generated autoregressively by an AttentionRNN (Khandelwal et al., 2019); classifier-activation-derived masks used as surrogate object regions in fine-grained recognition (Shen et al., 2018); class-discriminative maps explicitly regularized for separation and cross-layer consistency (Wang et al., 2018); continuous multimodal densities over image coordinates parameterized as Gaussian mixtures (Farinhas et al., 2021); and geometric, content-independent self-attention kernels with a learnable radius for image classification (Tan et al., 2020). Taken together, these works define a broad research program in which attention maps are specified, constrained, supervised, or parameterized in directly interpretable ways.

1. Definitions and recurring forms of explicitness

A common thread across these works is the treatment of attention as an explicit object with a known representation and a known role in computation. In the CNN saliency model of "Visual Attention driven by Convolutional Features" (Zanca et al., 2018), attention is a scalar field $M(x)$ computed directly from the last convolutional layer by channel averaging, then upsampled to image resolution. In "Learn To Pay Attention" (Jetley et al., 2018), attention is a spatial probability distribution $a_i^s$ over locations of an intermediate feature map, normalized by a spatial softmax and used to build a convex combination of local features. In "Embedding Human Knowledge into Deep Neural Network via Attention Map" (Mitsuhara et al., 2019), attention is a single-channel spatial map $M(\mathbf{x}) \in \mathbb{R}^{H \times W}$ that is part of the forward pass and also a direct training target. In "Information Bottleneck Approach to Spatial Attention Learning" (Lai et al., 2021), attention is a stochastic latent variable $A$ with a Gaussian parameterization and a quantized version $A_q$. In "Multimodal Continuous Visual Attention Mechanisms" (Farinhas et al., 2021), attention is a continuous density $p(x)$ over image coordinates, instantiated as a mixture of Gaussians.

A second recurring form of explicitness is architectural. Actor-Conditioned Attention Maps (ACAM) are defined as per-actor, per-location, per-channel tensors $$\mathbf{ACAM}_a \in \mathbb{R}^{T \times H \times W \times C}$$ that multiplicatively gate backbone features in video action detection (Ulutan et al., 2018). TrackNetV4 constructs motion attention maps from frame differencing, processes them with a motion prompt layer, and injects them into the decoder by a prescribed fusion rule (Raj et al., 2024). In the graph domain, Path-Aware Graph Attention defines attention between vertices as a sum over path-dependent functions of edge-type sequences, making the attention map explicitly dependent on paths rather than adjacency alone (Da et al., 2022). In 3D mesh classification, a CLS node and attention rollout expose per-node importance on tooth meshes (Buyukcakir et al., 9 Sep 2025).

A third form is supervisory. Attention maps can be optimized to resemble human edits, bubble annotations, lesion masks, or top-down perceptual signals. ABN-based human knowledge injection uses an explicit map loss $$L_{\text{map}}(\mathbf{x}_i) = \gamma \,\| M'(\mathbf{x}_i) - M(\mathbf{x}_i) \|_2$$ (Mitsuhara et al., 2019). ClickMe-supervised GALA networks compare normalized model attention maps against human-derived top-down maps (Linsley et al., 2018). ICASC introduces losses directly on gradient-based attention maps to enforce class separation and cross-layer consistency (Wang et al., 2018).

These formulations differ from implicit attention mechanisms in which attention-like behavior is distributed across many weights or emerges only in post hoc visualizations. This suggests that explicit modeling is less a single mechanism than a family of design choices that make attention maps spatially interpretable, mathematically specified, and operational in learning or inference.

2. Parameterizations of attention maps

Several papers make attention explicit by prescribing a concrete parametric form. In the geometric self-attention model "Explicitly Modeled Attention Maps for Image Classification" (Tan et al., 2020), the attention matrix is defined as $A = Norm(G+1)$, where $G \in \mathbb{R}^{HW \times HW}$ is a geometric kernel depending only on relative position, and $a_i^s$0 is row-wise normalization. The main learnable form is a Gaussian kernel

$a_i^s$1

with a single learnable scalar radius $a_i^s$2 per ExpAtt layer (Tan et al., 2020). The same work also studies fixed constant, linear, cosine, and exponential distance-based kernels, all content-independent and shared across heads.

The probabilistic IB framework parameterizes attention as

$a_i^s$3

with a quantized version $a_i^s$4 obtained by assigning each attention value to the nearest learnable anchor $a_i^s$5 (Lai et al., 2021). This makes both the distribution over attention maps and the codebook of attention values explicit.

Continuous multimodal attention uses a density

$a_i^s$6

where $a_i^s$7, $a_i^s$8, and $a_i^s$9 have direct geometric interpretations as centers, extents, and weights of attention modes (Farinhas et al., 2021). The resulting context vector is the linear combination $M(\mathbf{x}) \in \mathbb{R}^{H \times W}$0, where each $M(\mathbf{x}) \in \mathbb{R}^{H \times W}$1 is a unimodal continuous-attention context (Farinhas et al., 2021).

Excitation Backprop defines attention probabilistically over a CNN graph. Given a top-down prior $M(\mathbf{x}) \in \mathbb{R}^{H \times W}$2, the Marginal Winning Probability satisfies

$M(\mathbf{x}) \in \mathbb{R}^{H \times W}$3

and the conditional winning probability over excitatory connections is

$M(\mathbf{x}) \in \mathbb{R}^{H \times W}$4

The resulting spatial maps are probability histograms over locations rather than heuristic saliency overlays (Zhang et al., 2016).

Structured attention can also be explicit in dependency form. AttentionRNN models

$M(\mathbf{x}) \in \mathbb{R}^{H \times W}$5

and approximates each scalar attention variable by a Gaussian whose parameters are predicted from bi-directional diagonal LSTMs (Khandelwal et al., 2019). Here explicitness lies in the factorization of the joint distribution over the mask itself.

3. Construction from activations, features, and motion cues

One major strand constructs attention maps analytically from intermediate activations. In (Zanca et al., 2018), if the last convolutional layer produces feature maps $M(\mathbf{x}) \in \mathbb{R}^{H \times W}$6, a class-agnostic semantic attention map is defined by averaging channels: $M(\mathbf{x}) \in \mathbb{R}^{H \times W}$7 This map is derived directly from an ImageNet-trained Inception-v3 classifier, without saliency-specific supervision, then upsampled by cubic interpolation (Zanca et al., 2018). The paper explicitly interprets $M(\mathbf{x}) \in \mathbb{R}^{H \times W}$8 as a generic objectness or semantic saliency map suitable for free viewing.

A related but distinct construction appears in fine-grained recognition (Shen et al., 2018). There, multiple local classifiers produce activation volumes $M(\mathbf{x}) \in \mathbb{R}^{H \times W}$9. Aggregating over classifiers by max-pooling yields $A$0, and the single-channel attention map is then

$A$1

that is, the maximum over non-background classes at each location. After Otsu thresholding, the binary map $A$2 becomes a surrogate object mask used to supervise local classification and to extract object-level features (Shen et al., 2018). Attention is therefore not post hoc: it is defined from classifier activations and recursively used to train those same classifiers.

In sports tracking (Raj et al., 2024), motion attention maps are explicitly built from frame differencing. For grayscale frames $A$3, consecutive differences are

$A$4

and the paper uses absolute differences $A$5. These are passed through a parametric motion prompt layer,

$A$6

with only two learnable parameters, slope and shift, to obtain motion attention maps $A$7 (Raj et al., 2024). The construction is explicitly tied to a physically interpretable motion operator rather than latent feature similarity.

In actor-conditioned action detection, attention is formed by combining actor features and context embeddings. Given an actor representation $A$8 and context embedding $A$9, a relation tensor is defined as

$A_q$0

then converted to attention via a sigmoid: $A_q$1 The resulting attention maps multiplicatively gate the original feature map $A_q$2 to produce actor-conditioned features (Ulutan et al., 2018).

These examples define a spectrum. Some explicit maps are analytic functions of features (Zanca et al., 2018, Shen et al., 2018), some are geometric kernels (Tan et al., 2020), some are probabilistic latent variables (Lai et al., 2021, Farinhas et al., 2021), and some are derived from hand-specified physical operators such as frame differencing (Raj et al., 2024). A plausible implication is that explicitness often coincides with a deliberately narrow inductive bias.

4. Supervision, regularization, and human alignment

Explicit attention maps are often supervised directly rather than inferred only through task loss. The ABN-based framework (Mitsuhara et al., 2019) is the clearest instance. A human-edited target map $A_q$3 is aligned to the model’s internal map $A_q$4 by

$A_q$5

The feature extractor is frozen, and only the attention and perception branches are fine-tuned so that future attention maps reflect the edited or derived human priors (Mitsuhara et al., 2019). On ImageNet, CUB-200-2010, and IDRiD, this produces lower MSE to human or expert maps and improved task performance (Mitsuhara et al., 2019).

ClickMe supervision makes attention a large-scale human-aligned auxiliary task. GALA networks generate explicit attention volumes at several intermediate layers, and the training loss compares model attention maps against resized, normalized ClickMe maps derived from human gameplay (Linsley et al., 2018). The paper reports that GALA-ResNet-50 with ClickMe reaches 88.56 of human-human reliability in attention-map similarity, compared with 64.21 for GALA-ResNet-50 without ClickMe and 43.61 for ResNet-50 (Linsley et al., 2018). It also reports improved classification on both the ClickMe split and full ILSVRC12 (Linsley et al., 2018).

ICASC uses no external human maps, but it does impose explicit map-level constraints. Attention separation penalizes overlap between the target-class attention $A_q$6 and the most confusing class attention $A_q$7: $A_q$8 and cross-layer consistency encourages inner-layer attention to lie within the target region defined by a last-layer attention mask (Wang et al., 2018). This work effectively treats attention maps as trainable class-discriminative objects.

Attention can also be constrained by information theory rather than by external annotations. In AIB, the objective

$A_q$9

encourages the attention-modulated latent representation $p(x)$0 to preserve label information while compressing input-plus-attention information (Lai et al., 2021). Additional quantization and commitment losses on $p(x)$1 regulate the attention coding channel (Lai et al., 2021). This suggests a different notion of explicitness: attention is not supervised to match a human heatmap, but it is still governed by its own probabilistic and information-theoretic structure.

5. Dynamics, scanpaths, and structured dependencies

A distinct line of work treats explicit attention maps not merely as static saliency outputs but as components in dynamic or structured systems. In (Zanca et al., 2018), the CNN-derived saliency map $p(x)$2 is inserted into the EYMOL eye-movement model by augmenting the generalized potential: $p(x)$3 which yields an additional force term $p(x)$4 in the equations of motion (Zanca et al., 2018). Attention thereby becomes a top-down term in a variational dynamical system that generates scanpaths, not only fixation maps. The resulting CF-EYMOL model improves over the bottom-up dynamical baseline on CAT2000 and substantially outperforms random and center baselines in scanpath similarity metrics (Zanca et al., 2018).

AttentionRNN makes structure explicit by modeling dependencies among attention variables themselves. Rather than predicting each spatial weight independently, it uses diagonal bi-directional LSTMs so that each $p(x)$5 depends on previous attention values under a rasterized factorization (Khandelwal et al., 2019). This yields more coherent masks for attribute prediction, VQA, and image generation (Khandelwal et al., 2019). The work emphasizes that ordinary local or global attention layers often lack explicit spatial dependency modeling.

Path-Aware Graph Attention generalizes explicit attention to heterogeneous HD-map graphs. Attention between vertices $p(x)$6 and $p(x)$7 is defined as

$p(x)$8

where paths are ordered sequences of typed edges and $p(x)$9 is implemented with an LSTM over edge features (Da et al., 2022). The resulting attention map is not a function only of adjacency or node features; it is explicitly a function of paths. This is a particularly strong version of explicit modeling because it elevates domain structure—lane connectivity semantics—into the definition of attention itself.

In 3D mesh classification, CGAT uses a CLS node so that graph-level predictions must flow through an attention bottleneck, and attention rollout provides node-level explanations on the tooth surface (Buyukcakir et al., 9 Sep 2025). The final interpretability map is the rolled-out CLS-to-node attention rather than an arbitrary gradient saliency. This suggests that explicit attention maps can be transferred to domains in which spatial locations are graph vertices rather than image pixels.

6. Empirical patterns, benefits, and limitations

The empirical record across these works is mixed in methodology but consistent in one respect: explicit attention maps often improve interpretability and frequently improve task performance.

A compact summary of representative results is useful.

Work	Explicit map form	Reported effect
(Zanca et al., 2018)	CNN feature-average saliency $$\mathbf{ACAM}_a \in \mathbb{R}^{T \times H \times W \times C}$$0	CF raw: AUC = 0.80, NSS = 1.177; competitive with classical saliency and useful in scanpath modeling
(Raj et al., 2024)	Motion attention from frame differencing + prompt layer	Improves TrackNetV2/V3 on tennis and shuttlecock benchmarks; gains in recall/F1 and fewer false negatives
(Mitsuhara et al., 2019)	Human-edited single-channel attention maps in ABN	Improves classification and lowers MSE to human/expert maps on ImageNet, CUB, IDRiD
(Ulutan et al., 2018)	Actor-conditioned spatio-temporal gating tensors	Improves AVA action detection from 19.83 mAP to 23.29 mAP over RoI+Tail baseline
(Lai et al., 2021)	Variational and quantized spatial attention	Improves error rates and interpretability consistency scores over several baselines
(Zhang et al., 2016)	Probabilistic top-down attention via MWP and c-MWP	Strong Pointing Game performance; c-MWP especially strong on difficult subsets
(Shen et al., 2018)	Attention from classifier activations	87.9% on CUB-200-2011, 94.1% on Stanford Cars, 92.1% on FGVC-Aircraft
(Wang et al., 2018)	Class-discriminative gradient attention with explicit losses	Accuracy gains including CIFAR-100 (+3.33%), CUB-200-2011 (+4.8%), PASCAL VOC2012 (+5.73%)
(Farinhas et al., 2021)	Gaussian-mixture continuous attention density	Similar VQA accuracy to discrete attention but lower JS divergence to human attention
(Tan et al., 2020)	Geometric self-attention maps with learnable radius	Up to 2.2% over ResNet baselines; better than AA-ResNet152 by 0.9% with fewer params and GFLOPs

The interpretation of these gains depends on the setting. In fixation prediction (Zanca et al., 2018), explicitness means semantic saliency without saliency-specific training. In tracking (Raj et al., 2024), explicitness yields motion-aware robustness in low-visibility or occluded conditions. In human-supervised ABN (Mitsuhara et al., 2019) and ClickMe GALA (Linsley et al., 2018), explicitness supports human-machine cooperation and map-level interpretability. In ICASC (Wang et al., 2018), explicitness reduces visual confusion among classes. In ExpAtt (Tan et al., 2020), explicitness mainly buys efficiency and a strong inductive bias.

Limitations are equally recurrent. Many explicit maps inherit the assumptions built into their parameterization. Geometric kernels assume nearby pixels are generally more relevant (Tan et al., 2020). Frame differencing conflates object and camera motion (Raj et al., 2024). Human-edited attention may be costly or inconsistent (Mitsuhara et al., 2019, Linsley et al., 2018). Quantized or Gaussian attention may be too rigid for highly irregular object geometry (Lai et al., 2021, Farinhas et al., 2021). Contrastive attention maps can emphasize discriminative parts rather than full object extent, which is useful for pointing but suboptimal for box coverage (Zhang et al., 2016). A plausible implication is that explicit attention maps trade representational flexibility for transparency and control.

7. Relation to implicit attention, CAM-style explanations, and modern transformer practice

Several of these works explicitly contrast themselves with post hoc or implicit attention. Excitation Backprop distinguishes Marginal Winning Probabilities from gradients, deconvolution, and LRP by giving a normalized probabilistic interpretation (Zhang et al., 2016). ABN-based human knowledge injection contrasts its single-channel editable maps with multi-channel or channel-wise attention mechanisms that are harder to manipulate (Mitsuhara et al., 2019). ClickMe-supervised GALA networks position attention as a supervised module rather than a post hoc localization device (Linsley et al., 2018). The fine-grained classifier-activation method avoids specialized ranking or reinforcement-learning objectives by deriving attention directly from local classifier outputs and training it under standard softmax losses (Shen et al., 2018).

The relation to transformer-style attention is more nuanced. ExpAtt (Tan et al., 2020) is explicitly presented as an alternative to standard self-attention, replacing learned query-key interactions with content-independent geometric kernels. SCRAM (Calian et al., 2019), while focused on approximation, also makes a structural claim: image attention maps are sparse and spatially coherent, so one can explicitly model their support via PatchMatch-discovered modes and local neighborhoods. MAPS (Elhelo et al., 2024) reframes LLM attention heads as explicit vocabulary-to-vocabulary maps $$\mathbf{ACAM}_a \in \mathbb{R}^{T \times H \times W \times C}$$1, inferring head functions from parameters without any inference-time activations. Although MAPS addresses transformer attention heads rather than spatial maps, it extends the notion of explicit attention maps into function space: a head becomes an explicit operator over token pairs (Elhelo et al., 2024).

This suggests that explicitness can arise at multiple levels: the spatial map itself, the dynamic law into which it is inserted, the probabilistic family from which it is drawn, the supervisory signal against which it is trained, or the parameter-space operator from which it is inferred. The term does not imply a single family of architectures; it denotes a design philosophy that opposes opaque or purely emergent attention.

8. Broader significance and open directions

The literature points to several durable themes. First, explicit attention maps frequently improve scientific legibility. In medical and forensic contexts, CGAT’s attention rollout over tooth meshes is presented as a way to support expert validation (Buyukcakir et al., 9 Sep 2025). In human-in-the-loop classification, edited ABN maps allow direct intervention in what the model attends to (Mitsuhara et al., 2019). In XAIxArts, extracted cross-attention volumes in video diffusion become analyzable and even artistically reusable objects (Cole et al., 30 Aug 2025). This suggests that explicit maps can serve not only prediction but also audit, control, and creative reuse.

Second, explicit maps often function as inductive biases. The geometric prior in ExpAtt (Tan et al., 2020), the motion prior in TrackNetV4 (Raj et al., 2024), the top-down semantic prior in CNN activation saliency (Zanca et al., 2018), and the path prior in HD-map attention (Da et al., 2022) each encode a structured assumption about the domain. Where those assumptions are well matched to the task, the resulting models can be both simpler and better.

Third, explicitness does not require a single implementation recipe. Some methods use deterministic formulas (Zanca et al., 2018, Shen et al., 2018, Tan et al., 2020), some use differentiable latent-variable models (Lai et al., 2021, Farinhas et al., 2021), some use map-level losses (Mitsuhara et al., 2019, Wang et al., 2018, Linsley et al., 2018), and some use explicit extraction or rollout of preexisting attention tensors (Cole et al., 30 Aug 2025, Buyukcakir et al., 9 Sep 2025). What unifies them is that the attention map is no longer merely an incidental tensor buried in the network.

Finally, several open questions recur implicitly across the surveyed works. One concerns scalability: human supervision of attention is effective but expensive (Mitsuhara et al., 2019, Linsley et al., 2018). Another concerns fidelity: attention maps can be visually appealing without fully capturing causal computation, a concern that motivates probabilistic, rollout-based, or ablation-validated approaches (Zhang et al., 2016, Elhelo et al., 2024, Buyukcakir et al., 9 Sep 2025). A third concerns domain transfer: explicit priors such as local geometric decay (Tan et al., 2020) or frame differencing (Raj et al., 2024) are powerful when appropriate, but potentially brittle outside their intended regime. This suggests that future work will likely continue to combine explicit map parameterization with more adaptive or hybrid forms of learning, while preserving the interpretability benefits that motivated the line of research in the first place.