---
title: Explicitly Modeled Attention Maps
url: https://www.emergentmind.com/topics/explicitly-modeled-attention-maps
type: topic
---

# Explicitly Modeled Attention Maps

Explicitly modeled attention maps are attention representations whose form, semantics, or supervision are made first-class components of a model rather than left as implicit by-products of generic feature learning. Across the literature, this explicitness appears in several recurrent forms: spatial maps analytically derived from CNN activations for fixation prediction and scanpath generation [1807.10576]; motion maps constructed from frame differencing and injected into tracking networks [2409.14543]; single-channel attention maps directly edited and supervised by humans in an Attention Branch Network [1905.03540]; actor-conditioned spatio-temporal gating tensors in video action detection [1812.11631]; probabilistic and quantized spatial attention variables learned under an Information Bottleneck objective [2108.03418]; probabilistic top-down maps based on Marginal Winning Probabilities in Excitation Backprop [1608.00507]; structured spatial masks generated autoregressively by an AttentionRNN [1905.09400]; classifier-activation-derived masks used as surrogate object regions in fine-grained recognition [1811.10770]; class-discriminative maps explicitly regularized for separation and cross-layer consistency [1811.07484]; continuous multimodal densities over image coordinates parameterized as Gaussian mixtures [2104.03046]; and geometric, content-independent self-attention kernels with a learnable radius for image classification [2006.07872]. Taken together, these works define a broad research program in which attention maps are specified, constrained, supervised, or parameterized in directly interpretable ways.

## 1. Definitions and recurring forms of explicitness

A common thread across these works is the treatment of attention as an explicit object with a known representation and a known role in computation. In the CNN saliency model of "Visual Attention driven by Convolutional Features" [1807.10576], attention is a scalar field \(M(x)\) computed directly from the last convolutional layer by channel averaging, then upsampled to image resolution. In "Learn To Pay Attention" [1804.02391], attention is a spatial probability distribution \(a_i^s\) over locations of an intermediate feature map, normalized by a spatial softmax and used to build a convex combination of local features. In "Embedding Human Knowledge into Deep Neural Network via Attention Map" [1905.03540], attention is a single-channel spatial map \(M(\mathbf{x}) \in \mathbb{R}^{H \times W}\) that is part of the forward pass and also a direct training target. In "Information Bottleneck Approach to Spatial Attention Learning" [2108.03418], attention is a stochastic latent variable \(A\) with a Gaussian parameterization and a quantized version \(A_q\). In "Multimodal Continuous Visual Attention Mechanisms" [2104.03046], attention is a continuous density \(p(x)\) over image coordinates, instantiated as a mixture of Gaussians.

A second recurring form of explicitness is architectural. Actor-Conditioned Attention Maps (ACAM) are defined as per-actor, per-location, per-channel tensors \(\mathbf{ACAM}_a \in \mathbb{R}^{T \times H \times W \times C}\) that multiplicatively gate backbone features in video action detection [1812.11631]. TrackNetV4 constructs motion attention maps from frame differencing, processes them with a motion prompt layer, and injects them into the decoder by a prescribed fusion rule [2409.14543]. In the graph domain, Path-Aware Graph Attention defines attention between vertices as a sum over path-dependent functions of edge-type sequences, making the attention map explicitly dependent on paths rather than adjacency alone [2202.13772]. In 3D mesh classification, a CLS node and attention rollout expose per-node importance on tooth meshes [2509.07581].

A third form is supervisory. Attention maps can be optimized to resemble human edits, bubble annotations, lesion masks, or top-down perceptual signals. ABN-based human knowledge injection uses an explicit map loss \(L_{\text{map}}(\mathbf{x}_i) = \gamma \,\| M'(\mathbf{x}_i) - M(\mathbf{x}_i) \|_2\) [1905.03540]. ClickMe-supervised GALA networks compare normalized model attention maps against human-derived top-down maps [1805.08819]. ICASC introduces losses directly on gradient-based attention maps to enforce class separation and cross-layer consistency [1811.07484].

These formulations differ from implicit attention mechanisms in which attention-like behavior is distributed across many weights or emerges only in post hoc visualizations. This suggests that explicit modeling is less a single mechanism than a family of design choices that make attention maps spatially interpretable, mathematically specified, and operational in learning or inference.

## 2. Parameterizations of attention maps

Several papers make attention explicit by prescribing a concrete parametric form. In the geometric self-attention model "Explicitly Modeled Attention Maps for Image Classification" [2006.07872], the attention matrix is defined as \(A = Norm(G+1)\), where \(G \in \mathbb{R}^{HW \times HW}\) is a geometric kernel depending only on relative position, and \(Norm(\cdot)\) is row-wise normalization. The main learnable form is a Gaussian kernel
\[
G_{ij} = \exp\left( \frac{(\frac{i_x-j_x}{W})^2 + (\frac{i_y-j_y}{H})^2}{-2\sigma^2}\right),
\]
with a single learnable scalar radius \(\sigma\) per ExpAtt layer [2006.07872]. The same work also studies fixed constant, linear, cosine, and exponential distance-based kernels, all content-independent and shared across heads.

The probabilistic IB framework parameterizes attention as
\[
p(a|x) = \mathcal{N}\big(a \mid g_e^\mu(x), g_e^\Sigma(x)\big),
\]
with a quantized version \(A_q\) obtained by assigning each attention value to the nearest learnable anchor \(v_k\) [2108.03418]. This makes both the distribution over attention maps and the codebook of attention values explicit.

Continuous multimodal attention uses a density
\[
p(x) = \sum_{k=1}^K \pi_k\, \mathcal{N}(x;\mu_k,\Sigma_k),
\]
where \(\mu_k\), \(\Sigma_k\), and \(\pi_k\) have direct geometric interpretations as centers, extents, and weights of attention modes [2104.03046]. The resulting context vector is the linear combination \(c = \sum_{k=1}^K \pi_k c_k\), where each \(c_k\) is a unimodal continuous-attention context [2104.03046].

Excitation Backprop defines attention probabilistically over a CNN graph. Given a top-down prior \(P(A_0)\), the Marginal Winning Probability satisfies
\[
P(a_j) = \sum_{a_i \in \mathcal{P}_j} P(a_j \mid a_i)\,P(a_i),
\]
and the conditional winning probability over excitatory connections is
\[
P(a_j \mid a_i) =
\begin{cases}
Z_i\,\widehat{a}_j\, w_{ji}, & \text{if } w_{ji} \ge 0,\\
0, & \text{otherwise}.
\end{cases}
\]
The resulting spatial maps are probability histograms over locations rather than heuristic saliency overlays [1608.00507].

Structured attention can also be explicit in dependency form. AttentionRNN models
\[
p(\mathbf{A}\mid \mathbf{X}) = \prod_{i,j} p\bigl(a_{i,j} \,\big|\, \mathbf{a}_{<i,j}, \mathbf{X}\bigr),
\]
and approximates each scalar attention variable by a Gaussian whose parameters are predicted from bi-directional diagonal LSTMs [1905.09400]. Here explicitness lies in the factorization of the joint distribution over the mask itself.

## 3. Construction from activations, features, and motion cues

One major strand constructs attention maps analytically from intermediate activations. In [1807.10576], if the last convolutional layer produces feature maps \(f_k(x)\), a class-agnostic semantic attention map is defined by averaging channels:
\[
M(x) = \frac{1}{k} \sum_k f_k(x).
\]
This map is derived directly from an ImageNet-trained Inception-v3 classifier, without saliency-specific supervision, then upsampled by cubic interpolation [1807.10576]. The paper explicitly interprets \(M(x)\) as a generic objectness or semantic saliency map suitable for free viewing.

A related but distinct construction appears in fine-grained recognition [1811.10770]. There, multiple local classifiers produce activation volumes \(A_k \in \mathbb{R}^{(L+1)\times H \times W}\). Aggregating over classifiers by max-pooling yields \(A\), and the single-channel attention map is then
\[
M(i,j) = \max_{c=1,\dots,L} A_{c,i,j},
\]
that is, the maximum over non-background classes at each location. After Otsu thresholding, the binary map \(M_b\) becomes a surrogate object mask used to supervise local classification and to extract object-level features [1811.10770]. Attention is therefore not post hoc: it is defined from classifier activations and recursively used to train those same classifiers.

In sports tracking [2409.14543], motion attention maps are explicitly built from frame differencing. For grayscale frames \(I'_\tau\), consecutive differences are
\[
D_\tau = I'_{\tau+1} - I'_{\tau},
\]
and the paper uses absolute differences \(\mathbf{D}_t^{+} = |\mathbf{D}_t|\). These are passed through a parametric motion prompt layer,
\[
\mathbf{A}_t = a_{\boldsymbol{\theta}(\mathbf{D}_t^{+}),
\]
with only two learnable parameters, slope and shift, to obtain motion attention maps \(\mathbf{A}_t \in \mathbb{R}^{H \times W \times (T'-1)}\) [2409.14543]. The construction is explicitly tied to a physically interpretable motion operator rather than latent feature similarity.

In actor-conditioned action detection, attention is formed by combining actor features and context embeddings. Given an actor representation \(\mathbf{r}_a\) and context embedding \(\mathbf{E}_{t,h,w}\), a relation tensor is defined as
\[
\mathbf{R}_{a,t,h,w}  = \mathbf{w}_\Omega \mathbf{r}_a + \mathbf{w}_\gamma \mathbf{E}_{t,h,w} + \mathbf{b}_\beta,
\]
then converted to attention via a sigmoid:
\[
\mathbf{ACAM}_{a,t,h,w} = \sigma \big( \mathbf{R}_{a,t,h,w} \big).
\]
The resulting attention maps multiplicatively gate the original feature map \(\mathbf{I}\) to produce actor-conditioned features [1812.11631].

These examples define a spectrum. Some explicit maps are analytic functions of features [1807.10576, 1811.10770], some are geometric kernels [2006.07872], some are probabilistic latent variables [2108.03418, 2104.03046], and some are derived from hand-specified physical operators such as frame differencing [2409.14543]. A plausible implication is that explicitness often coincides with a deliberately narrow inductive bias.

## 4. Supervision, regularization, and human alignment

Explicit attention maps are often supervised directly rather than inferred only through task loss. The ABN-based framework [1905.03540] is the clearest instance. A human-edited target map \(M'(\mathbf{x})\) is aligned to the model’s internal map \(M(\mathbf{x})\) by
\[
L(\mathbf{x}_i) = L_{\text{att}(\mathbf{x}_i) + L_{\text{per}(\mathbf{x}_i) + \gamma \,\| M'(\mathbf{x}_i) - M(\mathbf{x}_i) \|_2.
\]
The feature extractor is frozen, and only the attention and perception branches are fine-tuned so that future attention maps reflect the edited or derived human priors [1905.03540]. On ImageNet, CUB-200-2010, and IDRiD, this produces lower MSE to human or expert maps and improved task performance [1905.03540].

ClickMe supervision makes attention a large-scale human-aligned auxiliary task. GALA networks generate explicit attention volumes at several intermediate layers, and the training loss compares model attention maps against resized, normalized ClickMe maps derived from human gameplay [1805.08819]. The paper reports that GALA-ResNet-50 with ClickMe reaches 88.56 of human-human reliability in attention-map similarity, compared with 64.21 for GALA-ResNet-50 without ClickMe and 43.61 for ResNet-50 [1805.08819]. It also reports improved classification on both the ClickMe split and full ILSVRC12 [1805.08819].

ICASC uses no external human maps, but it does impose explicit map-level constraints. Attention separation penalizes overlap between the target-class attention \(\mathcal{A}^T\) and the most confusing class attention \(\mathcal{A}^{Conf}\):
\[
L_{AS} = 2\cdot\frac{\sum_{ij}\left(\min(\mathcal{A}^T_{ij}, \mathcal{A}^{Conf}_{ij}) \cdot Mask_{ij}\right)}{\sum_{ij}(\mathcal{A}^T_{ij} + \mathcal{A}^{Conf}_{ij})},
\]
and cross-layer consistency encourages inner-layer attention to lie within the target region defined by a last-layer attention mask [1811.07484]. This work effectively treats attention maps as trainable class-discriminative objects.

Attention can also be constrained by information theory rather than by external annotations. In AIB, the objective
\[
\mathcal{L}_{\text{AttVIB} \;\equiv\; I(Z; Y) - \beta I(Z; X, A)
\]
encourages the attention-modulated latent representation \(Z\) to preserve label information while compressing input-plus-attention information [2108.03418]. Additional quantization and commitment losses on \(A_q\) regulate the attention coding channel [2108.03418]. This suggests a different notion of explicitness: attention is not supervised to match a human heatmap, but it is still governed by its own probabilistic and information-theoretic structure.

## 5. Dynamics, scanpaths, and structured dependencies

A distinct line of work treats explicit attention maps not merely as static saliency outputs but as components in dynamic or structured systems. In [1807.10576], the CNN-derived saliency map \(M(x)\) is inserted into the EYMOL eye-movement model by augmenting the generalized potential:
\[
\bar U(t, x, \dot x) = U(t, x, \dot x) - \gamma M(x),
\]
which yields an additional force term \(-\gamma M_x\) in the equations of motion [1807.10576]. Attention thereby becomes a top-down term in a variational dynamical system that generates scanpaths, not only fixation maps. The resulting CF-EYMOL model improves over the bottom-up dynamical baseline on CAT2000 and substantially outperforms random and center baselines in scanpath similarity metrics [1807.10576].

AttentionRNN makes structure explicit by modeling dependencies among attention variables themselves. Rather than predicting each spatial weight independently, it uses diagonal bi-directional LSTMs so that each \(a_{i,j}\) depends on previous attention values under a rasterized factorization [1905.09400]. This yields more coherent masks for attribute prediction, VQA, and image generation [1905.09400]. The work emphasizes that ordinary local or global attention layers often lack explicit spatial dependency modeling.

Path-Aware Graph Attention generalizes explicit attention to heterogeneous HD-map graphs. Attention between vertices \(u\) and \(v\) is defined as
\[
\Psi(u,v) = \sum_{l \le \lambda} \; \sum_{p \in P_l(u,v)} \Phi_l\left(\{ x_E(e) \mid e \in p \} \right),
\]
where paths are ordered sequences of typed edges and \(\Phi_l\) is implemented with an LSTM over edge features [2202.13772]. The resulting attention map is not a function only of adjacency or node features; it is explicitly a function of paths. This is a particularly strong version of explicit modeling because it elevates domain structure—lane connectivity semantics—into the definition of attention itself.

In 3D mesh classification, CGAT uses a CLS node so that graph-level predictions must flow through an attention bottleneck, and attention rollout provides node-level explanations on the tooth surface [2509.07581]. The final interpretability map is the rolled-out CLS-to-node attention rather than an arbitrary gradient saliency. This suggests that explicit attention maps can be transferred to domains in which spatial locations are graph vertices rather than image pixels.

## 6. Empirical patterns, benefits, and limitations

The empirical record across these works is mixed in methodology but consistent in one respect: explicit attention maps often improve interpretability and frequently improve task performance.

A compact summary of representative results is useful.

| Work | Explicit map form | Reported effect |
|---|---|---|
| [1807.10576] | CNN feature-average saliency \(M(x)\) | CF raw: AUC = 0.80, NSS = 1.177; competitive with classical saliency and useful in scanpath modeling |
| [2409.14543] | Motion attention from frame differencing + prompt layer | Improves TrackNetV2/V3 on tennis and shuttlecock benchmarks; gains in recall/F1 and fewer false negatives |
| [1905.03540] | Human-edited single-channel attention maps in ABN | Improves classification and lowers MSE to human/expert maps on ImageNet, CUB, IDRiD |
| [1812.11631] | Actor-conditioned spatio-temporal gating tensors | Improves AVA action detection from 19.83 mAP to 23.29 mAP over RoI+Tail baseline |
| [2108.03418] | Variational and quantized spatial attention | Improves error rates and interpretability consistency scores over several baselines |
| [1608.00507] | Probabilistic top-down attention via MWP and c-MWP | Strong Pointing Game performance; c-MWP especially strong on difficult subsets |
| [1811.10770] | Attention from classifier activations | 87.9% on CUB-200-2011, 94.1% on Stanford Cars, 92.1% on FGVC-Aircraft |
| [1811.07484] | Class-discriminative gradient attention with explicit losses | Accuracy gains including CIFAR-100 (+3.33%), CUB-200-2011 (+4.8%), PASCAL VOC2012 (+5.73%) |
| [2104.03046] | Gaussian-mixture continuous attention density | Similar VQA accuracy to discrete attention but lower JS divergence to human attention |
| [2006.07872] | Geometric self-attention maps with learnable radius | Up to 2.2% over ResNet baselines; better than AA-ResNet152 by 0.9% with fewer params and GFLOPs |

The interpretation of these gains depends on the setting. In fixation prediction [1807.10576], explicitness means semantic saliency without saliency-specific training. In tracking [2409.14543], explicitness yields motion-aware robustness in low-visibility or occluded conditions. In human-supervised ABN [1905.03540] and ClickMe GALA [1805.08819], explicitness supports human-machine cooperation and map-level interpretability. In ICASC [1811.07484], explicitness reduces visual confusion among classes. In ExpAtt [2006.07872], explicitness mainly buys efficiency and a strong inductive bias.

Limitations are equally recurrent. Many explicit maps inherit the assumptions built into their parameterization. Geometric kernels assume nearby pixels are generally more relevant [2006.07872]. Frame differencing conflates object and camera motion [2409.14543]. Human-edited attention may be costly or inconsistent [1905.03540, 1805.08819]. Quantized or Gaussian attention may be too rigid for highly irregular object geometry [2108.03418, 2104.03046]. Contrastive attention maps can emphasize discriminative parts rather than full object extent, which is useful for pointing but suboptimal for box coverage [1608.00507]. A plausible implication is that explicit attention maps trade representational flexibility for transparency and control.

## 7. Relation to implicit attention, CAM-style explanations, and modern transformer practice

Several of these works explicitly contrast themselves with post hoc or implicit attention. Excitation Backprop distinguishes Marginal Winning Probabilities from gradients, deconvolution, and LRP by giving a normalized probabilistic interpretation [1608.00507]. ABN-based human knowledge injection contrasts its single-channel editable maps with multi-channel or channel-wise attention mechanisms that are harder to manipulate [1905.03540]. ClickMe-supervised GALA networks position attention as a supervised module rather than a post hoc localization device [1805.08819]. The fine-grained classifier-activation method avoids specialized ranking or reinforcement-learning objectives by deriving attention directly from local classifier outputs and training it under standard softmax losses [1811.10770].

The relation to transformer-style attention is more nuanced. ExpAtt [2006.07872] is explicitly presented as an alternative to standard self-attention, replacing learned query-key interactions with content-independent geometric kernels. SCRAM [1905.10308], while focused on approximation, also makes a structural claim: image attention maps are sparse and spatially coherent, so one can explicitly model their support via PatchMatch-discovered modes and local neighborhoods. MAPS [2412.11965] reframes LLM attention heads as explicit vocabulary-to-vocabulary maps \(M = E W_{VO} U\), inferring head functions from parameters without any inference-time activations. Although MAPS addresses transformer attention heads rather than spatial maps, it extends the notion of explicit attention maps into function space: a head becomes an explicit operator over token pairs [2412.11965].

This suggests that explicitness can arise at multiple levels: the spatial map itself, the dynamic law into which it is inserted, the probabilistic family from which it is drawn, the supervisory signal against which it is trained, or the parameter-space operator from which it is inferred. The term does not imply a single family of architectures; it denotes a design philosophy that opposes opaque or purely emergent attention.

## 8. Broader significance and open directions

The literature points to several durable themes. First, explicit attention maps frequently improve scientific legibility. In medical and forensic contexts, CGAT’s attention rollout over tooth meshes is presented as a way to support expert validation [2509.07581]. In human-in-the-loop classification, edited ABN maps allow direct intervention in what the model attends to [1905.03540]. In XAIxArts, extracted cross-attention volumes in video diffusion become analyzable and even artistically reusable objects [2509.05323]. This suggests that explicit maps can serve not only prediction but also audit, control, and creative reuse.

Second, explicit maps often function as inductive biases. The geometric prior in ExpAtt [2006.07872], the motion prior in TrackNetV4 [2409.14543], the top-down semantic prior in CNN activation saliency [1807.10576], and the path prior in HD-map attention [2202.13772] each encode a structured assumption about the domain. Where those assumptions are well matched to the task, the resulting models can be both simpler and better.

Third, explicitness does not require a single implementation recipe. Some methods use deterministic formulas [1807.10576, 1811.10770, 2006.07872], some use differentiable latent-variable models [2108.03418, 2104.03046], some use map-level losses [1905.03540, 1811.07484, 1805.08819], and some use explicit extraction or rollout of preexisting attention tensors [2509.05323, 2509.07581]. What unifies them is that the attention map is no longer merely an incidental tensor buried in the network.

Finally, several open questions recur implicitly across the surveyed works. One concerns scalability: human supervision of attention is effective but expensive [1905.03540, 1805.08819]. Another concerns fidelity: attention maps can be visually appealing without fully capturing causal computation, a concern that motivates probabilistic, rollout-based, or ablation-validated approaches [1608.00507, 2412.11965, 2509.07581]. A third concerns domain transfer: explicit priors such as local geometric decay [2006.07872] or frame differencing [2409.14543] are powerful when appropriate, but potentially brittle outside their intended regime. This suggests that future work will likely continue to combine explicit map parameterization with more adaptive or hybrid forms of learning, while preserving the interpretability benefits that motivated the line of research in the first place.

Source: https://www.emergentmind.com/topics/explicitly-modeled-attention-maps