---
title: Declarative Attention (DA)
url: https://www.emergentmind.com/topics/declarative-attention-da
type: topic
---
# Declarative Attention (DA)
Declarative Attention (DA) denotes an attention mechanism in which an agent or model explicitly specifies the information scope required for a subsequent computation, rather than relying exclusively on an implicitly learned or externally inferred attention distribution. The term has multiple uses: in long-context language-model inference, DA is a protocol in which a model emits declarations selecting global, focused, or local context regions that an inference engine converts into KV-cache masks [2609.02737]; in dynamic epistemic logic, selective attention is represented through explicit propositions, event preconditions, and update rules [2303.13494]. Related mechanisms have also been described as “guided attention” in dialogue-act prediction, where a known utterance–label alignment is imposed as an attention prior, although that paper does not formally use “Declarative Attention” as the method name [2002.08801].
## 1. Terminology and conceptual scope
The central idea of DA is that attention selection is made explicit as a structural or declarative object. The declaration may specify which context region a language model should access, which propositions an epistemic agent attends to, or which source position is structurally associated with an output label. In each case, the attention mechanism exploits information about the task that is available independently of an unconstrained attention computation.
The term should not be conflated with every mechanism that performs feature selection or gating. “Dynamic Additive Attention Adaption” ($DA^3$) describes a memory-efficient multi-domain adaptation method whose supplied material establishes a dynamic spatial gate based on sigmoid and Gumbel-Sigmoid relaxation, hard thresholding at $0.5$, and differentiable training; it does not establish a formal Declarative Attention mechanism [2012.01362]. Similarly, “Diverse and Adaptive Attention” ($DA^2$-Net) is a convolutional attention module that generates multi-resolution feature maps and applies input-dependent local feature-map gating [2111.13157]. These systems are adaptive or parameterized attention mechanisms, but their supplied descriptions do not identify them as declarative.
The most direct commonality among the principal uses is the separation of **attention specification** from **attention computation**. A declaration, structural prior, or logical constraint determines what may be attended to; the underlying model then performs inference, decoding, or belief update over the permitted information.
## 2. Declarative Attention for long-context language models
The long-context formulation of DA is a protocol for reducing KV-cache reads during autoregressive decoding. At each decoding step, ordinary global attention reads the preceding context through the KV cache, even when useful information is concentrated in a small subset of positions. DA asks the language model to declare its intended attention scope in its generated reasoning trace. An inference-side state machine parses the declarations and modifies the visible KV-cache block table [2609.02737].
DA divides generation into three modes:
1. **Global mode**: the model attends to the entire addressable context.
2. **Focus mode**: the model attends to one or more named context regions.
3. **Local mode**: the model attends only to the persistent scaffold and generated response, without long-context segments.
The always-attended portion consists approximately of the system scaffold and all response tokens generated before the current step. Global mode adds every context chunk, focus mode adds the chunks named by the declaration, and local mode adds no long-context chunks. The model may alternate among these modes as needed rather than following a fixed schedule.
A typical protocol uses tags such as:
```text
I need the founding year and the IPO year.
Acme Corp was founded in 2003 in San Jose.
2011 - 2003 = 8 years.
```
The long context is divided into addressable “magic chunks,” approximately 2,048 tokens in the prompt-construction procedure, with a hard cap of approximately 2,560 tokens. Segmentation preferentially occurs at paragraph, newline, sentence, clause, and word boundaries. The chunks are lossless substrings with stable token offsets and are rendered as simulated tool responses, for example `Magic Chunk 2`.
The protocol is intended to make attention scope semantically interpretable. Global mode is used for navigation; focus mode extracts a short value from a selected region; local mode performs arithmetic, planning, synthesis, or answer construction over values already present in the response history. The model is instructed not to reconstruct unexamined chunk contents during local mode and instead to return to global mode if additional evidence is required.
## 3. Runtime implementation and computational behavior
The runtime parser watches the generated token stream. When it detects the completion of an opening focus tag such as ``, it parses the referenced chunk identifiers and updates the attention mask before decoding the next token. An opening `` tag switches to local mode. Closing `` and `` tags return the state to global mode. An explicit `` tag is primarily structural because global mode is already the default between declared spans.
The engine maps each named chunk to its token span and then to the KV-cache blocks containing that span. The visible block sets are therefore the persistent scaffold and response history plus either all context blocks, selected context blocks, or no context blocks. The mask is rounded outward to block boundaries, so a selected span may include up to approximately $b-1$ additional tokens at each edge for block size $b$.
DA does not evict KV entries. Previously hidden information remains available for later global or focus phases. This makes the mechanism reversible in the sense that a subsequent declaration can re-attend to a context region without re-prefilling the entire prompt. The prototype changes attention metadata and block tables while using existing FlashAttention or Triton paged-attention kernels; it does not modify kernels or the scheduler [2609.02737].
The attention operation itself remains ordinary causal attention over a mode-dependent index set. In global mode, the index set contains all preceding context positions. In focus mode, it contains the scaffold, response history, and selected chunks. In local mode, it contains only the scaffold and response history. DA applies to context-length-dependent global-attention layers, not to Gemma sliding-window-attention layers or Qwen Gated DeltaNet layers whose recurrent state has context-independent size.
The computational benefit is measured through total attended tokens:
$$
A=\sum_{t=1}^{D} a_t,
$$
where $a_t$ is the number of KV positions attended at decoding step $t$. Relative reduction is defined as
$$
1-\frac{A_{\mathrm{DA}}}{A_{\mathrm{vanilla}}}.
$$
This metric includes both attention sparsity and the number of generated tokens. DA often produces longer responses because the model explicitly generates navigation, extraction, and mode-transition text. Consequently, reduced per-step KV traffic is partially offset by additional decoding steps.
## 4. Empirical results and trade-offs
The reported evaluation covers 15 long-context sources from RULER, LongBench v1, LongBench v2, LooGLE, and ZeroScrolls. The principal comparison uses Gemma-4-31B and Qwen-3.6-27B. The evaluated systems are vanilla full attention, a maskless DA prompt condition, and DA with the runtime mask [2609.02737].
For Gemma-4-31B, vanilla decoding reaches $87.01\%$ accuracy and DA reaches $85.74\%$, a decline of $1.27$ percentage points. Attended tokens decrease from $13.43$ million to $6.45$ million, corresponding to a $52.0\%$ reduction. For Qwen-3.6-27B, accuracy decreases from $85.31\%$ to $82.56\%$, a decline of $2.75$ percentage points, while attended tokens decrease from $22.54$ million to $15.52$ million, a $31.1\%$ reduction.
The maskless DA prompt is nearly accuracy-neutral relative to vanilla for Gemma, but it increases attended tokens because the protocol elicits longer responses. Applying the runtime mask reduces attended tokens relative to the maskless condition by $71.1\%$ for Gemma and $46.5\%$ for Qwen. This isolates the principal computational contribution of DA: the savings arise from modifying the visibility of KV blocks rather than merely from chunked presentation or additional reasoning instructions.
Performance depends on model scale and protocol adherence. The largest models approach approximately $99\%$ focus-parse success, whereas Gemma-4-E4B reaches only $58\%$. Invalid or incorrect declarations can select the wrong region, fail to extract evidence, induce additional global phases, or cause format failures. A wrong focus declaration may be more damaging than full attention because the model can reason confidently over incomplete or irrelevant information.
DA performs most favorably when answers depend on a few localized spans and extracted values can be carried forward in the generated response. It is less reliable for multi-span reasoning, global counts, tables split across chunk boundaries, document-wide ordering, per-segment output, and tasks requiring output proportional to document size. These failure modes arise because independently focused regions may not preserve the cross-segment structure required by the task.
The principal efficiency trade-off is therefore between **declaration correctness** and **attention sparsity**. Global phases remain expensive because they read the entire context. In the Gemma-4-31B analysis, global mode accounts for roughly $27\%$ of generated tokens, while focus and local modes account for approximately $73\%$. Focus steps attend to roughly $12\%$ as many tokens as vanilla steps, and local steps attend to roughly $6\%$. Global-mode usage increases with context length, limiting savings on the longest inputs.
Roofline projections on a B200, using BF16, $40\%$ MFU, $70\%$ MBU, and $8$ TB/s peak HBM bandwidth, estimate total decode-time ratios of approximately $0.71$ for Gemma-4-31B and $0.77$ for Qwen-3.6-27B relative to vanilla. These are theoretical projections rather than measured end-to-end latency results. They assume a large-batch, memory-bound decoding regime and exclude prefill.
## 5. Selective attention in dynamic epistemic logic
A distinct declarative formulation treats attention as an explicit component of an agent’s epistemic state. Belardinelli and Bolander generalize earlier dynamic epistemic logic models in which agents were either fully attentive or entirely inattentive. Their model permits an agent to attend to arbitrary subsets of atomic propositions [2303.13494].
For each agent $a$ and proposition $p$, the attention atom
$$
\mathsf{h}_a p
$$
means that agent $a$ is paying attention to whether $p$. A Kripke model contains ordinary propositional atoms and attention atoms. The belief operator $B_a$ describes what holds throughout the worlds accessible to agent $a$. Attention need not be introspective: an agent may believe that she attends to $p$ while in fact failing to attend to it.
An event model specifies possible combinations of stimulus truth and attention states. Its edges encode how attention controls the information accessible after an event. The central principles are:
- **Attentiveness**: if $\mathsf{h}_a p$ holds in the source event, every event considered possible by $a$ contains both the attended fact and the fact that $a$ attended to $p$.
- **Inertia**: if $\mathsf{h}_a p$ does not hold, the update does not force the agent to learn the corresponding literal.
- **Defaulting**: in the default model, if $\mathsf{h}_a p$ does not hold, the agent’s accessible events contain the default value assigned to $p$.
Under the no-default model, unattended information remains subject to the agent’s prior epistemic state. Under the default model, inattention may generate a false belief. A default map assigns each proposition one of $p$, $\neg p$, or $\top$, where $\top$ preserves prior beliefs. If an agent attends to $p$ but not to $g$, and $d_a(g)=\neg g$, then an actual stimulus $p\wedge g$ can yield the posterior belief $B_a p\wedge B_a\neg g$ despite the fact that $g$ is true. This represents inattentional blindness as a systematic epistemic consequence of selective attention.
The framework therefore separates three components: what happens, what an agent attends to, and what the agent believes happened. Product update transforms the prior epistemic model into a posterior model by combining world states with event states whose preconditions and accessibility relations encode attention.
## 6. Structural priors and guided attention
Dialogue-act prediction provides another interpretation of declarative attention. In the sequence-to-sequence model of Milajevs and colleagues, a context window consists of utterances $(u_{i-T},\ldots,u_i)$ and corresponding dialogue-act labels $(y_{i-T},\ldots,y_i)$. The task has a known positional alignment: $y_k$ is the label of $u_k$. Unlike machine translation, the source and target sequences have equal length and a direct correspondence between positions [2002.08801].
The model reframes dialogue-act classification as sequence prediction:
$$
P(Y_i\mid C_i)
=
P(y_1,\ldots,y_{|C_i|}\mid u_1,\ldots,u_{|C_i|}).
$$
For a fixed context window, the decoder models the label sequence autoregressively, conditioning each label on the encoder representation and previously generated labels. This enables sequential and nonlocal dependencies among dialogue acts.
The paper calls its attention mechanism **guided attention**, specifically hard guided attention and soft guided attention. Hard guided attention imposes the known alignment:
$$
\alpha_{j,k}
=
\begin{cases}
0,& k\neq j,\\
1,& k=j.
\end{cases}
$$
Consequently, the attention context is the encoder representation of the current utterance. Because that representation is generated by a dialogue-level recurrent encoder, it still contains contextual information from neighboring utterances. Hard guided attention is therefore a deterministic diagonal attention matrix, not stochastic hard attention.
Soft guided attention retains the learned attention scores but adds a score bias of $1$ to the aligned position $j=k$ before normalization. The current utterance is favored, while neighboring utterances remain available if their learned scores are sufficiently high.
This mechanism is “declarative” only in the broad sense that it declares a known structural fact about the task and injects it as an inductive bias. The paper does not use “Declarative Attention” as the formal name, does not require a separate alignment annotation, and does not use dialogue-act labels as an additional attention mask. The alignment follows from the task representation and is available at inference.
The model uses a hierarchical encoder: a word-level bidirectional GRU produces utterance representations, and an utterance-level bidirectional GRU models dependencies across the context window. The reported experiments use $T=5$. An evaluated persona hierarchy models speaker turns, but the final model uses the ordinary hierarchical GRU because the persona layer substantially harms performance in some settings.
The strongest configuration combines the ordinary hierarchical encoder with hard guided attention. Sequence-level fine-tuning with beam-generated candidates produces final accuracies of $85.0\%$ on SwDA and $91.6\%$ on MRDA. On MRDA, the result is competitive but below a reported CRF-based result of $92.2\%$. The method’s limitations include fixed context length, dependence on known segmentation and one-label-per-utterance alignment, unstable persona modeling, class imbalance, and the absence of explicit legal-transition constraints.
## 7. Relation to adaptive and gated attention mechanisms
Declarative attention differs from mechanisms whose selection policy is learned implicitly from feature activations. In $DA^3$, the supplied material describes a continuous logistic gate, a differentiable Gumbel-Sigmoid relaxation, hard thresholding at $0.5$ during the forward pass, and gradient-based optimization through the relaxed gate [2012.01362]. This is a dynamic parameterized spatial gate. The available description does not establish an additive-attention formulation, a declarative constraint language, or a formal connection to Declarative Attention.
$DA^2$-Net separates diverse feature extraction from adaptive feature selection. It applies sequential grouped depthwise-separable convolutions with filter sizes such as $3\times3$, $5\times5$, and $7\times7$, performs global average pooling, computes local feature-map weights using a one-dimensional convolution, and reweights the feature maps through sigmoid gates [2111.13157]. Its attention is input-dependent and structurally lightweight, but the selection is produced by learned computation rather than an explicit external declaration or logical rule.
The distinction can be summarized as follows:
| Mechanism | Selection source | Primary object selected |
|---|---|---|
| Long-context DA | Model-generated scope declaration | KV-cache context chunks |
| Epistemic selective attention | Logical attention atoms and event rules | Propositional information |
| Guided attention | Known task alignment | Source utterance position |
| $DA^3$ gate | Learned sigmoid/Gumbel-Sigmoid scores | Spatial feature locations |
| $DA^2$-Net | Learned local feature-map weights | CNN feature maps |
The common principle is explicit control over information access, but the control may be linguistic, logical, structural, or learned. Only the first three mechanisms are directly described in the supplied material as declarative or structurally specified attention. The latter two are best classified as dynamic gating or adaptive feature selection unless a broader editorial definition of DA is adopted.
Across these formulations, the principal unresolved issues are capacity limits, graded or probabilistic attention, robust handling of segmentation boundaries, efficient execution of declarative policies, training models to produce valid declarations, and integrating attention with awareness, observability, or richer temporal environments.
Source: https://www.emergentmind.com/topics/declarative-attention-da