---
title: Indirect Attention in AI Systems
url: https://www.emergentmind.com/topics/indirect-attention
type: topic
---

# Indirect Attention in AI Systems

Indirect attention denotes a family of attention phenomena in which relevance is not established by a direct, same-source query–key–value relation alone, but is mediated by external content, graph topology, structural priors, or intermediate computational circuitry. The phrase is not used uniformly across the literature: in attention mechanism design it refers to settings where keys and values arise from different sequences or modalities; in graph learning it denotes influence propagated through indirect neighbors and all valid paths; in mechanistic interpretability it describes circuits that resolve a target through intermediate suppressive or contrastive heads; and in LLM security it names the vulnerability created when models attend to tool-returned or environmental content containing hidden instructions [2509.26015] [2511.16871] [2211.00593] [2502.16580]. This suggests a family resemblance rather than a single standardized definition.

## 1. Scope and principal meanings

Across current usage, “indirect attention” is best understood as a structural property: the attended signal is mediated rather than directly aligned. The common theme is that what ultimately influences a model is separated from the mechanism that selects it. In some cases the separation is architectural, as when keys and values are misaligned; in others it is topological, as when influence is accumulated over graph walks; and in others it is adversarial, as when malicious instructions are embedded in retrieved documents or GUI elements rather than the user prompt [2509.26015] [2511.16871] [2505.14289].

| Domain | Meaning of indirect attention | Representative paper |
|---|---|---|
| Attention architectures | Keys and values come from different sequences or modalities | [2509.26015] |
| Graph learning | Nodes attend through direct and indirect neighbors via propagation | [2511.16871] |
| Mechanistic interpretability | The answer is produced through intermediate attention circuits | [2211.00593] |
| LLM security | Models attend to malicious instructions hidden in external content | [2502.16580] |
| GUI agents | Visual attention is steered by environmental injection | [2505.14289] |

A recurring misconception is to treat indirect attention as synonymous with ordinary long-range attention. The literature is more specific. In graph models, the relevant distinction is between explicit pairwise dense attention and propagation-induced influence over all walks; in transformer analysis, the distinction is between direct copying and multi-stage circuits; in security, the distinction is between user-provided instructions and externally injected content that the model nevertheless treats as actionable [2511.16871] [2510.25013] [2502.16580].

## 2. Misaligned-context attention as an architectural problem

The most explicit architectural formalization appears in “Indirect Attention: Turning Context Misalignment into a Feature” [2509.26015]. That work studies the regime in which keys and values are derived from different sequences or modalities. It begins by analyzing standard attention under additive Gaussian noise in the values and shows that attention propagates value noise rather than denoising it. With noisy values, the expected squared output error is
$$
\mathbb{E}\left[\|\hat{o} - o^*\|^2\right] = \sigma^2 d \sum_{i=1}^n a_i^2,
$$
and the output signal-to-noise ratio simplifies to
$$
SNR = \frac{1}{\sigma^2}.
$$
This yields a critical threshold $\sigma^* = 1$, beyond which noise energy dominates signal energy [2509.26015].

The paper then interprets key–value misalignment itself as structured noise. If keys are produced from $x_i \sim P_x(\mu_x,\Sigma_x)$ and values from $y_i \sim P_y(\mu_y,\Sigma_y)$, the deviation between the misaligned output and its aligned reference is treated as effective noise. Under orthogonal $W_v$ and normalized inputs, the expected misalignment noise energy becomes
$$
\gamma = 2d + \|\mu_y - \mu_x\|^2.
$$
Because the baseline term grows linearly with dimension, the induced noise can substantially exceed the critical threshold, especially at initialization and in early training [2509.26015].

Indirect Attention (IA) is introduced to address that regime directly. Queries are constructed as
$$
q_i = m_i + y_{\pi(i)},
$$
keys and values are
$$
K = XW_k,\qquad V = YW_v,
$$
and attention scores are modulated by a learnable structural bias:
$$
\tilde{S}_{ij} = \frac{q_i \cdot k_j + f(P_{ij})}{\sqrt{d_k}}.
$$
The positional or structural matrix is updated across layers via
$$
p^{(l+1)} = g(o^{(l)}),
$$
so the notion of alignment becomes content-informed and layer-dependent rather than fixed [2509.26015]. The paper also gives a Bayesian interpretation in which $f(P_{ij})$ approximates a positional prior $\log p(j\mid i)$.

Empirically, IA is evaluated on two synthetic tasks—arbitrary sorting and sequence retrieval—and on one-shot object detection. In the real-world setting, IA-DETR outperforms both a misaligned-attention DETR and a double cross-attention DETR on Pascal VOC and MS COCO; on Pascal VOC the reported values include about 82.94 on seen classes and about 65.13 on unseen classes for IA-DETR, versus 77.90 / 62.31 for double cross-attention DETR and 29.21 / 33.8 for misaligned attention DETR [2509.26015]. The paper’s stated limitation is that the theory primarily characterizes initialization-time behavior under simplifying assumptions rather than the full dynamics of trained transformers.

## 3. Topological and circuit-level indirectness

In graph learning, indirect attention is formalized without an explicit all-pairs attention matrix. “Topologic Attention Networks” defines attention through inference in a Gaussian graphical model with precision matrix $J$ and information vector $h$:
$$
p(x) \propto \exp\!\left(-\tfrac{1}{2}x^\top J x + h^\top x\right), \qquad \mu = J^{-1}h.
$$
The component form,
$$
\mu_i = \sum_j (J^{-1})_{ij} h_j,
$$
makes the central point: the influence of node $j$ on node $i$ is mediated by the inverse precision matrix rather than a direct edge score, so indirect neighbors contribute through all valid paths [2511.16871].

The model computes this influence by Gaussian Belief Propagation (GaBP) rather than explicit inversion. The fixed point encodes a global solution even though the algorithm uses only local edge-wise messages. To ensure convergence on cyclic graphs, the paper restricts to walk-summable precision matrices satisfying
$$
\rho\bigl(|I - \tilde{J}|\bigr) < 1,
$$
under which the global influence becomes a convergent geometric series over walks [2511.16871]. The architecture uses a multi-head GaBP block and a node-wise feed-forward network, with learned or fixed precision parameterizations including Pairwise Normal, Diagonally Dominant, and Laplacian constructions. A stated systems advantage is that implicit differentiation reduces memory from $O(TE)$ to $O(E+N)$, where $T$ is the number of GaBP iterations and $E$ the number of edges [2511.16871].

A second line of work locates indirect attention inside transformer circuits for Indirect Object Identification (IOI). In GPT-2 small, IOI is explained by a sparse circuit of 26 attention heads grouped into 7 main classes, including Name Mover Heads, S-Inhibition Heads, Duplicate Token Heads, Induction Heads, Previous Token Heads, Negative Name Mover Heads, and Backup Name Mover Heads [2211.00593]. Over 100,000 examples, the model achieves mean logit difference 3.56, predicts IO over S 99.3% of the time, and has mean IO probability 49% [2211.00593]. The mechanistic claim is that the model does not simply attend to the correct name. Earlier heads detect duplication and construct a “do not attend here” signal for the repeated subject; later Name Mover Heads then copy the indirect object into the output position.

Minimal task-specific models make that structure even more explicit. In the symbolic IOI setting, a single-layer, two-head attention-only transformer achieves perfect IOI accuracy despite having no MLPs and no normalization layers [2510.25013]. Residual-stream decomposition shows that one head behaves as an additive or aggregative subcircuit aligned with the sum direction of the candidate names, while the other behaves as a contrastive or suppressive subcircuit aligned with their difference direction [2510.25013]. A two-layer, one-head model reaches similar performance through layerwise composition, with ablations showing that query composition fails almost completely when removed and value composition also causes a very large failure [2510.25013]. In this literature, indirect attention therefore names a mediated reasoning procedure: selection emerges through intermediate representations rather than one-step direct copying.

## 4. Indirect attention as an LLM security surface

In LLM security, indirect attention arises because models attend not only to the user’s instruction but also to retrieved documents, web pages, tool outputs, screenshots, or other external context. “Can Indirect Prompt Injection Attacks Be Detected and Removed?” distinguishes direct prompt injection, where the attacker is the user, from indirect prompt injection, where malicious instructions are embedded in external content later retrieved by a tool [2502.16580]. The benchmark samples are represented as
$$
[(p, d, a, x, y)],
$$
with $p$ the original user instruction, $d$ the clean document, $a$ the answer, $x$ the injected instruction, and $y$ the probe used to verify whether $x$ was executed [2502.16580]. The benchmark is built from SQuAD and TriviaQA validation data and yields Inj-SQuAD with 900 samples and Inj-TriviaQA with 900 samples [2502.16580].

The paper evaluates instructed LLMs, open-source detectors, and custom detectors trained on crafted indirect-injection data. It also studies two post-processing removal methods: segmentation removal, which detects and drops suspicious sentence-level segments, and extraction removal, which trains an extraction model to recover the injected instruction and delete the longest common substring shared by the extraction and the document [2502.16580]. The principal findings are that existing LLMs and off-the-shelf detectors struggle on indirect prompt injections, custom-trained detectors perform much better but introduce over-defense, injection position strongly affects generalization, segmentation removal is better overall, extraction removal is especially strong at the tail position, and a detection-plus-removal pipeline outperforms prompt-engineering baselines such as Sandwich and Instructional while being competitive with or better than StruQ in many cases [2502.16580].

Subsequent attack work emphasizes that success depends not only on wording but on the allocation of attention to malicious content. “TopicAttack” replaces abrupt appended injections with a fabricated multi-turn transition prompt
$$
T_{\text{inj}} = T_b \oplus T_t \oplus I_{\text{inj}},
$$
where the transition is formatted as a fake dialogue beginning with “OK.” and fixed at $m=5$ turns [2507.13686]. The paper reports attack success rates over 90\% in most cases, even when various defense methods are applied, and argues that a higher injected-to-original attention ratio corresponds to a higher success probability [2507.13686]. Its analysis also reports that TopicAttack lowers the perplexity of the injected instruction, supporting the claim that smoother topical transition makes the injection more plausible [2507.13686].

Defense work has increasingly treated attention itself as the detection signal. “Attention is All You Need to Defend Against Indirect Prompt Injection Attacks in LLMs” introduces Rennervate, which extracts attention from response tokens to input tokens, compresses it with a 2-step attentive pooling mechanism over response tokens and attention heads, and performs token-level detection and sanitization [2512.08417]. The paper also introduces the FIPI dataset with 100,000 injected instances and 10,000 benign instances [2512.08417]. On FIPI, reported detection accuracies are 99.05% for ChatGLM, 97.88% for Dolly, 99.58% for Falcon, 99.43% for LLaMA2, and 99.37% for LLaMA3, while sanitization sharply reduces attack success rates and preserves high text fidelity with Jaccard similarity mostly 0.9–1.0 [2512.08417]. The stated limitations are that the method relies on access to internal attention weights and that sanitization may not fully remove injected-task-specific content [2512.08417].

## 5. Multimodal, dense-scene, and behavioral extensions

Indirect attention also appears in multimodal agents, where the relevant “prompt” is partly visual. “EVA: Red-Teaming GUI Agents via Evolving Indirect Prompt Injection” studies environmental injection attacks in which misleading instructions are embedded in pop-ups, chat messages, payments, and email-composition interfaces rendered into screenshots [2505.14289]. EVA treats the attack as a closed-loop optimization: it seeds a keyword lexicon, samples keywords according to learned utility, renders an injection, runs the agent, extracts effective trigger words from successful attacks, updates utility, and periodically prunes and replaces low-utility terms [2505.14289]. The work is explicitly black-box and uses behavioral feedback as a proxy for attention dynamics. On six GUI agents, it consistently outperforms static one-shot baselines; a highlighted example is a pop-up scenario on GLM-4v-Plus in which the baseline achieves 48% ASR and EVA reaches 80% ASR [2505.14289]. The paper further reports that persuasive and urgency styles dominate successful transfers across models [2505.14289].

A different usage appears in crowd counting. “Indirect-Instant Attention Optimization for Crowd Counting in Dense Scenes” defines “indirect” not as mediated inference over external content, but as supervision applied to a transformed one-dimensional map derived from a high-dimensional attention tensor rather than to the tensor directly [2206.05648]. The SoftMax-Attention step computes
$$
F_{wei_{i,j}} = \sum_{k=1}^{C}(SoftMax(F_{att}) \otimes F_{mul})_{k,i,j},
$$
producing a one-channel weighted density-like map for immediate loss computation [2206.05648]. In the reported ablation, dimensional averaging gives MAE 109.14 and MSE 320.54, dimensional convolution gives 65.39 / 241.60, SoftMax-Attention gives 57.85 / 219.62, and SoftMax-Attention plus RCLoss gives 54.08 / 212.73 [2206.05648]. Here indirectness refers to the supervision pathway rather than the attentional selection pathway.

The phrase also appears in cognitive neuroscience in a looser but conceptually related sense. “Attention acts to suppress goal-based conflict under high competition” studies conditions where relevant and irrelevant stimuli share a receptive field and occur with minimal temporal separation [1610.09431]. In 20 healthy young adults performing a multitasking EEG paradigm, P100 amplitude at 0 ms event onset asynchrony was suppressed for both relevant and irrelevant stimuli in the rivalry conditions, whereas the passive control did not show a comparable reduction [1610.09431]. The authors interpret this as a non-selective top-down suppression mode that reduces feedforward interference within about 100 ms of stimulus onset [1610.09431]. Although this is not a transformer or deep-learning definition, it captures the same mediated-control theme: attention operates through an intermediate suppressive mechanism rather than simple selective enhancement.

## 6. Evaluation criteria, limitations, and unresolved questions

The literatures surveyed here use very different evaluation criteria, reflecting the fact that indirect attention is not yet a unified field. Security papers emphasize true positive rate, false positive rate, removal rate, and attack success rate, and in some cases utility-preservation measures such as Win Rate and Jaccard similarity [2502.16580] [2512.08417]. Architectural and graph papers emphasize synthetic-task accuracy, benchmark performance, convergence behavior, and systems quantities such as memory scaling [2509.26015] [2511.16871]. Mechanistic interpretability adds faithfulness, completeness, and minimality as explicit desiderata for an explanatory circuit [2211.00593].

Several limitations recur. In indirect prompt injection defense, generalization to unseen attack styles remains limited, over-defense on clean or out-of-domain documents is a central problem, and token-level sanitization can damage utility when adversarial spans are entangled with task-specific content [2502.16580] [2512.08417]. In GUI red teaming, attention is inferred only behaviorally rather than mechanistically, so causality remains unresolved [2505.14289]. In topologic attention, fixed Laplacian constructions can be slow because the spectrum lies near the convergence boundary [2511.16871]. In the misalignment literature, the central theory characterizes initialization-time fragility under simplifying assumptions rather than trained-model behavior in full generality [2509.26015]. In mechanistic interpretability, even a strong IOI explanation leaves gaps due to redundancy and incompleteness, while minimal symbolic models show that the underlying algorithm can nevertheless be compact and highly interpretable [2211.00593] [2510.25013].

Taken together, these works suggest that indirect attention is best viewed as a recurrent design and analysis pattern rather than a single mechanism. The pattern appears whenever the decisive signal is mediated by a structure that is not identical to the directly queried representation: a graph walk, a structural bias, an inhibitory subcircuit, a retrieved document, or a visually salient environmental cue. That shared structure explains both the promise and the risk of indirect attention. It enables long-range reasoning, multimodal alignment, and compact circuit formation, but it also creates failure modes in which relevance is inferred from a context that is noisy, misaligned, adversarial, or only partially observable [2509.26015] [2511.16871] [2502.16580].

Source: https://www.emergentmind.com/topics/indirect-attention