Global Token Dependency (GTD)

• Global Token Dependency (GTD) is a framework that explicitly models long-range, hierarchical token interactions beyond direct, one-hop attention.
• It serves both as a diagnostic metric for multi-hop dependencies and as an inductive bias embedded into embeddings and attention mechanisms.
• GTD methods balance efficiency and performance by integrating global token mixing tailored to task-specific structural priors.

Global Token Dependency (GTD) denotes the explicit modeling, measurement, or exploitation of token interactions that extend beyond local neighborhoods or direct one-hop attention links. In the cited works, the term appears in two closely related senses: as a diagnostic metric that quantifies how much a Transformer head relies on multi-hop, indirect token interactions rather than direct attention alone, and as a broader modeling objective in which long-range, hierarchical, or semantically structured relations are encoded directly into embeddings, attention computations, or other token-mixing operators. Taken together, these works suggest that GTD is less a single architecture than a family of formalisms for preserving and propagating nonlocal token structure across language modeling, vision, MRI restoration, and micro-expression analysis (Lee et al., 9 Sep 2025, Blades et al., 30 Jan 2025, Hou et al., 2 Mar 2026, Wei et al., 17 Mar 2026, Huang et al., 2023).

1. Conceptual scope and core distinctions

A recurrent distinction in the literature is between direct token interaction and structured or indirect dependency propagation. In standard transformer self-attention, token relations emerge from dot products between queries and keys. In the GTD-oriented formulations, token relations are not left entirely to be discovered implicitly; they are injected into the representation itself, used to constrain or bias attention, or measured through multi-hop paths in the attention graph.

The large-language-model formulation in "Contextually Structured Token Dependency Encoding for LLMs" states this contrast explicitly: conventional LLMs rely on self-attention to infer relationships between tokens on the fly, whereas the proposed method tries to encode structured token relations directly into embeddings and attention computations, so long-range and hierarchical dependencies are not left entirely to be discovered implicitly (Blades et al., 30 Jan 2025). The MRI restoration study uses global token mixing as a broad umbrella for mechanisms that let each spatial token depend on many or all other tokens, including self-attention, attention-free state-space mixers such as Mamba-like modules, and other large-context token mixers that go beyond local convolutions (Hou et al., 2 Mar 2026). The Micro-AU CLIP work does not use the term GTD; its own terminology is Global Semantic Dependency (GSD), implemented by Global Dependency Attention (GDA) plus Global Dependency Loss (GDLoss), but the paper’s mechanism is explicitly described as very close in spirit to GTD because it models interactions among AU tokens/features instead of treating each AU independently (Wei et al., 17 Mar 2026).

A plausible implication is that GTD should be understood as a structural property of token processing rather than as a synonym for self-attention. In the cited papers, GTD can be a metric, an inductive bias, an architectural operator, or a task-specific dependency module. The common thread is that token influence is allowed to propagate through paths, graphs, or globally supported mixers that exceed strictly local or direct pairwise interactions.

2. GTD as a quantitative metric on attention graphs

The clearest formal definition of GTD appears in "Mitigating Attention Localization in Small Scale: Self-Attention Refinement via One-step Belief Propagation," where GTD is introduced as a diagnostic metric for quantifying how much a Transformer head relies on multi-hop, indirect token interactions rather than just direct one-hop attention (Lee et al., 9 Sep 2025). For layer $l$ and head $h$, with attention matrix $A^{(l,h)}$, the paper defines a multi-hop interaction matrix

$$G^{(l,h)}_{ij} = \sum_{t=2}^{K} \beta^{t-1}\left(A^{(l,h)}\right)^t_{ij},$$

where $t$ is the path length, $K$ is the maximum hop length considered, $\beta \in (0,1)$ is a discount factor, and $\left(A^{(l,h)}\right)^t$ captures length-$t$ token-to-token transitions. GTD is then

$$\mathrm{GTD}(A^{(l,h)})=\frac{\lVert G^{(l,h)}\rVert_F^2}{\lVert A^{(l,h)}\rVert_F^2+\lVert G^{(l,h)}\rVert_F^2}.$$

This formula makes GTD a ratio of multi-hop dependency strength to total dependency strength. The numerator is the energy in the indirect or global paths; the denominator is the energy in both direct and indirect paths. The paper therefore interprets higher GTD as a larger fraction of attention influence coming from multi-hop propagation rather than direct one-step attention. GTD is bounded between 0 and 1.

The same paper uses GTD to diagnose attention localization and entropy collapse. Low GTD corresponds to attention dominated by direct, concentrated links; moderate GTD corresponds to a more balanced use of global context; very high GTD can also be problematic because attention may become overly diffuse or noisy. The reported empirical pattern is that heads with GTD roughly around 0.6–0.8 often perform best, while heads below about 0.5 or above about 0.85 can be associated with worse or unstable performance. The paper further reports significant Pearson correlations between GTD and task performance across checkpoints, often with $h$0 and $h$1, including positive correlations with accuracy on SST-2, HellaSwag, and RACE, and negative correlation with perplexity on WikiText (Lee et al., 9 Sep 2025).

The graph-theoretic reading is central. The attention matrix is treated as a weighted transition matrix over tokens, and powers of $h$2 encode longer paths through intermediate tokens. GTD is therefore not a sparsity score, an entropy score, or a thresholded graph statistic. The authors emphasize that GTD preserves full weighted-matrix information and remains stable even when the attention graph becomes nearly fully connected. In their experiments, GTD is computed per head using $h$3 and $h$4, and then averaged in various analyses.

3. Explicit dependency encoding in LLMs

"Contextually Structured Token Dependency Encoding for LLMs" proposes a direct architectural realization of GTD-like modeling in which dependency structure is treated as a first-class signal rather than an emergent by-product of attention (Blades et al., 30 Jan 2025). The paper introduces a dependency-aware embedding initialization and a dependency-modulated attention mechanism. Instead of treating a token as an isolated vector, each token representation is refined by incorporating information from its dependent tokens:

$h$5

where $h$6 is the set of tokens dependent on token $h$7, $h$8 is a learned dependency weight between tokens $h$9 and $A^{(l,h)}$0, and $A^{(l,h)}$1 is a nonlinear projection of the dependent token embedding.

The dependency matrix is itself defined through a hierarchical attention-style scoring function,

$A^{(l,h)}$2

where $A^{(l,h)}$3 and $A^{(l,h)}$4 are query and key vectors, and $A^{(l,h)}$5 is a dependency alignment function that penalizes structurally inconsistent token pairs. In the model’s attention computation,

$A^{(l,h)}$6

so the dependency matrix $A^{(l,h)}$7 is incorporated into the attention computation as a structural bias. The notation is a bit unconventional, but the intent is that attention is not purely based on dot products; dependency structure directly influences the weight distribution. The same dependency signal also enters the residual/feed-forward path:

$A^{(l,h)}$8

The algorithm includes iterative updates, and despite the formatting error in the paper, the intended meaning is that $A^{(l,h)}$9 is refined by gradient-based optimization during training.

Architecturally, the model remains transformer-based but modifies embedding initialization, attention computation, the residual/FFN path, and dependency refinement. The implementation is described as fitting into a 6-layer, 8-head transformer, adapted from an open-source model, and the paper claims that the model does not require additional trainable parameters beyond standard transformer components, though it does require extra matrix computations.

The reported empirical results support the paper’s GTD claim. Perplexity drops across all datasets: Structured Text Generation: 24.7 → 19.2, Coherence Benchmark: 30.5 → 25.8, Dependency Parsing Corpus: 27.9 → 22.4, and Multilingual Corpus: 32.2 → 27.1. The paper also reports more coherent generated sequences, fewer abrupt phrase transitions, more balanced sentence lengths, and greater lexical diversity, with type-token ratio: 0.52 → 0.58, unique words per 10K tokens: 1370 → 1540, and lexical density: 0.41 → 0.46. The clearest long-range result is dependency consistency across sequence lengths: 20 tokens: 82.4 → 86.1, 50 tokens: 75.2 → 81.5, 100 tokens: 65.8 → 78.2, and 200 tokens: 52.3 → 74.6. The paper attributes the main overhead to extra matrix operations in embedding initialization and attention modulation, with training time: 54 min/epoch → 63 min/epoch and memory usage: 22.1 GB → 25.4 GB.

A plausible implication is that this formulation turns GTD from a diagnostic quantity into a persistent mechanism for dependency propagation across the transformer stack: the dependency matrix starts as an initialized dependency structure, is refined during training, modulates attention across layers, and is also used in residual/FFN updates.

4. Global token mixing beyond pairwise attention

The GTD literature also includes formulations in which global dependency is realized without explicit pairwise query-key attention. "Adaptive Frequency Filters As Efficient Global Token Mixers" argues that adaptive frequency filtering can serve as an efficient global token mixer and provides a mathematically exact alternative to self-attention, large kernels, or fully connected token mixers (Huang et al., 2023).

The paper begins from a token-mixing view in which contextual aggregation is written as

$$G^{(l,h)}_{ij} = \sum_{t=2}^{K} \beta^{t-1}\left(A^{(l,h)}\right)^t_{ij},$$0

with the GTD requirement that $$G^{(l,h)}_{ij} = \sum_{t=2}^{K} \beta^{t-1}\left(A^{(l,h)}\right)^t_{ij},$$1 should be global and the weights should be semantic-adaptive. Adaptive Frequency Filtering (AFF) recasts token mixing as a global convolution,

$$G^{(l,h)}_{ij} = \sum_{t=2}^{K} \beta^{t-1}\left(A^{(l,h)}\right)^t_{ij},$$2

with a kernel $$G^{(l,h)}_{ij} = \sum_{t=2}^{K} \beta^{t-1}\left(A^{(l,h)}\right)^t_{ij},$$3 whose spatial support matches the full resolution of $$G^{(l,h)}_{ij} = \sum_{t=2}^{K} \beta^{t-1}\left(A^{(l,h)}\right)^t_{ij},$$4. The operator is implemented in the frequency domain:

$$G^{(l,h)}_{ij} = \sum_{t=2}^{K} \beta^{t-1}\left(A^{(l,h)}\right)^t_{ij},$$5

$$G^{(l,h)}_{ij} = \sum_{t=2}^{K} \beta^{t-1}\left(A^{(l,h)}\right)^t_{ij},$$6

where $$G^{(l,h)}_{ij} = \sum_{t=2}^{K} \beta^{t-1}\left(A^{(l,h)}\right)^t_{ij},$$7 is an instance-adaptive mask generated by a group $$G^{(l,h)}_{ij} = \sum_{t=2}^{K} \beta^{t-1}\left(A^{(l,h)}\right)^t_{ij},$$8 convolution (linear) layer + ReLU + another group linear layer. By the convolution theorem,

$$G^{(l,h)}_{ij} = \sum_{t=2}^{K} \beta^{t-1}\left(A^{(l,h)}\right)^t_{ij},$$9

so AFF is exactly a spatial-domain token mixer with a dynamic convolution kernel

$t$0

whose spatial shape matches the full $t$1 map. Circular padding is adopted for the equivalence to hold cleanly.

In this formulation, GTD is achieved through global receptive scope and a frequency-domain global basis. Every output token can, in principle, depend on every input token, but the operator avoids explicit pairwise attention. The paper contrasts this with self-attention, which has complexity $t$2, whereas AFF uses FFT/iFFT plus elementwise filtering, with complexity about $t$3. It also contrasts AFF with large-kernel convolutions and fully connected token mixers, arguing that AFF preserves global scope, semantic adaptivity, channel-wise mixing, and low cost.

The ablations are used as direct evidence for global token mixing. On ImageNet-1K, the backbone without token mixer gives 77.9%, with 3×3 conv mixer gives 78.6%, with AFF w/o FFT gives 78.4%, and with full AFF gives 79.8%. Against other frequency-domain methods on the same backbone, AFNO: 78.8%, GFNet: 79.1%, FFC: 79.1%, DFF: 79.3%, FNO: 79.7% but with 141M parameters, and AFF: 79.8% with 5.5M parameters. When the learned mask is averaged across channels, performance drops from 79.8% to 79.3%, and among global transforms the reported results are Wavelet: 78.6%, DCT: 79.6%, and FFT: 79.8%. These results are presented as evidence that the frequency-domain mechanism is essential and that channel-specific token mixing matters.

5. Physics-conditioned GTD in MRI restoration

"Revisiting Global Token Mixing in Task-Dependent MRI Restoration: Insights from Minimal Gated CNN Baselines" places GTD in a domain where the utility of global token interaction depends on the inverse problem itself rather than on a generic preference for larger context (Hou et al., 2 Mar 2026). The paper’s central claim is that global token dependency is not universally beneficial in MRI restoration. Whether self-attention-style global mixing or state-space-model-style long-range token interaction is needed depends on the physics of the inverse problem and on how the degradation is structured in image/k-space space.

The paper studies three representative settings. In accelerated MRI reconstruction with explicit data consistency, the observed undersampled multi-coil k-space data are modeled as

$t$4

and the iterative update is

$t$5

The argument is that Fourier encoding and repeated data-consistency updates already propagate information globally, so added GTD inside the learned regularizer is expected to be less critical. In MRI super-resolution via k-space center cropping,

$t$6

with

$t$7

so low-frequency anatomy remains intact and the model mainly needs to inject missing high-frequency details. In denoising with spatially heteroscedastic noise,

$t$8

and the corruption is strongly spatially non-uniform, making long-range aggregation more natural because the model benefits from comparing distant regions to estimate which areas are reliable and which are corrupted.

The controlled testbed uses a shared backbone family. The minimal local baseline is NAFNet, with gating

$t$9

and the intermediate large-field variant is Large-Small Dynamic Gated (LSG) based on LSConv, with dynamic kernel generation

$K$0

and grouped dynamic aggregation

$K$1

These are benchmarked against transformers such as SwinIR, Restormer, Xformer, ReconFormer, state-space models such as MambaIR, MMR-Mamba, DH-Mamba, Restore-RWKV, and MRI-specific reconstruction models such as HUMUSNet, PDAC, E2E-VarNet.

The empirical conclusion is explicitly task-dependent. For accelerated reconstruction, the minimal unrolled gated-CNN baseline is already highly competitive compared to recent token-mixing approaches in public reconstruction benchmarks. On single-coil fastMRI knee, NAFRecon reaches 32.74 dB PSNR at 4× and 31.21 dB at 8×; for multi-coil reconstruction, on fastMRI knee it gets 37.13 dB / 0.8911 SSIM / 0.0085 NMSE, and on Stanford2D 37.10 dB / 0.9261 SSIM / 0.0198 NMSE. The large-field variant LSGRecon does not improve performance and can slightly degrade it. For super-resolution on IXI, NAFNet: 32.10 dB / 0.9415 SSIM / 28.94 RMSE, while LSGNet: 32.26 dB / 0.9422 SSIM / 28.86 RMSE, so large-field context is only marginally helpful. For SNAP carotid MRI denoising, the best overall method is Xformer, with 19.65 dB PSNR / 0.4599 SSIM / 0.0646 NMSE; the local NAFNet baseline gives 19.48 dB / 0.4452 / 0.0665, and LSGNet gives 19.56 dB / 0.4530 / 0.0704. The paper’s final takeaway is therefore that GTD is necessary or strongly helpful for spatially heterogeneous corruption, only marginally helpful for controlled low-pass super-resolution, and often unnecessary when the inverse problem already contains strong physics-based global coupling.

6. Semantically structured dependency over AU tokens

"Micro-AU CLIP: Fine-Grained Contrastive Learning from Local Independence to Global Dependency for Micro-Expression Action Unit Detection" provides a GTD-style formulation in which the tokens are not arbitrary patches but semantically defined AU-level features (Wei et al., 17 Mar 2026). The paper decomposes the problem into local semantic independence (LSI) and global semantic dependency (GSD). The visual backbone extracts patch tokens from optical flow,

$K$2

and facial landmarks map AU-specific landmarks to corresponding patch tokens,

$K$3

For AU $K$4, Patch Token Attention (PTA) computes

$K$5

followed by

$K$6

This stage enforces local semantic independence by letting each AU be handled in its own pipeline.

Global dependency is then modeled across the AU features

$K$7

Global Dependency Attention (GDA) computes

$K$8

transforms the attention matrix into dependency weights

$K$9

constructs a global context feature

$\beta \in (0,1)$0

and injects this context into each AU prediction:

$\beta \in (0,1)$1

Global Dependency Loss (GDLoss) is defined as

$\beta \in (0,1)$2

where $\beta \in (0,1)$3 is the AU label vector. The paper emphasizes that dependency is modeled at the feature interaction level, not merely at the label level. It also adds a micro-AU contrastive loss, MiAUCL, for fine-grained visual-text alignment:

$\beta \in (0,1)$4

with a label matrix that treats samples with the same AU state as semantically similar, and total objective

$\beta \in (0,1)$5

The reported ablations are important because they separate local evidence from global dependency. LSI only beats GSD alone, but LSI + GSD is best. On CASME II, LSI only: F1 0.761, GSD only: F1 0.730, and both: F1 0.782. On SAMM, LSI only: F1 0.714, GSD only: F1 0.638, and both: F1 0.730. The paper also reports that GDA performs better than Cat+MLP and Add+MLP, and that MiAUCL performs better than No CL, global OrigCL, and local OrigCL. The paper’s own caveat is that this is not a generic transformer over all image patches or video tokens; dependency is modeled between semantically named AU tokens, making the mechanism more structured, more task-specific, and more semantically interpretable.

7. Empirical regularities, trade-offs, and recurring misconceptions

Several recurring patterns emerge across these works. First, GTD is not identical to self-attention. AFF implements global token mixing through FFT/iFFT and semantic-adaptive frequency filtering rather than through explicit pairwise query-key relations (Huang et al., 2023). The MRI paper treats global token mixing as an umbrella category that includes self-attention, state-space mixers, and other large-context token mixers (Hou et al., 2 Mar 2026). The Micro-AU CLIP work models dependency between AU features rather than across all image patches (Wei et al., 17 Mar 2026). The LLM paper injects dependency structure directly into embeddings, attention, and the feed-forward/residual path (Blades et al., 30 Jan 2025).

Second, more GTD is not always better. The SAOBP paper explicitly reports that moderate GTD is often associated with the best performance, while very low GTD coincides with localization and entropy collapse and very high GTD can be overly diffuse or unstable (Lee et al., 9 Sep 2025). It further notes that BP-Low, despite sometimes producing high GTD numerically, often hurts performance because it over-regularizes and skews entropy too much. The MRI study reaches an analogous conclusion at the task level: global token dependency is not universally beneficial, because physics-driven data consistency can already provide strong global coupling in accelerated reconstruction, whereas denoising with pronounced spatially heteroscedastic noise benefits most from global aggregation (Hou et al., 2 Mar 2026). The Micro-AU CLIP ablations add a related nuance: global dependency alone is weaker than good local AU features, but the combination of local independence and global dependency is best (Wei et al., 17 Mar 2026).

Third, GTD-oriented methods usually involve an accuracy–efficiency trade-off rather than a free gain. The LLM dependency-encoding method reports a 16.7% increase in training time and a 14.9% increase in memory usage due to additional matrix computations, although scalability is said to remain feasible within conventional transformer architectures (Blades et al., 30 Jan 2025). AFF is presented as an alternative point in that trade-off space: it seeks global scope and semantic adaptivity with complexity about $\beta \in (0,1)$6 rather than the $\beta \in (0,1)$7 pairwise cost of self-attention (Huang et al., 2023).

Finally, the cited works suggest that GTD is best understood as a task-conditioned structural prior. In language modeling, it is used to preserve syntactic and semantic dependencies across multiple processing layers. In MRI restoration, it should be tailored to the underlying imaging physics and degradation structure. In micro-expression analysis, it operates over semantically named AU tokens and is supervised by co-activation structure. In self-attention diagnostics, it serves as a normalized measure of how much attention flows through multi-hop token dependencies rather than direct links. Taken together, these formulations suggest that GTD is most useful when the problem requires reliable propagation of long-range, hierarchical, or semantically structured dependencies, and least useful when such coupling is already imposed elsewhere in the model or the measurement process.