Random Attention: Definition and Models

• Random attention encompasses models where randomness influences different aspects of the attention mechanism; includes random sampling, feature maps, and routing, it serves multiple purposes like reducing computational complexity and providing inductive biases
• Random attention is extensively used in both economics (e.g., Random Attention Model in choice theory) and neural networks (e.g., Vision Big Bird and MixiT within Transformers) to reduce quadratic computational complexities and permutations, inducing desired biases and limitations for tractability and interpretability.
• Randomness in attention can vary from parameter initialization to token routing; methods like Wavelet-Enhanced Random Spectral Attention (WERSA) and Vision Big Bird illustrate varied motivations and use cases.

Random attention refers to a family of models in which randomness enters the formation, approximation, routing, or interpretation of attention rather than being confined to parameter initialization. The term has several distinct uses: nonparametric random consideration in economic choice theory; randomized feature maps for linearizing neural attention; random sparsification of attention graphs; fixed random mixing or projections in Transformers; and random, structured information-flow protocols in multi-agent systems. Across these uses, randomness can represent latent consideration, approximate a kernel, reduce quadratic complexity, break permutation symmetry, or provide an inductive bias. It does not designate a single architecture or mathematical model.

1. Conceptual taxonomy

Random attention can be organized according to the object randomized and the role randomness plays.

Random consideration concerns whether a decision-maker attends to a subset of available alternatives. In the Random Attention Model (RAM), a fixed preference ordering is combined with a random consideration set; observed stochastic choice is attributed to attention randomness rather than random preferences (Cattaneo et al., 2017). In the Random Attention and Utility Model (RAUM), both preferences and consideration are heterogeneous, and preference and attention may be statistically dependent (Kashaev et al., 2021). Reference-dependent extensions allow attention to depend on an observable status quo (Kovach et al., 2021), whereas models with unobserved references treat some alternatives as always attended to but unknown to the observer (Bansal, 2024).

Random feature attention uses random projections to approximate a similarity kernel. Random Feature Attention (RFA) replaces the softmax kernel with a randomized low-dimensional feature representation, permitting associative aggregation of keys and values (Peng et al., 2021). Random Maclaurin Feature Attention (RMFA) performs an analogous construction for dot-product kernels through random polynomial features (Guo et al., 2024). Wavelet-Enhanced Random Spectral Attention (WERSA) combines random spectral features with multiresolution Haar-wavelet filtering (Dentamaro, 11 Jul 2025).

Random sparse attention randomizes which token pairs interact. Vision Big Bird uses Random Sampling Windows (RS-Win), which randomly partitions image tokens into fixed-size groups and applies ordinary attention within each group (Zhang et al., 2023). This retains sparse, approximately linear interaction complexity while allowing a randomly formed group to contain spatially distant tokens.

Random mixing and random routing remove or constrain input-dependent attention. MixiT uses fixed, input-independent random token mixing while retaining trainable values and MLPs (Dong et al., 1 Jun 2025). Diamond Attention uses per-agent random scalars to create a temporary rank ordering and a structured cross-agent mask, thereby breaking symmetry among otherwise identical cooperative agents (Patil et al., 7 May 2026). In sparse-attention studies, a fixed random gate serves as a baseline for learned routing and exposes the effects of representation co-adaptation (Aquino-Michaels, 11 Feb 2026).

These categories are not interchangeable. Random consideration concerns latent cognitive or informational access; random feature attention approximates a kernel; random sparsification changes graph connectivity; and random mixing supplies a fixed communication operator. A method may combine several categories, but the source of randomness and the induced computational or behavioral restriction must be specified.

2. Random attention in revealed-preference theory

In economic choice theory, random attention models formalize decisions in which alternatives are not necessarily all considered. Let $S$ be a finite menu, $T\subseteq S$ a nonempty consideration set, $\mu(T\mid S)$ the probability of considering $T$, and $\succ$ a strict preference ordering. The decision-maker chooses the $\succ$-best element of $T$, yielding

$$\pi(a\mid S)= \sum_{T\subseteq S} 1\{a\text{ is }\succ\text{-best in }T\}\mu(T\mid S).$$

In RAM, the principal restriction is Monotonic Attention:

$\mu(T\mid S)\leq \mu(T\mid S-a), \qquad a\in S-T.$

Removing an alternative that is not in $T$ cannot reduce the probability assigned to the surviving consideration set. The restriction concerns latent consideration-set probabilities rather than observed choice probabilities. Consequently, RAM can generate violations of regularity: adding an alternative can increase the probability of choosing an existing alternative.

RAM yields a revealed-preference implication unavailable under standard regularity. If

$T\subseteq S$0

then $T\subseteq S$1. The direct revealed relation generated by such inequalities, together with its transitive closure, contains all preference comparisons common to every RAM representation. A choice rule has a RAM representation if and only if this direct revealed relation has no cycle (Cattaneo et al., 2017). Preferences are therefore generally partially identified, while the latent attention rule is usually not identified. A triangular attention representation provides a canonical constructive representation for a fixed candidate preference ordering, but it need not describe the actual cognitive process.

RAUM extends this framework to heterogeneous preference orderings. It imposes stability of the marginal preference distribution across menus and set-monotonicity of attention: conditional on preferences, the probability of considering a particular set cannot increase when the menu expands. Stability does not impose independence between preferences and attention. Under these joint restrictions, the model can be characterized as a finite-dimensional linear feasibility problem involving joint preference–filter probabilities and slack variables. The framework supports partial identification, out-of-sample prediction bounds, welfare bounds, and statistical testing, including under incomplete menu variation (Kashaev et al., 2021).

Several extensions alter the source of identifying variation. Reference-Dependent RAM requires the reference alternative to be included in every consideration set and allows the attention rule to depend arbitrarily on both menu and reference. Under full support, the observable conditions of No-Cycle, Status Quo Asymmetry, and Nontrivial Reference Effect characterize the model, and the common preference order is uniquely identified (Kovach et al., 2021). Random Attention Span instead uses variation in decision time on a fixed menu. Preference stability across time and monotonic expansion of accumulated attention allow stopping-time variation to substitute for menu variation (Wei, 2024). In RAM with unobserved references, alternatives selected with certainty from singleton menus identify the reference set, while preferences are generally only coarsely identified; Independent Random Attention restores complete identification of preferences and attention parameters (Bansal, 2024).

Attention Overload imposes a different restriction on the marginal attention frequency

$T\subseteq S$2

requiring

$T\subseteq S$3

Thus, adding alternatives cannot increase the probability that an existing alternative receives attention. Under homogeneous preferences, the resulting $T\subseteq S$4-Regularity inequalities sharply characterize compatible preference orderings and provide bounds on attention frequencies (Cattaneo et al., 2021).

3. Random feature attention and linearization

In neural sequence models, conventional softmax attention for queries $T\subseteq S$5, keys $T\subseteq S$6, and values $T\subseteq S$7 uses the kernel

$T\subseteq S$8

and explicitly forms pairwise query–key interactions. For sequence length $T\subseteq S$9, this produces quadratic time and memory costs. Random feature attention approximates the kernel by

$\mu(T\mid S)$0

where $\mu(T\mid S)$1 is a randomized finite-dimensional feature map. Associativity then permits keys and values to be aggregated before the query is applied:

$\mu(T\mid S)$2

$\mu(T\mid S)$3

The $\mu(T\mid S)$4 attention matrix is never constructed. For causal attention, the sufficient statistics are updated recurrently,

$\mu(T\mid S)$5

so decoding uses a fixed-size state independent of sequence length (Peng et al., 2021).

RFA uses random Fourier features for a Gaussian-kernel approximation of the exponential dot-product kernel. It can be used as a drop-in replacement for conventional attention and adds fewer than $\mu(T\mid S)$6 additional parameters in the reported implementation. The random projections are sampled rather than ordinarily learned, while feature variance and Transformer projection parameters may be learned. The kernel estimator is unbiased in expectation, but the normalized attention output is generally not: the expectation of a ratio is not generally the ratio of expectations. Approximation variance decreases with feature dimension, approximately at the expected $\mu(T\mid S)$7 rate for the kernel estimate, but finite feature dimension can produce noisy or unstable normalized attention.

Random Maclaurin Feature Attention uses a Maclaurin expansion for a dot-product kernel,

$\mu(T\mid S)$8

and random Rademacher projections to estimate powers of $\mu(T\mid S)$9. The resulting feature maps support the associative computation

$T$0

The approximate cost is $T$1, where $T$2 is the random-feature dimension, instead of the $T$3 cost of standard attention. The method supports exponential, inverse, logarithmic, hyperbolic, and square-root dot-product kernels, subject to nonnegative Maclaurin coefficients, bounded inputs, and nonzero denominators (Guo et al., 2024).

WERSA adds Haar-wavelet decomposition, content-adaptive scale filtering, learnable scale weights, and a random spectral feature map. Queries and keys are wavelet-transformed, filtered using coefficients derived from pooled query content, reconstructed, and then mapped through random projections. The key-value statistic is

$T$4

followed by

$T$5

For fixed model dimension, wavelet depth, and feature dimension, the stated complexity is linear in sequence length. Its approximation error combines random-feature error with wavelet filtering and truncation error (Dentamaro, 11 Jul 2025).

4. Random sparsification, mixing, and sensitivity

Random sparsification preserves conventional attention within a randomized subset of interactions. In Vision Big Bird, tokens are assigned independent uniform random scores, sorted into a random permutation, partitioned into fixed-size windows, processed by ordinary attention within each window, and restored to their original order. If $T$6 tokens are grouped into windows of size $T$7, the number of interactions is approximately $T$8, yielding $T$9 computation and $\succ$0 scaling for fixed $\succ$1. VBB combines RS-Win with CNN heads for local structure and implicit positional information, and average-pooled global keys and values for global context (Zhang et al., 2023).

Random attention can also mean a fixed random mixing operator rather than a randomized approximation to softmax. MixiT removes input dependence from attention and uses

$\succ$2

where $\succ$3 is frozen random mixing and the column-wise centering term preserves the constant direction. The scaling of the random mixing supports a covariance stochastic differential equation in a joint depth–width limit and is designed to prevent rank collapse. MixiT performs strongly on arithmetic, Dyck-1, sentiment classification, and memorization, but performs poorly on content-addressable retrieval and induction tasks. Frozen-QK occupies an intermediate position: its query and key projections are random and frozen, but its attention remains input-dependent because the hidden representations evolve. A universal-approximation result is established for continuous causal functions under the Frozen-QK construction (Dong et al., 1 Jun 2025).

Random attention can provide high sensitivity to individual tokens. In a random-attention feature map based on softmax,

$\succ$4

a perturbation to one token can cause many query positions to concentrate their attention on that token. Under the stated assumptions, word sensitivity is $\succ$5, whereas standard flattened random features have sensitivity $\succ$6. The result depends critically on the exponential concentration of softmax; a ReLU-based attention variant exhibits lower sensitivity that decreases with context length (Bombari et al., 2024).

The sensitivity result does not imply universal robustness. The perturbations are continuous embedding-space changes rather than necessarily valid vocabulary substitutions, and high sensitivity can also imply vulnerability to adversarial perturbations. The theorem concerns a simplified single-head layer and does not establish comparable generalization guarantees for a full trained Transformer.

5. Random attention in structured prediction and coordination

Random attention can alter information flow without approximating a kernel or representing cognitive consideration. In Diamond Attention, each cooperative agent samples a scalar

$\succ$7

and the values induce a transient ordering. Agent $\succ$8 masks lower-ranked agents according to

$\succ$9

Task attention remains unmasked. Because identical agents receive different rank-dependent receptive fields, parameter sharing no longer forces identical latent states. The mechanism supplies protocol-space asymmetry while preserving set-based processing and variable team sizes (Patil et al., 7 May 2026).

The distinction between structured randomness and unstructured noise is central. Dropout may produce different activations but does not establish a common ordering or a mutually interpretable information-flow protocol. Diamond Attention uses shared random values to create a coherent hierarchy. In the reported XOR game, Diamond Attention reaches success $\succ$0, whereas deterministic and unstructured-randomness baselines remain at the random-action floor. It also supports zero-shot deployment across team sizes in continuous coordination tasks and zero-shot transfer in SMACLite, although the transfer advantage disappears when the structured mask is removed or replaced with dropout.

Random-tree attention provides another structured-data interpretation. Attention-Based Random Forests (ABRF) assign query-dependent weights to trees using the distance between an input and the mean feature vector of the reached leaf. For regression,

$\succ$1

ABRF-1 combines distance-based weights with a Huber contamination mixture,

$\succ$2

where $\succ$3 is the normalized distance-based component and $\succ$4 is a trainable tree distribution. The resulting squared-loss problem is a convex quadratic program over the simplex. ABRF-2 and ABRF-3 introduce trainable feature weights and softmax parameters, increasing flexibility but also optimization complexity and overfitting risk (Utkin et al., 2022).

A related multi-head soft random forest uses probabilistic soft tree splits, tree-specific reliability, and instance-dependent attention over trees. Its attention score combines reliability and query–key distance,

$\succ$5

The model was applied to patient no-show prediction and reports two levels of feature importance: tree-level importance from soft-node weights and attention-level importance obtained by weighting tree contributions according to instance-specific attention (Amalina et al., 22 May 2025).

6. Training, identification, and limitations

The central methodological issue across random-attention models is the relationship between randomness and trainability. Random feature methods freeze query–key projections and train value parameters, producing convex or comparatively simple learning problems. Their principal trade-off is approximation error: increasing feature dimension reduces variance but increases computation and memory. Random sparsification reduces complexity by discarding interactions, while random mixing changes the inductive bias more radically by making communication independent of current content.

End-to-end learned routing introduces a separate problem. In sparse attention, a small gate may be trained jointly with a much larger Q/K/V substrate. The underlying representations can adapt to the imposed mask, making learned routing only marginally better than a frozen random gate. This phenomenon, termed routing absorption, is distinct from the zero-gradient problem of hard top-$\succ$6 masks. Hard masks receive essentially no gradient through the discrete selection operation; soft gates restore differentiability but remain vulnerable to Q/K/V co-adaptation. In the reported experiments, learned and random soft gates achieved perplexities of $\succ$7 and $\succ$8, respectively, whereas a dense baseline achieved $\succ$9. Post-hoc gate training after freezing a dense model produced a substantially larger learned-versus-random difference, showing that deployment performance and mask-prediction quality are not equivalent (Aquino-Michaels, 11 Feb 2026).

Several limitations recur across the literature. Random feature approximations are not exact softmax attention, and unbiased kernel estimates do not imply unbiased normalized outputs. Random sparse patterns do not reproduce every dense interaction in a single layer. Static random mixing cannot directly implement content-addressable retrieval. Random coordination protocols require shared access to random values, synchronized execution, and tie handling in finite precision. Economic random-attention models require substantive assumptions such as monotonicity, stability, full support, or time monotonicity; without them, preferences and attention may be weakly identified or observationally inseparable.

The term “random attention” therefore should be interpreted by specifying four properties: what is random, whether randomness is fixed or resampled, whether attention remains input-dependent, and which object is being approximated or identified. A random feature map, a random token mask, a random consideration set, a fixed random mixing matrix, and a random rank protocol can all be called random attention, but they have different mathematical objects, computational costs, empirical implications, and failure modes.