--- title: 'Random Attention: Definition and Models' url: https://www.emergentmind.com/topics/random-attention type: topic --- # Random Attention: Definition and Models Random attention refers to a family of models in which randomness enters the formation, approximation, routing, or interpretation of attention rather than being confined to parameter initialization. The term has several distinct uses: nonparametric random consideration in economic choice theory; randomized feature maps for linearizing neural attention; random sparsification of attention graphs; fixed random mixing or projections in Transformers; and random, structured information-flow protocols in multi-agent systems. Across these uses, randomness can represent latent consideration, approximate a kernel, reduce quadratic complexity, break permutation symmetry, or provide an inductive bias. It does not designate a single architecture or mathematical model. ## 1. Conceptual taxonomy Random attention can be organized according to the object randomized and the role randomness plays. **Random consideration** concerns whether a decision-maker attends to a subset of available alternatives. In the Random Attention Model (RAM), a fixed preference ordering is combined with a random consideration set; observed stochastic choice is attributed to attention randomness rather than random preferences [1712.03448]. In the Random Attention and Utility Model (RAUM), both preferences and consideration are heterogeneous, and preference and attention may be statistically dependent [2105.11268]. Reference-dependent extensions allow attention to depend on an observable status quo [2106.13350], whereas models with unobserved references treat some alternatives as always attended to but unknown to the observer [2407.01528]. **Random feature attention** uses random projections to approximate a similarity kernel. Random Feature Attention (RFA) replaces the softmax kernel with a randomized low-dimensional feature representation, permitting associative aggregation of keys and values [2103.02143]. Random Maclaurin Feature Attention (RMFA) performs an analogous construction for dot-product kernels through random polynomial features [2408.11656]. Wavelet-Enhanced Random Spectral Attention (WERSA) combines random spectral features with multiresolution Haar-wavelet filtering [2507.08637]. **Random sparse attention** randomizes which token pairs interact. Vision Big Bird uses Random Sampling Windows (RS-Win), which randomly partitions image tokens into fixed-size groups and applies ordinary attention within each group [2311.05988]. This retains sparse, approximately linear interaction complexity while allowing a randomly formed group to contain spatially distant tokens. **Random mixing and random routing** remove or constrain input-dependent attention. MixiT uses fixed, input-independent random token mixing while retaining trainable values and MLPs [2506.01115]. Diamond Attention uses per-agent random scalars to create a temporary rank ordering and a structured cross-agent mask, thereby breaking symmetry among otherwise identical cooperative agents [2605.06825]. In sparse-attention studies, a fixed random gate serves as a baseline for learned routing and exposes the effects of representation co-adaptation [2603.02227]. These categories are not interchangeable. Random consideration concerns latent cognitive or informational access; random feature attention approximates a kernel; random sparsification changes graph connectivity; and random mixing supplies a fixed communication operator. A method may combine several categories, but the source of randomness and the induced computational or behavioral restriction must be specified. ## 2. Random attention in revealed-preference theory In economic choice theory, random attention models formalize decisions in which alternatives are not necessarily all considered. Let \(S\) be a finite menu, \(T\subseteq S\) a nonempty consideration set, \(\mu(T\mid S)\) the probability of considering \(T\), and \(\succ\) a strict preference ordering. The decision-maker chooses the \(\succ\)-best element of \(T\), yielding \[ \pi(a\mid S)= \sum_{T\subseteq S} 1\{a\text{ is }\succ\text{-best in }T\}\mu(T\mid S). \] In RAM, the principal restriction is **Monotonic Attention**: \[ \mu(T\mid S)\leq \mu(T\mid S-a), \qquad a\in S-T. \] Removing an alternative that is not in \(T\) cannot reduce the probability assigned to the surviving consideration set. The restriction concerns latent consideration-set probabilities rather than observed choice probabilities. Consequently, RAM can generate violations of regularity: adding an alternative can increase the probability of choosing an existing alternative. RAM yields a revealed-preference implication unavailable under standard regularity. If \[ \pi(a\mid S)>\pi(a\mid S-b), \] then \(a\succ b\). The direct revealed relation generated by such inequalities, together with its transitive closure, contains all preference comparisons common to every RAM representation. A choice rule has a RAM representation if and only if this direct revealed relation has no cycle [1712.03448]. Preferences are therefore generally partially identified, while the latent attention rule is usually not identified. A triangular attention representation provides a canonical constructive representation for a fixed candidate preference ordering, but it need not describe the actual cognitive process. RAUM extends this framework to heterogeneous preference orderings. It imposes **stability** of the marginal preference distribution across menus and **set-monotonicity** of attention: conditional on preferences, the probability of considering a particular set cannot increase when the menu expands. Stability does not impose independence between preferences and attention. Under these joint restrictions, the model can be characterized as a finite-dimensional linear feasibility problem involving joint preference–filter probabilities and slack variables. The framework supports partial identification, out-of-sample prediction bounds, welfare bounds, and statistical testing, including under incomplete menu variation [2105.11268]. Several extensions alter the source of identifying variation. Reference-Dependent RAM requires the reference alternative to be included in every consideration set and allows the attention rule to depend arbitrarily on both menu and reference. Under full support, the observable conditions of No-Cycle, Status Quo Asymmetry, and Nontrivial Reference Effect characterize the model, and the common preference order is uniquely identified [2106.13350]. Random Attention Span instead uses variation in decision time on a fixed menu. Preference stability across time and monotonic expansion of accumulated attention allow stopping-time variation to substitute for menu variation [2405.11578]. In RAM with unobserved references, alternatives selected with certainty from singleton menus identify the reference set, while preferences are generally only coarsely identified; Independent Random Attention restores complete identification of preferences and attention parameters [2407.01528]. Attention Overload imposes a different restriction on the marginal attention frequency \[ \phi(a\mid S)= \sum_{T\subseteq S:\,a\in T}\mu(T\mid S), \] requiring \[ \phi(a\mid S)\leq \phi(a\mid T), \qquad a\in T\subseteq S. \] Thus, adding alternatives cannot increase the probability that an existing alternative receives attention. Under homogeneous preferences, the resulting \(\succ\)-Regularity inequalities sharply characterize compatible preference orderings and provide bounds on attention frequencies [2110.10650]. ## 3. Random feature attention and linearization In neural sequence models, conventional softmax attention for queries \(q_t\), keys \(k_i\), and values \(v_i\) uses the kernel \[ \kappa(q,k)=\exp(q^\top k/\sigma^2) \] and explicitly forms pairwise query–key interactions. For sequence length \(N\), this produces quadratic time and memory costs. Random feature attention approximates the kernel by \[ \kappa(q,k)\approx \phi(q)^\top\phi(k), \] where \(\phi\) is a randomized finite-dimensional feature map. Associativity then permits keys and values to be aggregated before the query is applied: \[ S=\sum_i\phi(k_i)\otimes v_i, \qquad z=\sum_i\phi(k_i), \] \[ h_t= \frac{\phi(q_t)^\top S} {\phi(q_t)^\top z}. \] The \(N\times N\) attention matrix is never constructed. For causal attention, the sufficient statistics are updated recurrently, \[ S_t=S_{t-1}+\phi(k_t)\otimes v_t, \qquad z_t=z_{t-1}+\phi(k_t), \] so decoding uses a fixed-size state independent of sequence length [2103.02143]. RFA uses random Fourier features for a Gaussian-kernel approximation of the exponential dot-product kernel. It can be used as a drop-in replacement for conventional attention and adds fewer than \(0.1\%\) additional parameters in the reported implementation. The random projections are sampled rather than ordinarily learned, while feature variance and Transformer projection parameters may be learned. The kernel estimator is unbiased in expectation, but the normalized attention output is generally not: the expectation of a ratio is not generally the ratio of expectations. Approximation variance decreases with feature dimension, approximately at the expected \(1/D\) rate for the kernel estimate, but finite feature dimension can produce noisy or unstable normalized attention. Random Maclaurin Feature Attention uses a Maclaurin expansion for a dot-product kernel, \[ \mathcal K(z)=\sum_{\eta=0}^{\infty}a_\eta z^\eta, \qquad a_\eta\geq 0, \] and random Rademacher projections to estimate powers of \(x^\top y\). The resulting feature maps support the associative computation \[ \operatorname{RMFA}_{\mathcal K}(Q,K,V) = \frac{ \widetilde Q(\widetilde K^\top V) }{ \widetilde Q(\widetilde K^\top \mathbf 1) }. \] The approximate cost is \(O(ndD)\), where \(D\) is the random-feature dimension, instead of the \(O(n^2d)\) cost of standard attention. The method supports exponential, inverse, logarithmic, hyperbolic, and square-root dot-product kernels, subject to nonnegative Maclaurin coefficients, bounded inputs, and nonzero denominators [2408.11656]. WERSA adds Haar-wavelet decomposition, content-adaptive scale filtering, learnable scale weights, and a random spectral feature map. Queries and keys are wavelet-transformed, filtered using coefficients derived from pooled query content, reconstructed, and then mapped through random projections. The key-value statistic is \[ S_K=\phi(K_F)^\top V, \] followed by \[ \widetilde A = \frac{\phi(Q_F)S_K} {\phi(Q_F)\phi(K_F)^\top\mathbf 1+\epsilon}. \] For fixed model dimension, wavelet depth, and feature dimension, the stated complexity is linear in sequence length. Its approximation error combines random-feature error with wavelet filtering and truncation error [2507.08637]. ## 4. Random sparsification, mixing, and sensitivity Random sparsification preserves conventional attention within a randomized subset of interactions. In Vision Big Bird, tokens are assigned independent uniform random scores, sorted into a random permutation, partitioned into fixed-size windows, processed by ordinary attention within each window, and restored to their original order. If \(L\) tokens are grouped into windows of size \(M\), the number of interactions is approximately \(LM\), yielding \(O(LM)\) computation and \(O(L)\) scaling for fixed \(M\). VBB combines RS-Win with CNN heads for local structure and implicit positional information, and average-pooled global keys and values for global context [2311.05988]. Random attention can also mean a fixed random mixing operator rather than a randomized approximation to softmax. MixiT removes input dependence from attention and uses \[ \operatorname{Attn}(h_\ell) = W_\ell^Vh_\ell \left(I+W_\ell^M-\overline W_\ell^M\right), \] where \(W_\ell^M\) is frozen random mixing and the column-wise centering term preserves the constant direction. The scaling of the random mixing supports a covariance stochastic differential equation in a joint depth–width limit and is designed to prevent rank collapse. MixiT performs strongly on arithmetic, Dyck-1, sentiment classification, and memorization, but performs poorly on content-addressable retrieval and induction tasks. Frozen-QK occupies an intermediate position: its query and key projections are random and frozen, but its attention remains input-dependent because the hidden representations evolve. A universal-approximation result is established for continuous causal functions under the Frozen-QK construction [2506.01115]. Random attention can provide high sensitivity to individual tokens. In a random-attention feature map based on softmax, \[ \varphi_{\mathrm{RAF}}(X) = \operatorname{softmax}\left(\frac{XWX^\top}{\sqrt d}\right)X, \] a perturbation to one token can cause many query positions to concentrate their attention on that token. Under the stated assumptions, word sensitivity is \(\Omega(1)\), whereas standard flattened random features have sensitivity \(O(1/\sqrt n)\). The result depends critically on the exponential concentration of softmax; a ReLU-based attention variant exhibits lower sensitivity that decreases with context length [2402.02969]. The sensitivity result does not imply universal robustness. The perturbations are continuous embedding-space changes rather than necessarily valid vocabulary substitutions, and high sensitivity can also imply vulnerability to adversarial perturbations. The theorem concerns a simplified single-head layer and does not establish comparable generalization guarantees for a full trained Transformer. ## 5. Random attention in structured prediction and coordination Random attention can alter information flow without approximating a kernel or representing cognitive consideration. In Diamond Attention, each cooperative agent samples a scalar \[ r_i\sim \operatorname{Uniform}[0,1], \] and the values induce a transient ordering. Agent \(i\) masks lower-ranked agents according to \[ M_{i,k}= \begin{cases} 0,&r_k\ge r_i,\\ -\infty,&r_k