Polynomial Mixer (PoM): Efficient Token Mixing

• Polynomial Mixer (PoM) is a token-mixing mechanism that aggregates sequence information into a global polynomial state, replacing quadratic self-attention with a linear alternative.
• It uses token-wise gating and polynomial aggregation to enable effective contextual mapping and support universal approximation in transformer architectures.
• PoM has demonstrated efficiency and competitive performance in self-supervised speech encoders, image generation, and other multidomain applications.

Searching arXiv for papers on “Polynomial Mixer (PoM)” and closely related usages. Polynomial Mixer (PoM) is a token-mixing mechanism that replaces multi-head self-attention by aggregating an input sequence into a compact polynomial state and then letting each token retrieve contextual information through a learned selector or gating function. In the formulations introduced for self-supervised speech encoders, diffusion transformers, and general sequence models, PoM is a drop-in replacement for self-attention with linear complexity in sequence length, and published analyses further prove the contextual mapping property and a universality theorem for PoM-based transformer architectures (Feillet et al., 28 Feb 2026, Picard et al., 7 Apr 2026, Picard et al., 2024).

1. Definition and design rationale

PoM was introduced in response to the quadratic memory and compute cost of self-attention. Standard self-attention forms or implicitly computes token-pair interactions over an $n \times n$ structure, so its cost scales as $O(n^2)$ in sequence length. The PoM literature instead replaces explicit pairwise interactions with a shared global state derived from the full sequence and a token-wise mechanism that selects from that state. In the speech setting, this state is described as a global polynomial state summarizing the sequence, which each token reads from via a learnable selector; in the general transformer setting, it is a compact representation obtained through a learned polynomial function from which each token retrieves contextual information (Feillet et al., 28 Feb 2026, Picard et al., 7 Apr 2026).

The design motivation is partly domain specific. In automatic speech recognition, the authors note that many long-range dependencies are not strictly necessary, and that upper-layer attention maps tend to be mostly diagonal, which suggests that exhaustive pairwise modeling may be unnecessary. In images and videos, the number of tokens grows with spatial or spatiotemporal resolution, so the quadratic behavior of attention becomes a severe bottleneck. PoM addresses both cases by keeping global mixing while avoiding attention maps altogether (Feillet et al., 28 Feb 2026, Picard et al., 2024).

A plausible synthesis is that PoM defines a family of global-state mixers rather than a single fixed operator. Across published variants, the common pattern is stable: polynomial feature construction, token aggregation into a shared state, token-wise gating or selection, and projection back to the model dimension.

2. Mathematical construction

In the speech formulation, the input is $X \in \mathbb{R}^{d \times n}$, and PoM is defined by

$$\text{PoM}(X) = W_o \left[ \sigma(W_s X) \circ H(X)\,\mathbf{1}^{\top} \right],$$

where $W_s \in \mathbb{R}^{kD \times d}$, $W_o \in \mathbb{R}^{d \times kD}$, $\sigma$ is a sigmoid gate, and $H(X)$ is a global polynomial state. That state is constructed as

$$H(X) = \left[ \;h(W_1 X) \;\middle|\; h(W_1 X) \circ h(W_2 X) \;\middle|\; \dots \;\middle|\; \prod_{m=1}^k h(W_m X) \right]\mathbf{1},$$

with learnable $W_1,\dots,W_k \in \mathbb{R}^{D \times d}$. The token-wise selector is

$O(n^2)$0

The output therefore has the same shape as the input, and the polynomial order is controlled by $O(n^2)$1, while $O(n^2)$2 controls expansion capacity (Feillet et al., 28 Feb 2026).

A second published formulation uses a single projection $O(n^2)$3 and per-dimension polynomial coefficients $O(n^2)$4. With $O(n^2)$5, the shared state is

$O(n^2)$6

the gate is $O(n^2)$7, and the mixer is

$O(n^2)$8

Here the polynomial is applied featurewise through powers of $O(n^2)$9, then summed over tokens (Picard et al., 7 Apr 2026).

Both constructions are explicitly linear in sequence length. The speech paper states time complexity $X \in \mathbb{R}^{d \times n}$0 and memory complexity linear in $X \in \mathbb{R}^{d \times n}$1. The broader transformer paper gives the PoM cost per layer as $X \in \mathbb{R}^{d \times n}$2, in contrast to $X \in \mathbb{R}^{d \times n}$3 for standard attention (Feillet et al., 28 Feb 2026, Picard et al., 7 Apr 2026).

3. Expressive power and sequence-modeling theory

The most explicit theoretical treatment appears in the general transformer paper and the earlier diffusion-oriented paper. A central result is that PoM satisfies the contextual mapping property: there exists $X \in \mathbb{R}^{d \times n}$4 for which a Polynomial Mixer of degree $X \in \mathbb{R}^{d \times n}$5 can be made contextual, in the sense used by Yun et al. for universality arguments. This result is then used to prove a universal approximation theorem: for any continuous sequence-to-sequence map on a compact domain, there exists a PoM-based architecture with learned positional encoding that approximates it arbitrarily well in $X \in \mathbb{R}^{d \times n}$6 distance (Picard et al., 7 Apr 2026, Picard et al., 2024).

PoM is also permutation equivariant in the unmasked setting. For any column permutation matrix $X \in \mathbb{R}^{d \times n}$7,

$X \in \mathbb{R}^{d \times n}$8

The proof relies on the fact that the aggregated state is permutation invariant because it sums over tokens, while the gating branch is tokenwise and therefore permutes consistently. As with self-attention, positional encodings are required when the task depends on sequence order rather than set structure (Picard et al., 7 Apr 2026).

The theoretical picture is therefore not merely one of efficiency. Published work argues that replacing attention by a single global state does not force a collapse to simple pooling, because the state is polynomial, token access is selective, and the resulting block remains a universal sequence-to-sequence approximator under the stated conditions.

4. Architectural realizations and masking schemes

PoM is used as the core mixer inside otherwise standard transformer-like blocks. In both the general transformer paper and the speech paper, the block takes the form

$X \in \mathbb{R}^{d \times n}$9

with residual connections and a standard two-layer feed-forward network. In practice, this means that attention can be removed while leaving most of the surrounding architecture unchanged (Picard et al., 7 Apr 2026, Feillet et al., 28 Feb 2026).

The published design space includes pure PoM stacks and hybrid stacks. In speech, all multi-head attention layers in a Conformer-based BEST-RQ encoder are replaced by PoM. In the broader transformer study, hybrid models alternate PoM and attention, sometimes using local attention rather than full attention. The reported pattern is consistent across tasks: pure PoM often preserves most of the baseline quality, while hybrid PoM/attention designs frequently recover or slightly exceed the attention baseline while retaining substantial efficiency gains (Picard et al., 7 Apr 2026).

Masked and causal variants are also explicit. For a binary mask $$\text{PoM}(X) = W_o \left[ \sigma(W_s X) \circ H(X)\,\mathbf{1}^{\top} \right],$$0, the aggregated state becomes token dependent: $$\text{PoM}(X) = W_o \left[ \sigma(W_s X) \circ H(X)\,\mathbf{1}^{\top} \right],$$1 With a lower-triangular causal mask, the state admits a recurrence

$$\text{PoM}(X) = W_o \left[ \sigma(W_s X) \circ H(X)\,\mathbf{1}^{\top} \right],$$2

which yields $$\text{PoM}(X) = W_o \left[ \sigma(W_s X) \circ H(X)\,\mathbf{1}^{\top} \right],$$3 per-token autoregressive inference. The diffusion paper further develops block-causal masks for video, so that tokens within a frame or block can mix non-causally while time remains causal across blocks (Picard et al., 7 Apr 2026, Picard et al., 2024).

5. Self-supervised speech encoders

In speech representation learning, PoM is integrated into a BEST-RQ pipeline with Mel filterbank inputs and a Conformer backbone. For base models of about $$\text{PoM}(X) = W_o \left[ \sigma(W_s X) \circ H(X)\,\mathbf{1}^{\top} \right],$$4M parameters, the encoder uses 12 PoM blocks with polynomial order $$\text{PoM}(X) = W_o \left[ \sigma(W_s X) \circ H(X)\,\mathbf{1}^{\top} \right],$$5 and expansion factor $$\text{PoM}(X) = W_o \left[ \sigma(W_s X) \circ H(X)\,\mathbf{1}^{\top} \right],$$6; for large models of about $$\text{PoM}(X) = W_o \left[ \sigma(W_s X) \circ H(X)\,\mathbf{1}^{\top} \right],$$7M parameters, it uses 24 blocks with $$\text{PoM}(X) = W_o \left[ \sigma(W_s X) \circ H(X)\,\mathbf{1}^{\top} \right],$$8 and $$\text{PoM}(X) = W_o \left[ \sigma(W_s X) \circ H(X)\,\mathbf{1}^{\top} \right],$$9. Pre-training is performed on LibriSpeech-960h, and ASR fine-tuning is carried out on the LibriSpeech-100h clean subset with a 3-layer linear CTC decoder (Feillet et al., 28 Feb 2026).

The reported base-model word error rates place PoM close to regular MHA and clearly ahead of SummaryMixing. For the $W_s \in \mathbb{R}^{kD \times d}$0M PoM model, the reported WERs are $W_s \in \mathbb{R}^{kD \times d}$1 on test-clean, $W_s \in \mathbb{R}^{kD \times d}$2 on test-clean with a LLM, $W_s \in \mathbb{R}^{kD \times d}$3 on test-other, and $W_s \in \mathbb{R}^{kD \times d}$4 on test-other with a LLM. The paper states that PoM is strictly better than SummaryMixing in all conditions, close to regular MHA, and competitive with other linear-complexity alternatives such as Mamba and HyperConformer, although RelPos MHA and RoPE MHA remain the strongest overall in WER (Feillet et al., 28 Feb 2026).

The efficiency results are equally central. On A100 GPUs, for an 80-second input sequence, PoM uses $W_s \in \mathbb{R}^{kD \times d}$5 less memory than RelPos MHA. Its runtime is reported as close to SummaryMixing and faster than RoPE MHA, while inference time and VRAM usage grow much more slowly than MHA as input length increases from 10 to 80 seconds (Feillet et al., 28 Feb 2026).

The speech study also examines PoM variants. A highest-degree-only mode-jump variant hurts performance relative to the base design, selective PoM does not outperform the base design, and frequency-aware variants slightly improve WER for underfitting encoders but do not help once pre-training is continued. The reported conclusion is that base PoM, which combines all orders up to $W_s \in \mathbb{R}^{kD \times d}$6 and mixes all features, is the best design in that setting (Feillet et al., 28 Feb 2026).

6. Multidomain transformer replacement

The general transformer study evaluates PoM across text generation, handwritten text recognition, Earth observation, 3D semantic segmentation, and class-conditional image generation. In GPT-2-small-style language modeling, a fully PoM model is slightly below the attention baseline on validation loss and some downstream metrics, but a hybrid PoM plus local-attention model reaches a validation loss of $W_s \in \mathbb{R}^{kD \times d}$7, essentially matching the baseline $W_s \in \mathbb{R}^{kD \times d}$8, while slightly improving HellaSwag, Winogrande, and MMLU. At long contexts, the same study reports PoM as $W_s \in \mathbb{R}^{kD \times d}$9–$W_o \in \mathbb{R}^{d \times kD}$0 faster than MHA with FlashAttention (Picard et al., 7 Apr 2026).

In OCR on the LAM dataset, pure PoM slightly trails attention on CER and WER, while the hybrid model matches the attention baseline on both single-line and multi-line settings and still improves throughput on long sequences. In Earth observation on PASTIS, TSViPoM preserves OA $W_o \in \mathbb{R}^{d \times kD}$1 and mIoU $W_o \in \mathbb{R}^{d \times kD}$2 against a retrained TSViT baseline with OA $W_o \in \mathbb{R}^{d \times kD}$3 and mIoU $W_o \in \mathbb{R}^{d \times kD}$4, while delivering about $W_o \in \mathbb{R}^{d \times kD}$5 higher throughput. In 3D segmentation, pure PoM is slightly below PointTransformerV3 on mIoU, whereas hybrid variants recover or slightly exceed baseline accuracy with modest speedups (Picard et al., 7 Apr 2026).

The diffusion literature predates the broader transformer paper and already frames PoM as a linear-time replacement for attention in image and video generation. In ImageNet $W_o \in \mathbb{R}^{d \times kD}$6 class-conditional generation, DiPoM-XL/2 trained with diffusion loss and sampled with 250 DDIM steps achieves FID $W_o \in \mathbb{R}^{d \times kD}$7, IS $W_o \in \mathbb{R}^{d \times kD}$8, precision $W_o \in \mathbb{R}^{d \times kD}$9, and recall $\sigma$0. In video generation on WebVid-2M, block-causal masking improves dynamic degree, human action, temporal style consistency, and overall consistency relative to the no-mask variant, while slightly reducing subject and background consistency (Picard et al., 2024).

The later large-scale image-generation experiments extend the same idea. Replacing attention by PoM in SiT-style generators yields SiPoM-L/2 with FID $\sigma$1 versus $\sigma$2 for SiT-L/2 and SiPoM-XL/2 with the same FID $\sigma$3 as SiT-XL/2, while the PoM models are faster; at $\sigma$4, SiPoM-XL/2 is reported at $\sigma$5 images per second versus $\sigma$6 for SiT-XL/2, and the runtime gap widens at high resolution because attention scales roughly quartically with resolution whereas PoM scales roughly quadratically through token count (Picard et al., 7 Apr 2026).

7. Alternative uses of the term

The expression “Polynomial Mixer” also appears in unrelated mathematical contexts. In number theory and probability, it is used informally for the effect of a nonlinear polynomial map $\sigma$7 on a random binary expansion $\sigma$8. There the emphasis is not neural token mixing but digit mixing induced by polynomial interactions and carries. For $\sigma$9, one main theorem states that if

$H(X)$0

for all sufficiently large $H(X)$1, then $H(X)$2 is almost surely absolutely normal; another theorem shows that if $H(X)$3 and $H(X)$4, then $H(X)$5 is almost surely not normal in base $H(X)$6 (Manai, 6 Jun 2026).

A second distinct usage appears in cryptography through the MMO problem, where a prospective PoM primitive is based on

$H(X)$7

With known moduli $H(X)$8 and $H(X)$9, reconstruction is equivalent to a closest vector problem in a lattice with respect to the $$H(X) = \left[ \;h(W_1 X) \;\middle|\; h(W_1 X) \circ h(W_2 X) \;\middle|\; \dots \;\middle|\; \prod_{m=1}^k h(W_m X) \right]\mathbf{1},$$0 norm; with secret moduli, the paper states that no efficient solution is known (Garcia-Morchon et al., 2014).

A third usage arises in nonparametric statistics, where polynomial expansions are used to estimate a mixing density in continuous mixture models. There the construction is an orthogonal series estimator involving Legendre polynomials, with minimax upper and lower bounds for the mean integrated squared error, and in exponential mixtures an adaptive estimator is obtained when the projection order satisfies $$H(X) = \left[ \;h(W_1 X) \;\middle|\; h(W_1 X) \circ h(W_2 X) \;\middle|\; \dots \;\middle|\; \prod_{m=1}^k h(W_m X) \right]\mathbf{1},$$1 (Rebafka et al., 2010).

These usages share the vocabulary of mixing and polynomials, but they describe different mathematical objects: a neural token mixer, a digit-mixing phenomenon under polynomial maps, a modular polynomial reconstruction problem, and a polynomial-basis estimator for continuous mixtures.