Self-Attention Fusion: N-to-One Mechanism

Updated 2 April 2026

The paper introduces a novel self-attention fusion operator that flexibly integrates an arbitrary number of input features without rigid constraints.
It details various architectural paradigms, including flat token, hierarchical, and windowed fusion, each balancing computational complexity and expressiveness.
Empirical evaluations in applications like medical imaging and activity recognition demonstrate significant gains in accuracy and efficiency over traditional fusion methods.

Self-attention for arbitrary N-to-one fusion refers to the use of self-attention mechanisms for flexibly combining an arbitrary number (N) of heterogeneous or homogeneous feature vectors, modalities, or views into a single fused output suitable for downstream decision, regression, or classification tasks. Unlike rigid early/late fusion or pairwise attention, self-attention fusion methods are designed to accommodate variable N, extract higher-order interdependence, provide robust behavior in missing-data regimes, and offer compositional flexibility with manageable computational complexity.

1. Foundations: Self-Attention as a Flexible N-to-One Fusion Operator

The core principle is that self-attention consists of learnable projections from a set of N input entities—these may be feature vectors (as in SAF-Net (Adalioglu et al., 2023)), spatial tokens (as in MFSWFM (Huang et al., 12 Apr 2025)), modality-representative vectors (as in FOAA (Alwazzan et al., 2024)), or higher-order composites (as in poly-attention (Chakrabarti et al., 2 Feb 2026))—to queries, keys, and values, which jointly determine how information from all sources is combined in the fused representation.

The canonical single-head self-attention fusion block considered in SAF-Net is representative:

Project each input $x_i \in \mathbb{R}^D$ to the model dimension $d_\mathrm{model}$ ,
Learn independent Q/K/V matrices, yielding $Q, K, V \in \mathbb{R}^{N \times d_k}$ ,
Form similarity scores $S = Q K^\top / \sqrt{d_k}$ and attention-weighted sums $F = \operatorname{softmax}(S) V$ ,
Collapse $N$ fused outputs to a single fused vector $f_\text{pool} = \frac{1}{N} \sum_{i=1}^N F_i$ . This construction is agnostic to $N$ and can be trivially extended to multi-head attention or variable input sets, with masking as needed for missing data (Adalioglu et al., 2023).

In all aforementioned schemas, attention learns the dependency structure among the $N$ inputs and adaptively weights their contributions in the fused result, allowing complex interaction patterns to be automatically modeled.

2. Architectural Variants: Patterns of N-way Fusion

Several architectural paradigms for N-to-one self-attention fusion have emerged:

Flat Token Fusion: All $N$ input feature vectors are treated symmetrically as tokens and jointly fused via global or windowed self-attention layers, possibly followed by pooling or readout (Liu et al., 2022, Adalioglu et al., 2023, Alwazzan et al., 2024). This yields a fused vector capturing both individual and cross-input dependencies.
Hierarchical or Bottlenecked Fusion: Models such as MBT insert bottleneck fusion latents to restrict cross-modal exchange to a reduced-dimensional channel, decreasing cross-modal complexity from $d_\mathrm{model}$ 0 to $d_\mathrm{model}$ 1 while still supporting arbitrary $d_\mathrm{model}$ 2 (Nagrani et al., 2021).
Windowed or Shifted-Window Fusion: To address quadratic complexity in long-token or multi-view regimes, shifted-window self-attention (e.g., MFSWFM (Huang et al., 12 Apr 2025)) partitions inputs into fixed-size regions and alternates self-attention inside and across window boundaries, enabling linear scaling in $d_\mathrm{model}$ 3.
Poly-attention and Higher-order Schemas: Polynomial-tensor-based attention generalizes self-attention to higher-order (R-ary) dependencies, allowing N-way hypergraph or compositional fusion at the cost of higher computational complexity, but with special cases (tree-attention) permitting efficient $d_\mathrm{model}$ 4 computation for certain composition tasks (Chakrabarti et al., 2 Feb 2026).
Arithmetic-operator Attention: FOAA fuses $d_\mathrm{model}$ 5 flattened modality features not only by dot-product but via a battery of outer addition, subtraction, product, and division operations, which are row-wise normalized and summed across heads before aggregation into a single vector (Alwazzan et al., 2024).
Masked and Non-invasive Attention: In missing-modality or side-information-rich regimes, mechanisms such as SFusion (Liu et al., 2022) and NOVA (Liu et al., 2021) enable variable $d_\mathrm{model}$ 6 fusion without zero-padding; NOVA guides attention using arbitrary side-features in Q/K while preserving the primary representation in V.

The table below summarizes selected design choices:

Method	Token Granularity	Fusion Output	$d_\mathrm{model}$ 7-Scalability	Notable Mechanism
SAF-Net (Adalioglu et al., 2023)	View-level vectors	Avg-pool	Arbitrary $d_\mathrm{model}$ 8	Global self-attn, pooling
MFSWFM (Huang et al., 12 Apr 2025)	Patch tokens × views	Spatial map	Any $d_\mathrm{model}$ 9	Shifted window, pooling
SFusion (Liu et al., 2022)	R tokens/modality	Feature map	Any $Q, K, V \in \mathbb{R}^{N \times d_k}$ 0	MHA+modal att., no padding
FOAA (Alwazzan et al., 2024)	Flattened vector/modality	Single vector	Any $Q, K, V \in \mathbb{R}^{N \times d_k}$ 1	Outer arithmetic ops
MBT (Nagrani et al., 2021)	All tokens/stream	Fused tokens	Any $Q, K, V \in \mathbb{R}^{N \times d_k}$ 2	Bottleneck fusion, per-layer
Poly-attention (Chakrabarti et al., 2 Feb 2026)	Arbitrary tokens	Fused tokens	Any ( $Q, K, V \in \mathbb{R}^{N \times d_k}$ 3)	General polynomial/tensor attn

All entries support arbitrary $Q, K, V \in \mathbb{R}^{N \times d_k}$ 4 by design.

3. Mathematical Formulation and Mechanistic Details

The mathematical core typically consists of learnable Q, K, V projections and a softmax normalization. Differences arise in:

Fusion attention score computation: Traditional models use $Q, K, V \in \mathbb{R}^{N \times d_k}$ 5, while FOAA employs four distinct elementwise operators (add, subtract, multiply, divide) to enhance the diversity of similarity functions (Alwazzan et al., 2024). Poly-attention generalizes further to arbitrary multilinear polynomials over the input vectors (Chakrabarti et al., 2 Feb 2026).
Masking for missing or variable-length inputs: SAF-Net and SFusion support binary masks for attention scoring, enabling fusion over variable or missing input sets without resorting to zero-padding or synthetic vectors (Adalioglu et al., 2023, Liu et al., 2022).
Pooling/aggregation: After attention, pooling or averaging reduces the $Q, K, V \in \mathbb{R}^{N \times d_k}$ 6 fused representations to one. SAF-Net uses average pooling; FOAA uses reduce-mean post-head combination; SFusion applies softmax-derived weights voxel-wise across modalities (Liu et al., 2022).

Multi-head generalizations split $Q, K, V \in \mathbb{R}^{N \times d_k}$ 7 across heads, each with independent Q/K/V, concatenated and projected for output, as in standard Transformers; this offers richer subspace fusion and feature disentanglement (Adalioglu et al., 2023, Liu et al., 2022, Lee et al., 2023).

Poly-attention's higher-order variant for N-to-1 fusion expresses the fused vector as a sum over N-ary softmax-weighted elementwise products:

$Q, K, V \in \mathbb{R}^{N \times d_k}$ 8

where $Q, K, V \in \mathbb{R}^{N \times d_k}$ 9 is the appropriate attention polynomial, reducing to standard self-attention for $S = Q K^\top / \sqrt{d_k}$ 0 (Chakrabarti et al., 2 Feb 2026).

4. Computational Complexity and Scalability

Quadratic complexity in $S = Q K^\top / \sqrt{d_k}$ 1 is a major practical bottleneck for full self-attention; hence windowed or bottlenecked schemas are prominent:

Shifted-window attention reduces overall cost from $S = Q K^\top / \sqrt{d_k}$ 2 to $S = Q K^\top / \sqrt{d_k}$ 3 by computing attention only within fixed-size local windows, alternating between regular and shifted partitions to guarantee cross-view mixing (Huang et al., 12 Apr 2025).
MBT’s bottleneck fusion reduces cross-modal cost from $S = Q K^\top / \sqrt{d_k}$ 4 to $S = Q K^\top / \sqrt{d_k}$ 5 per layer by routing all interaction through $S = Q K^\top / \sqrt{d_k}$ 6 bottleneck tokens (Nagrani et al., 2021).
Poly-attention admits arbitrary-order correlation modeling, but generic mechanisms require $S = Q K^\top / \sqrt{d_k}$ 7 for order- $S = Q K^\top / \sqrt{d_k}$ 8 attention; tree-structured specializations permit $S = Q K^\top / \sqrt{d_k}$ 9 computation for fixed-depth composition tasks (Chakrabarti et al., 2 Feb 2026).

FOAA incurs approximately 5× full self-attention cost (4 outer ops plus final V-agglication) for each block (Alwazzan et al., 2024).

Mechanisms supporting missing modalities (e.g., SFusion) process only the observed $F = \operatorname{softmax}(S) V$ 0 modalities, with all equations automatically adapting to $F = \operatorname{softmax}(S) V$ 1 via dynamic tokenization and appropriate softmax normalization, avoiding computational cost for unobserved streams (Liu et al., 2022).

5. Empirical Effects, Comparative Performance, and Ablations

Experiments across modalities and application domains demonstrate significant gains from N-to-one self-attention fusion:

Medical imaging (SAF-Net, MFSWFM, FOAA): SAF-Net achieves 88.26% precision, 77.64% sensitivity, and 78.13% accuracy for myocardial infarction detection from multi-view echocardiography (Adalioglu et al., 2023). MFSWFM reduces inference time by ~40% and improves accuracy by 2–7 points vs. global/fixed-self-attention baselines (Huang et al., 12 Apr 2025). FOAA outperforms both standard self-attention and prior arithmetic-operator-based fusion by 6–12 points in AUC or F1-micro on tumor classification tasks (Alwazzan et al., 2024).
Activity recognition and segmentation (SFusion): Outperforms selection-based and convolutional-gated fusers by 2–6 points, with ablations showing notable drops if either self-attention or modal-attention is ablated (Liu et al., 2022).
Multimodal video/audio (MBT): Improves mAP on AudioSet (mini) from 41.8% (late fusion) to 43.9% (bottleneck fusion) and achieves superior accuracy-top-1 on VGGSound (64.1% vs. 52.3/51.2% audio/visual only) (Nagrani et al., 2021).
Robot relocalization (FusionLoc): Self-attention-based fusion yields up to ~30% improvement in positional error and large orientation accuracy gains versus concatenation-only or unimodal baselines (Lee et al., 2023).
Sequential recommendation (NOVA): Consistently outperforms BERT4Rec with invasive fusion, stably providing 1–13% relative gains in HR@10 across public and industrial datasets, with negligible additional overhead (Liu et al., 2021).

Ablation studies confirm:

The necessity of explicit position encoding in multi-view fusion (Huang et al., 12 Apr 2025),
The value of operator diversity in FOAA (Alwazzan et al., 2024),
The importance of both self-attention and modal-attention sub-blocks in SFusion (Liu et al., 2022),
The optimality of mid-layer fusion and small bottleneck size in MBT (Nagrani et al., 2021).

6. Extensions, Generalizations, and Future Directions

Self-attention for arbitrary N-to-one fusion has been generalized in multiple directions:

Flexible support for dynamic and missing input sets: Tokenization and masking remove the need for synthetic or zero-padded features (Liu et al., 2022, Adalioglu et al., 2023).
Local/global, hierarchical, and compositional structures: Shifted-window, bottleneck, tree-attention allow modeling of both fine and global dependencies at manageable cost (Huang et al., 12 Apr 2025, Nagrani et al., 2021, Chakrabarti et al., 2 Feb 2026).
Operator-enriched and higher-order correlation capture: FOAA outer-arithmetic and poly-attention frameworks explicitly capture multi-way statistical interactions, enabling modeling of tasks (e.g., compositional or hyperedge detection) not accessible to plain self-attention (Alwazzan et al., 2024, Chakrabarti et al., 2 Feb 2026).
Modalities and output types: Techniques generalize across vision, audio, language, medical, robotic sensor, and recommendation systems; outputs range from fused vectors, attention maps for decoder input, and modality-aware context sets.

A plausible implication is that further advances in N-to-one self-attention fusion will require new designs that balance expressiveness (high-order, richer similarity measures), efficiency (linear or quadratic complexity in $F = \operatorname{softmax}(S) V$ 2), and robustness to data variation, as formalized in the complexity/expressiveness trade-offs of poly-attention (Chakrabarti et al., 2 Feb 2026).

7. Relationships to Other Attention Schemes and Open Challenges

N-to-one self-attention fusion shares commonalities with cross-attention, early/late fusion, and ensemble strategies, but its distinguishing features are:

Symmetric, order-free treatment of inputs,
Learnable weighting and interdependency modeling across arbitrary sets,
On-the-fly adaptation to missing or variable $F = \operatorname{softmax}(S) V$ 3,
The possibility (with suitable parameterizations) of capturing higher-order or compositional effects.

Challenges remain in scaling poly-attention beyond constant $F = \operatorname{softmax}(S) V$ 4 regimes, efficiently blending context from extremely large numbers of input entities, and learning truly "modal-invariant" fusion schemes that generalize across data heterogeneity. Future research directions are suggested by ongoing empirical and theoretical work on the relationships between parameter norms, position encoding, expressiveness, and computational bounds (Chakrabarti et al., 2 Feb 2026, Huang et al., 12 Apr 2025).