---
title: Hierarchical Grouped-MLP Fusion
url: https://www.emergentmind.com/topics/hierarchical-grouped-mlp-fusion
type: topic
---

# Hierarchical Grouped-MLP Fusion

Hierarchical grouped-MLP fusion refers to a family of neural network design paradigms in which multilayer perceptrons (MLPs) are hierarchically organized and fused across feature, spatial, or modal groupings, often with learnable grouping, pooling, or grouping-permutation strategies. This class includes architectures such as Group-Connected MLPs, CS-Mixer, Butterfly/Dimension Mixers, and gMLP-based hierarchical vision models, each leveraging structured group-wise mixing and hierarchical pooling or merging to increase efficiency, expressive power, or multi-level interaction—all without global, fully dense connections. Core to this paradigm are learnable or structured grouping operations, local groupwise MLP interactions, and the aggregation of higher-order dependencies via hierarchical tree-like or pyramid mechanisms.

## 1. Core Principles of Hierarchical Grouped-MLP Fusion

The central principle underlying hierarchical grouped-MLP fusion is partitioning a high-dimensional input (features, tokens, modalities, or spatial locations) into groups, applying trainable MLP-based mixing or gating within each group, and hierarchically merging or permuting groups at deeper layers. This process can be realized with:

- **Learned or structured group assignments:** Via softmaxed grouping matrices with entropy regularization, fixed windowing, or pre-specified blockings.
- **Local group-wise MLP fusion:** MLPs operate only on group-contained features or tokens, reducing parameter and computational complexity.
- **Hierarchical aggregation:** Deeper layers merge, pool, or mix outputs from child groups, forming tree or pyramid hierarchies capturing high-order dependencies.
- **Cross-scale and multi-level fusion:** Multi-stage or skip-connected groupwise fusions enable both fine and global interactions.

These strategies directly contrast the all-to-all connectivity of traditional dense MLPs, offering efficiency, modularity, and—when groups and their merging are engineered or learned judiciously—state-of-the-art performance in domains ranging from tabular data to large-scale vision tasks [1912.09600, 2308.13363, 2311.18735, 2208.11718].

## 2. Learned Grouping and Local Mixing Mechanisms

Several methodologies exploit learnable grouping for fusing input features:

- **Soft Grouping Matrices:** In Group-Connected MLPs (GMLP), each layer uses a soft grouping matrix $G^t \in \mathbb{R}^{d \times g_t}$ parameterized by logits $Z^t$ and temperature $T$. As $T \to 0$, features are essentially assigned to a single group, promoting hard partitions. An entropy penalty $-\sum_{i,j} G^t_{ij} \log G^t_{ij}$ is added to the loss, balancing sharp assignments against smooth optimization [1912.09600].
- **Blockwise and Windowed Local MLPs:** In gSwin, the spatial feature map is partitioned into non-overlapping or shifted windows, with each window processed by a windowed spatial gating unit (SGU) that mixes tokens/grouped locations via an MLP and gating mechanism. The SGU’s gating matrix is block-diagonal over all windows, keeping parameter growth in check [2208.11718].
- **Butterfly/Dimension Groupings:** Hierarchically permuted groupings, inspired by the FFT’s butterfly structure, perform blockwise MLP mixing with a new group structure at each stage, such that after $O(\log_r N)$ steps all coordinates can communicate globally. These groupings are determined by radix-$r$ representations of indices and systematic permutations [2311.18735].
- **Low-Rank and Grouped Spatio-Channel Mixing:** CS-Mixer splits spatial tokens into groups in both local and global configurations and applies low-rank, multi-head MLP mixing in each, capturing interactions across height, width, and channels jointly via grouped transforms [2308.13363].

These mechanisms impose and exploit structural sparsity and partitioning for scalable, expressive mixing, and group-based regularization.

## 3. Hierarchical Fusion and Aggregation Structures

Hierarchical grouped-MLP fusion leverages tree- or pyramid-like aggregation, enabling:

- **Tree-like Pooling Hierarchies:** After groupwise MLP mixing, GMLP merges groups in a binary-tree fashion using mean, max, or linear pooling operators ($P^t$). With each pooling, the number of groups reduces by half, increasing group capacity at coarser levels. The top-level concatenation feeds into a final output MLP [1912.09600].
- **Pyramidal Staging in Vision:** Models like gSwin and CS-Mixer use multi-stage spatial downsampling (via patch merging or pooling) to form spatial and channel pyramids. This yields a multiscale hierarchy akin to CNN backbones but relies on grouped MLP or gating fusion within each stage [2208.11718, 2308.13363].
- **Repeated Permutation-Hierarchies (Butterfly):** Dimension Mixer/Butterfly MLPs recursively permute and hierarchically regroup dimensions according to radix-decompositions, ensuring efficient all-to-all communication with O(log N) mixing depth, and groupwise local MLPs per permutation stage [2311.18735].
- **Dense Stacking Across Modalities:** In dense multimodal fusion, shared MLP fusion layers are stacked between unimodal subnetworks at every depth, with skip and dense connections among all shared fusion points, allowing multi-level cross-modal supervision and gradient propagation [1810.03414].

These hierarchical strategies ensure that higher-level units integrate increasingly global contextual information, either spatially, across features, or across modalities.

## 4. Computational and Memory Efficiency

Hierarchical grouped-MLP fusion enables dramatic reductions in parameter count and FLOPs compared to dense MLPs:

- **Complexity Scaling:** The Butterfly/Dimension Mixer achieves total parameters $O(N m \log_r N)$ versus dense $O(N^2)$ for fixed per-block hidden size $m$ (e.g., $m=r$). FLOPs match, with $O(N r \log_r N)$ versus dense $O(N^2)$. Empirically, 4× lower MACs and parameter count can be attained at minimal performance loss (e.g., CIFAR-10 with 144K params and 5M MACs for Butterfly vs. 352K/20M for dense, with <1% accuracy drop) [2311.18735].
- **Groupwise and Block-diagonal Efficiency:** In gSwin, gating weight matrices scale as $K \cdot (M^2 \cdot M^2 + M^2)$ per block (e.g., $12 \cdot 2450=29400$ for $M=7$, $K=12$), much lower than global $N_i^2$ when $N_i$ is large [2208.11718].
- **Low-Rank Grouped Mixing:** CS-Mixer's low-rank, grouped mixing attains $O(m N (c^2 + c d + L^2 d^2))$ per layer, with $L=g^2 \ll N$, and $d\ll c$, avoiding the $O(N^2 c^2)$ explosion of naïve 3-axis full mixing [2308.13363].
- **Sparse Activation Storage:** With nearly one-hot group assignment matrices, GMLP’s memory for assignment storage is $O(nd)$ in lower layers and $O(g_{t-1} g_t)$ in higher ones, offering linear scale with group count rather than quadratic with dimension [1912.09600].

A plausible implication is that such architectures are especially advantageous for large input spaces, long sequences, or high-resolution visual inputs, where dense all-to-all mixing is prohibitive or over-parameterized.

## 5. Empirical Performance and Domain Applications

Hierarchical grouped-MLP fusion has been applied and evaluated across several domains:

- **Tabular and Non-structured Data:** GMLP attains state-of-the-art classification performance on tabular and mixed-feature datasets, automatically discovering expressive feature subsets and interactions [1912.09600].
- **Computer Vision:** 
    - gSwin (“gated MLP in shifted window hierarchy”) achieves higher accuracy than equivalently sized Swin Transformers on ImageNet-1K (+0.4%), COCO detection (+0.45 box-AP) and ADE20K (+1.8 mIoU), with ~20% fewer parameters [2208.11718].
    - CS-Mixer demonstrates ImageNet-1k top-1 accuracy of 83.2% at 13.7 GFLOPs and 94M parameters (CS-Mixer-L), surpassing same-compute 2-axis mixers by >1.5% and matching or exceeding axial/H–C mixers with much lower compute [2308.13363].
- **Sequence Modeling and Efficient Attention:** Butterfly-MLP and Butterfly Attention in Vision Transformers and LRA tasks provide sub-quadratic scaling (e.g., 20% lower memory, up to +2% accuracy versus dense attention) and unique ability to handle sequence lengths up to 16K+ [2311.18735].
- **Multimodal Fusion:** Dense Multimodal Fusion (DMF) leverages hierarchically layered shared MLP fusion across modalities, resulting in richer joint representations and accelerated convergence, particularly when one modality is noisy or absent [1810.03414].

A plausible implication is that hierarchical grouped-MLP fusion enhances robustness and generalization, particularly in settings with heterogeneous input structure or multi-resolution dependencies.

## 6. Structural Attributes and Generalization Extensions

Hierarchical grouped-MLP fusion subsumes and generalizes numerous classical and contemporary ideas:

- **Permutation and Grouping Flexibility:** The index-based groupings in Butterfly/Dimension Mixers are theoretically universal for invertible transformations, recovering the full connectivity of dense MLPs as group size and depth scale [2311.18735].
- **Multi-head and Multi-modality:** Grouped mixing can be extended with multi-head gating (as in gSwin) or with task-specific grouping functions for multi-task or cross-modal learning [2208.11718, 1912.09600].
- **Shifted and Shift-free Grouping:** Alternation of standard and cyclically shifted window partitions in vision ensures eventual full neighborhood interaction across spatial locations [2208.11718].
- **Low-Rank and Dynamic Mixers:** Low-rank grouped mixing (CS-Mixer) balances capacity and efficiency, with the rank $d$ and group size $g$ as critical hyperparameters for trade-offs in expressive power versus compute [2308.13363].
- **Generalization to Nonlinear and Hierarchical Domains:** The same principles extend across domains—tabular, sequential, visual, and multimodal—with appropriate design of groupings, local mixing blocks, and hierarchical aggregation.

## 7. Representative Architectures: Comparison Table

Below is a summary contrasting representative hierarchical grouped-MLP fusion models and their key attributes:

| Model         | Grouping Strategy                   | Hierarchical Fusion             | Core Domain         |
|---------------|------------------------------------|---------------------------------|---------------------|
| GMLP [1912.09600]          | Learnable, sparse softmax groups         | Tree-like pooling, pooling after MLP | Tabular, general    |
| gSwin [2208.11718]      | Non-overlapping and shifted windows      | Multi-stage spatial pyramid      | Vision              |
| CS-Mixer [2308.13363] | Local/global grouped, low-rank            | Four-stage cross-scale hierarchy | Vision              |
| Butterfly MLP [2311.18735]  | Radix-based, permuted groupings           | Log-depth permutation-mixing tree | Sequence, vision    |
| DMF [1810.03414]               | Dense shared MLP layers between modalities | Hierarchical shared-layer stacking  | Multimodal fusion   |

Together, these architectures demonstrate the flexibility and effectiveness of hierarchical grouped-MLP fusion as a general deep learning design paradigm, with efficiency, expressive capacity, and domain adaptability at its core.

Source: https://www.emergentmind.com/topics/hierarchical-grouped-mlp-fusion