---
title: State-Scale MLP Branch Architecture
url: https://www.emergentmind.com/topics/state-scale-mlp-branch
type: topic
---

# State-Scale MLP Branch Architecture

A State Scale MLP Branch refers to a neural network architectural pattern that attaches multi-scale output branches—"tails"—at various hidden layers of a Multi-Layer Perceptron (MLP) to enable multi-resolution or level-of-detail (LoD) signal representation and supervision. This approach is prominent in two recent lines of research: (1) Tailed Multi-Layer Perceptron (T-MLP) for LoD signal modeling in implicit neural representations and (2) multi-branch MLP modules, such as PCAS-MLP, for extracting multi-scale features in U-Net-style architectures for image segmentation [2509.00066][2304.12615]. Such designs facilitate hierarchical supervision and efficient capture of both coarse and fine signal components within a single model.

## 1. Architectural Principles of State Scale MLP Branches

The central architectural principle is the attachment of dedicated output branches—"tails"—to hidden layers of an MLP backbone. Each tail captures the signal at a distinct effective scale due to the progressively increasing receptive field and abstraction level of deeper layers. In the context of T-MLP, for an MLP backbone of depth $k$ (hidden layers), $k$ tails are appended, with each tail producing an intermediate output $\mathbf{t}_i$ from hidden activation $\mathbf{h}_i$ [2509.00066]. The first tail is linear, while subsequent tails employ a multiplicative elementwise design to facilitate residual learning. Each intermediate output is accumulated to form a sequence of LoD predictions:
\[
\mathbf{y}_i = \sum_{j=1}^i \mathbf{t}_j,\quad i=1,\dots,k
\]
Similarly, in PCAS-MLP within STM-UNet, multiple convolutional branches of varying kernel sizes (1×1, 3×3, 5×5) operate in parallel on intermediate MLP representations, providing a multi-scale response before further processing [2304.12615].

## 2. Mathematical Formulations

In T-MLP, the forward computation is defined as follows [2509.00066]:
- Hidden layer activations:
  \[
  \mathbf{h}_0 = \mathbf{x},\quad
  \mathbf{h}_\ell = \sigma(\mathbf{W}_\ell\,\mathbf{h}_{\ell-1} + \mathbf{b}_\ell),\quad \ell=1\dots k
  \]
- Output of the first tail:
  \[
  \mathbf{t}_1 = \mathbf{W}^{\text{out}_1}\,\mathbf{h}_1 + \mathbf{b}^{\text{out}_1}
  \]
- Output of subsequent tails (elementwise multiplication for residuals):
  \[
  \mathbf{t}_\ell
  =
  (\mathbf{W}^{\text{out}_{\ell,0}}\,\mathbf{h}_\ell + \mathbf{b}^{\text{out}_{\ell,0}})
  \circ
  (\mathbf{W}^{\text{out}_{\ell,1}}\,\mathbf{h}_\ell + \mathbf{b}^{\text{out}_{\ell,1}})
  ,\quad \ell=2\dots k
  \]
- Progressive output:
  \[
  \mathbf{y}_\ell = \mathbf{y}_{\ell-1} + \mathbf{t}_\ell
  \]
Where $\circ$ denotes the Hadamard product.

The PCAS-MLP module combines three convolutions in parallel, then processes the result with nonlinearity (GELU), dropout, and serial height/width shifts, before a final channel-wise linear projection and residual connection [2304.12615].

## 3. Training and Supervision Strategies

Each tail output $\mathbf{y}_\ell$ in T-MLP is trained with explicit supervision against the ground-truth signal or its residual, yielding a layerwise supervised, multi-loss framework:
\[
L^{(\ell)} = \mathbb{E}_{\mathbf{x}\sim\mathcal{Q}} \bigl[\,\|\mathbf{y}_\ell(\mathbf{x}) - y_{\text{gt}}(\mathbf{x})\|^p\,\bigr]
\]
The overall loss is
\[
\mathcal{L}_{\text{total}} = \sum_{\ell=1}^k \lambda_\ell\,L^{(\ell)}
\]
where $\lambda_\ell$ are per-tail weights, typically increasing with depth ($\lambda_1 \approx 0$, and higher $\lambda$ for finer predictions).

Progressive training curricula may first freeze all but the shallowest tail, then incrementally activate deeper tails to stabilize coarse-to-fine learning. This approach alleviates vanishing gradients and forces each layer to encode semantically meaningful multi-scale features [2509.00066].

## 4. Practical Implementations and Hyperparameters

Practical instantiations of State Scale MLP Branch architectures differ by application domain:

- **T-MLP for LoD Signal Representation [2509.00066]:**
  - MLP backbone width (per layer): $N$ (e.g., 256).
  - Nonlinearities: ReLU or sine (as used in SIREN).
  - Number of tails: $k$ (number of hidden layers).
  - Output heads: linear for first tail; elementwise-multiplicative for others.
  - Loss weights: example $\lambda = (0,0.5,0.5,0.5,2.5)$ for $k=5$.
  - Optimizer: Adam, initial LR $10^{-4}$–$3\times10^{-4}$, decayed at 0.25× schedule.
  - Data: Random pixel sampling for images, surface-oriented sampling for SDFs.

- **PCAS-MLP for STM-UNet [2304.12615]:**
  - Operates at C=512 channels in the bottleneck.
  - Parallel convolutions: 1×1, 3×3 (pad=1), 5×5 (pad=2).
  - Axial shifting blocks $S_h, S_w$ (e.g., 16), with uniform shift offsets.
  - GELU activation, Dropout $p=0.1$ at two points.
  - Optimizer: Adam, LR $1\times10^{-4}$.
  - U-Net stages: channels [32, 64, 128, 256, 512].
  - Training: 300 epochs, batch size 8, 512×512 image size.

## 5. Comparative Performance and Applications

State Scale MLP Branch designs deliver substantial empirical gains in representation efficiency, quality, and speed, especially for LoD tasks.

| Model/Task         | Params        | 3D Shape CD$\downarrow$ | 3D Normal Cons$\uparrow$ | Image PSNR$\uparrow$ | ISIC-2018 mIoU$\uparrow$ |
|:-------------------|:-------------|:-----------------------:|:-----------------------:|:--------------------:|:------------------------:|
| T-MLP [2509.00066] | ∼266k        | 1.513                   | 98.03%                  | 30.63                | –                        |
| SIREN              | ∼265k        | 1.769                   | –                       | 28.02                | –                        |
| NGLOD              | 1.35M        | 1.975                   | –                       | –                    | –                        |
| BACON              | 264k         | 1.787                   | –                       | –                    | –                        |
| BANF               | 2.08M        | 4.683                   | –                       | –                    | –                        |
| Baseline U-NeXt-L  | 3.99M        | –                       | –                       | –                    | 0.7720                   |
| STM-UNet           | 6.12M        | –                       | –                       | –                    | 0.7984                   |

T-MLP demonstrates lower Chamfer distance and higher normal consistency on 3D modeling benchmarks, outperforming SIREN, NGLOD, BACON, and BANF at comparable or lower parameter counts. For 2D image regression (DIV2K), T-MLP achieves a PSNR gain of over 2 dB versus SIREN. Training times are an order of magnitude faster than comparable LoD approaches.

PCAS-MLP, as integrated into STM-UNet, improves mIoU and mDice by approximately +2.6% and +1.1% over AS-MLP baselines, with smaller model size increases compared to pure CNN or hybrid Transformer models [2304.12615].

## 6. Significance and Broader Implications

State Scale MLP Branch architectures enable a single network to efficiently encode a continuous, explicit level-of-detail hierarchy. Shallow tails capture low-frequency coarse structure, while deeper branches add high-frequency detail. Layerwise supervision provides effective gradient propagation, mitigating vanishing gradient issues typical of deep MLPs. For practical systems, progressive output can support real-time coarse-to-fine rendering or streaming.

A plausible implication is that these multi-scale branching MLP techniques may inform the design of future signal representation models and efficient vision backbones by providing robust layerwise decodability and multi-scale feature extraction in compact, end-to-end trainable frameworks.

## 7. Related Approaches and Future Directions

While State Scale MLP Branches as described in T-MLP and PCAS-MLP are specific to LoD and segmentation domains, the principle of layerwise multi-scale branching is extensible to other tasks requiring explicit scale hierarchies, including neural rendering, adaptive transmission, and multi-resolution generative modeling. The use of elementwise-multiplicative branches (T-MLP) and parallel axial-shifted filtering (PCAS-MLP) suggests a convergence between efficient MLP-based models and convolutional/attention counterparts. Future directions include scaling to larger models, integration with attention, and exploration of dynamic per-branch supervision schedules to further enhance adaptation to varying task requirements [2509.00066][2304.12615].

Source: https://www.emergentmind.com/topics/state-scale-mlp-branch