---
title: Tailed Multi-Layer Perceptron (T-MLP)
url: https://www.emergentmind.com/topics/tailed-multi-layer-perceptron-t-mlp
type: topic
---

# Tailed Multi-Layer Perceptron (T-MLP)

Tailed Multi-Layer Perceptron (T-MLP) is a neural architecture for level-of-detail (LoD) signal representation that modifies a standard Multi-Layer Perceptron (MLP) so that implicit neural representations (INRs) can natively support multiple granularities of the same signal, from coarse to fine. Its defining mechanism is to attach an output branch, or **tail**, to every hidden layer and to train these tails with direct supervision so that earlier layers produce coarse approximations while deeper layers progressively add finer residual details. In the reported formulation, T-MLP is evaluated on 3D shape signed distance function (SDF) representation, surface reconstruction from point clouds, and image fitting, with additional NeRF experiments mentioned in supplementary material [2509.00066].

## 1. Problem setting and motivation

In the reported usage, **LoD representation** means representing the same signal at multiple granularities, from coarse to fine. The stated motivations are **efficient transmission**, where a coarse version can be transmitted first and then refined progressively; **adaptive rendering or reconstruction**, where lower or higher detail can be selected depending on constraints; and **storage/parameter scalability**, where meaningful partial models are supported rather than requiring the full network at once [2509.00066].

The central critique of a conventional MLP is that it is **inherently single-scale**. If an MLP has \(N\) hidden layers, the first \(N-1\) layers alone do not form a valid LoD predictor. In this account, a standard MLP therefore does not naturally support **multiple LoDs**, **progressive transmission**, or **partial parameter usage for coarse rendering**. A second limitation is that only the last hidden representation is directly supervised; earlier hidden layers are optimized only indirectly by backpropagation. The paper argues that this is inefficient and may weaken learning in early layers because of vanishing-gradient effects. A third limitation is that quantization or pruning are not, by themselves, a LoD solution, because storing multiple LoDs would ordinarily require multiple independent network versions [2509.00066].

A key empirical observation motivates the architecture: in a single MLP, hidden representations at increasing depth tend to capture progressively higher-frequency components of the signal. This suggests that earlier hidden layers already contain coarse, low-frequency approximations, but that these approximations are poor LoD predictors in a standard MLP because they lack direct supervision. T-MLP is designed to expose and train this latent coarse-to-fine hierarchy explicitly [2509.00066].

## 2. Architecture and mathematical formulation

T-MLP is built from a standard MLP backbone with an output tail after each hidden layer. If the network has \(k\) hidden layers, T-MLP adds \(k\) tails, one per hidden layer. Each tail produces an intermediate prediction, and these predictions are accumulated to form increasingly detailed outputs:
\[
\mathbf{y}_1, \mathbf{y}_2, \ldots, \mathbf{y}_k.
\]
Each \(\mathbf{y}_i\) corresponds to a different LoD [2509.00066].

The paper defines a standard MLP as
\[
\begin{aligned}
\mathbf{h}_0 & = \mathbf{x}, \\
\mathbf{h}_{i} & =\sigma\left(\mathbf{W}_{i} \mathbf{h}_{i-1}+\mathbf{b}_{i}\right), \; i=1, \ldots, k \\
\mathbf{y} & =\mathbf{W}^{out} \mathbf{h}_{k}+\mathbf{b}^{out},
\end{aligned}
\]
and T-MLP as
\[
\begin{aligned}
\mathbf{h}_0 & = \mathbf{x}, \\
\mathbf{h}_{i}  & =\sigma\left(\mathbf{W}_{i} \mathbf{h}_{i-1}+\mathbf{b}_{i}\right),\\
\mathbf{t}_{i} & =\mathbf{W}_{i}^{out} \mathbf{h}_{i}+\mathbf{b}_{i}^{out},\\
\mathbf{y}_{0} & = \mathbf{0},\\
\mathbf{y}_{i} & =\mathbf{y}_{i-1} + \mathbf{t}_{i}, \; i=1, \ldots, k.
\end{aligned}
\]
Equivalently,
\[
\mathbf{y}_i = \sum_{j=1}^{i}\mathbf{t}_j.
\]

Within this formulation, the semantic role of the layers is explicit. **Hidden layer 1 / tail 1** learns a coarse approximation of the target signal. **Hidden layer 2 / tail 2** learns the residual between the target and the coarse approximation. Later tails continue this residual refinement. The paper states that each layer is designed to focus on learning the residual between two consecutive levels of detail [2509.00066].

For tails \(i>1\), the architecture introduces a **multiplicative formulation**:
\[
\mathbf{t}_{i} = (\mathbf{W}_{i_0}^{out} \mathbf{h}_{i}+\mathbf{b}_{i_0}^{out}) \circ (\mathbf{W}_{i_1}^{out} \mathbf{h}_{i}+\mathbf{b}_{i_1}^{out}) , \quad i=2, \ldots, k,
\]
where \(\circ\) is the Hadamard product. In the stated design, tail 1 is linear, while tails 2 through \(k\) are products of two affine projections of \(\mathbf{h}_i\). The rationale given is that residual magnitudes are often very small, typically smaller than 1, and that a small-magnitude value can be represented as the product of larger-magnitude values [2509.00066].

For the main experiments, the reported configuration uses **5 hidden layers**, **256 hidden features per layer**, **sine activation**, and **SIREN initialization** for both image fitting and 3D shape representation. The first tail is often not supervised in practice because the subnetwork up to the first tail has too few parameters to produce a useful result [2509.00066].

## 3. Supervision, losses, and optimization

The overall training objective is a multi-output supervision objective:
\[
\mathcal{L}_{total} = \sum_{i = 1}^{k}\lambda_i\mathcal{L}(\mathbf{y}_{i}),
\]
where \(\mathbf{y}_i\) is the output at LoD \(i\), and \(\lambda_i\) is the loss weight for output \(i\). Larger \(\lambda_i\) emphasizes accuracy at LoD \(i\). The paper also identifies this weighting as important and limiting, because improving one LoD can hurt others [2509.00066].

Task-specific losses are differentiated by signal type. For **3D shape SDF representation**, the paper uses an \(L_1\) loss over SDF values. For **image fitting**, it uses an \(L_2\) loss over RGB values. For **surface reconstruction from point clouds**, it uses the loss function from **StEik**, and the formula is not reproduced in the provided description [2509.00066].

The main training strategy in the reported experiments is **joint end-to-end training** with the weighted sum loss above. The paper does **not** use staged or cascaded training in its main results. It does mention a **progressive training strategy** in the limitations and future-work discussion: initially train only the parameters from input to the first output tail, then gradually add more layers as training proceeds. That strategy is described as promising and comparable to the current strategy, but not as the main reported method [2509.00066].

For both 3D shape representation and image fitting, the practical tail weights are
\[
(\lambda_{1}, \lambda_{2}, \lambda_{3}, \lambda_{4}, \lambda_{5}) = (0, 0.5, 0.5, 0.5, 2.5).
\]
This means that the **first tail is not supervised** in practice, the final LoD is emphasized heavily, and the intermediate LoDs receive moderate supervision. The stated reason is that the first tail usually yields low-quality results because the subnetwork up to the first hidden layer has too few parameters [2509.00066].

Optimization details are also task-specific. For **3D shape representation**, the network has 5 hidden layers and 256 hidden units each, uses sine activation and SIREN initialization, is trained with **Adam**, uses initial learning rate \(3 \times 10^{-4}\), runs for **10k** iterations, and decays the learning rate by multiplying by \(0.25\) at iterations **7000, 8000, 9000**. At each iteration it samples **100k points**: **20% random in bounding box**, **40% surface points**, and **40% near-surface points obtained by Gaussian perturbation of surface points with \(\sigma = 0.01\)**. All shapes are normalized to \([-1,1]^3\). For **image fitting**, the optimizer is again **Adam**, with initial learning rate \(2.5 \times 10^{-4}\), **10k** iterations, the same decay schedule, and the same tail weights [2509.00066].

## 4. Empirical evaluation and reported results

The reported baselines are **Fourier Features**, **SIREN**, **NGLOD**, **BACON**, and **BANF**. Fourier Features and SIREN do **not** support LoD, whereas NGLOD, BACON, and BANF do. BANF was reimplemented by the authors for 3D shape representation because code was unavailable [2509.00066].

The evaluation spans three primary tasks. For **3D shape representation**, the datasets are the **Thingi32 subset of Thingi10K** and the **Stanford 3D Scanning Repository**, and the task is to fit **signed distance functions (SDFs)** at multiple LoDs. Meshes are extracted by **Marching Cubes** at resolution \(512^3\), **500k points** are sampled from each mesh, and the metrics are **Chamfer Distance (CD)**, where lower is better, and **Normal Consistency (NC)**, where higher is better. For **surface reconstruction from point clouds**, the dataset is again the Stanford 3D Scanning Repository. For **image fitting**, the dataset is **DIV2K**, at resolutions \(512 \times 512\) and \(1024 \times 1024\), evaluated with **PSNR** and **SSIM** [2509.00066].

At the highest LoD for **3D shape representation**, the reported results are as follows:

| Dataset | Method | Result |
|---|---|---|
| Thingi10K | Fourier Features | CD 1.871, NC 98.22 |
| Thingi10K | SIREN | CD 1.769, NC 99.19 |
| Thingi10K | NGLOD | CD 1.975, NC 99.02 |
| Thingi10K | BACON | CD 1.787, NC 99.06 |
| Thingi10K | BANF | CD 4.683, NC 96.08 |
| Thingi10K | **T-MLP** | **CD 1.740, NC 99.39** |
| Stanford | Fourier Features | CD 1.763, NC 95.52 |
| Stanford | SIREN | CD 1.613, NC 96.90 |
| Stanford | NGLOD | CD 1.711, NC 96.86 |
| Stanford | BACON | CD 1.638, NC 96.63 |
| Stanford | BANF | CD 1.870, NC 94.82 |
| Stanford | **T-MLP** | **CD 1.513, NC 98.03** |

The parameter counts are reported as **T-MLP: 266k**, **SIREN: 265k**, **Fourier Features: 263k**, **BACON: 264k**, **NGLOD: 1.35M**, and **BANF: 2.08M**. This indicates that T-MLP adds LoD capability with almost no parameter inflation relative to SIREN. Training time per shape is reported as **0.815 min** for Fourier Features, **2.988 min** for SIREN, **44.80 min** for NGLOD, **6.217 min** for BACON, **67.31 min** for BANF, and **3.548 min** for **T-MLP** [2509.00066].

For **image fitting**, the reported results are:

| Resolution | Method | Result |
|---|---|---|
| \(512 \times 512\) | Fourier Features | PSNR 29.39, SSIM 90.09 |
| \(512 \times 512\) | SIREN | PSNR 33.39, SSIM 94.18 |
| \(512 \times 512\) | BACON | PSNR 31.73, SSIM 89.81 |
| \(512 \times 512\) | BANF | PSNR 32.46, SSIM 95.40 |
| \(512 \times 512\) | **T-MLP** | **PSNR 37.60, SSIM 96.82** |
| \(1024 \times 1024\) | Fourier Features | PSNR 25.81, SSIM 77.73 |
| \(1024 \times 1024\) | SIREN | PSNR 28.02, SSIM 83.83 |
| \(1024 \times 1024\) | BACON | PSNR 24.43, SSIM 58.20 |
| \(1024 \times 1024\) | BANF | PSNR 27.39, SSIM 85.48 |
| \(1024 \times 1024\) | **T-MLP** | **PSNR 30.63, SSIM 88.52** |

The paper states that T-MLP clearly outperforms all baselines on image fitting, and that LoD comparisons for images are in supplementary material [2509.00066].

Qualitatively, the reported comparisons indicate that standard MLP hidden layers contain progressively finer content, but their intermediate outputs are not faithful LoD representations; T-MLP makes those intermediate outputs more meaningful through direct supervision. For surface reconstruction from point clouds, the paper highlights that, for **clean point clouds**, higher-detail outputs recover finer geometry, whereas for **noisy point clouds**, lower-detail outputs suppress noise via underfitting. This suggests a practical denoising utility for multi-LoD inference [2509.00066].

## 5. Ablations, interpretation, and relation to MLP modularity

Ablation studies on **Stanford 3D shape representation** isolate the contributions of supervision, residual design, and multiplicative tails. The reported results are **Standard MLP: CD 1.613, NC 96.90**; **MLP w Res. Conn.: CD 1.540, NC 97.67**; **T-MLP w/o Res. Des.: CD 1.582, NC 97.52**; **T-MLP w Res. Conn.: CD 1.517, NC 97.97**; and **Full T-MLP: CD 1.513, NC 98.03**. For the multiplicative design, **T-MLP w/o multiplicative design** gives **CD 1.521, NC 97.94**, compared with **Full T-MLP: CD 1.513, NC 98.03**. The paper interprets these results as showing that layer-wise supervision helps, residual tail design helps further, residual connections from ResNet improve a standard MLP but do not match T-MLP, and adding residual connections to T-MLP has almost no effect [2509.00066].

The reported explanation is that T-MLP already embodies a residual principle: each hidden representation has its own tail parameters, and the accumulated output
\[
\mathbf{y}_i = \mathbf{y}_{i-1} + \mathbf{t}_i
\]
causes later layers to focus on new detail rather than relearning previously represented content. The justification for the depth-wise coarse-to-fine behavior is explicitly empirical rather than formal-theoretical. A probing experiment trains a full \(K\)-layer standard MLP \(M^K\), removes its final hidden and output layer to obtain \(M^{K-1}\), freezes the remaining hidden layers and trains a new output head, and repeats down to \(M^1\). This is reported to reveal that standard MLP hidden layers progressively capture higher-frequency content, but still provide poor LoD representations compared with T-MLP’s directly supervised tails [2509.00066].

A broader conceptual comparison can be made to the network-construction formalism in "Multilayer Perceptron Algebra" [1701.04968], but only indirectly. That paper does **not** introduce a model explicitly called a **Tailed Multi-Layer Perceptron (T-MLP)**. What it does provide is a general algebra of MLP construction and transformation, including **O-Product Net** for multi-output assembly and **Identical Extension-\(\beta\)** for appending an identity-preserving layer. This does not constitute a T-MLP theory, but it provides a mathematical vocabulary for appended layers, parallel branches, output decomposition, and modular extension. A plausible implication is that T-MLP can be situated within a longer line of work on MLP modularization, even though its specific tail-per-hidden-layer LoD mechanism is introduced separately and directly only in [2509.00066].

## 6. Strengths, limitations, and scope of use

The reported strengths are sixfold. First, T-MLP provides **native LoD in a single MLP-like network**. Second, it supports **progressive transmission**, because early layers already produce a coarse output and later layers refine it. Third, it provides **better supervision of hidden layers** by attaching a tail to every hidden layer. Fourth, it achieves **competitive or superior final accuracy**, so the architecture is not only adding LoD functionality. Fifth, it has **low overhead** relative to SIREN-like networks. Sixth, for noisy point clouds or noisy images, **lower-detail outputs can suppress high-frequency noise** [2509.00066].

The limitations are explicit. The main one is **loss-weight sensitivity**: the quality at each LoD depends on the choice of \(\lambda_i\), and increasing one weight generally improves that LoD at the expense of others. A second limitation is that the **first tail is often weak**; in practice, the reported setup uses \(\lambda_1 = 0\), suggesting that the coarsest LoD is not fully satisfactory in the current architecture. A third limitation is the absence of a strong theory for the coarse-to-fine depth behavior; the justification is empirical rather than formally derived. A fourth limitation is that **progressive training** is mentioned as promising but is not developed as the main method in the paper [2509.00066].

Within the stated evidence, T-MLP is particularly suitable when a single compact INR is required to support multiple LoDs, progressive transmission, or better utilization of intermediate hidden layers. The reported applications are **image fitting**, **3D SDF/shape representation**, and **surface reconstruction from point clouds**. The paper also notes additional NeRF experiments in supplementary material, but no further details are provided in the supplied description. If only the final full-resolution output matters and LoD or progressive use is irrelevant, a standard MLP may remain simpler; however, the reported results suggest that T-MLP can also improve highest-LoD fidelity while preserving an MLP-like parameter budget [2509.00066].

Source: https://www.emergentmind.com/topics/tailed-multi-layer-perceptron-t-mlp