Tailed Multi-Layer Perceptron (T-MLP)

• T-MLP is a neural network model that attaches output tails to each hidden layer to enable coarse-to-fine, multi-level detail representation.
• It overcomes conventional MLP limitations by providing direct supervision at every stage, facilitating progressive refinement of signal approximations.
• Empirical results in 3D shape SDFs, image fitting, and point cloud reconstruction show T-MLP achieves superior fidelity with a compact parameter budget.

Tailed Multi-Layer Perceptron (T-MLP) is a neural architecture for level-of-detail (LoD) signal representation that modifies a standard Multi-Layer Perceptron (MLP) so that implicit neural representations (INRs) can natively support multiple granularities of the same signal, from coarse to fine. Its defining mechanism is to attach an output branch, or tail, to every hidden layer and to train these tails with direct supervision so that earlier layers produce coarse approximations while deeper layers progressively add finer residual details. In the reported formulation, T-MLP is evaluated on 3D shape signed distance function (SDF) representation, surface reconstruction from point clouds, and image fitting, with additional NeRF experiments mentioned in supplementary material (Yang et al., 26 Aug 2025).

1. Problem setting and motivation

In the reported usage, LoD representation means representing the same signal at multiple granularities, from coarse to fine. The stated motivations are efficient transmission, where a coarse version can be transmitted first and then refined progressively; adaptive rendering or reconstruction, where lower or higher detail can be selected depending on constraints; and storage/parameter scalability, where meaningful partial models are supported rather than requiring the full network at once (Yang et al., 26 Aug 2025).

The central critique of a conventional MLP is that it is inherently single-scale. If an MLP has $N$ hidden layers, the first $N-1$ layers alone do not form a valid LoD predictor. In this account, a standard MLP therefore does not naturally support multiple LoDs, progressive transmission, or partial parameter usage for coarse rendering. A second limitation is that only the last hidden representation is directly supervised; earlier hidden layers are optimized only indirectly by backpropagation. The paper argues that this is inefficient and may weaken learning in early layers because of vanishing-gradient effects. A third limitation is that quantization or pruning are not, by themselves, a LoD solution, because storing multiple LoDs would ordinarily require multiple independent network versions (Yang et al., 26 Aug 2025).

A key empirical observation motivates the architecture: in a single MLP, hidden representations at increasing depth tend to capture progressively higher-frequency components of the signal. This suggests that earlier hidden layers already contain coarse, low-frequency approximations, but that these approximations are poor LoD predictors in a standard MLP because they lack direct supervision. T-MLP is designed to expose and train this latent coarse-to-fine hierarchy explicitly (Yang et al., 26 Aug 2025).

2. Architecture and mathematical formulation

T-MLP is built from a standard MLP backbone with an output tail after each hidden layer. If the network has $k$ hidden layers, T-MLP adds $k$ tails, one per hidden layer. Each tail produces an intermediate prediction, and these predictions are accumulated to form increasingly detailed outputs: $\mathbf{y}_1, \mathbf{y}_2, \ldots, \mathbf{y}_k.$ Each $\mathbf{y}_i$ corresponds to a different LoD (Yang et al., 26 Aug 2025).

The paper defines a standard MLP as

$$\begin{aligned} \mathbf{h}_0 & = \mathbf{x}, \ \mathbf{h}_{i} & =\sigma\left(\mathbf{W}_{i} \mathbf{h}_{i-1}+\mathbf{b}_{i}\right), \; i=1, \ldots, k \ \mathbf{y} & =\mathbf{W}^{out} \mathbf{h}_{k}+\mathbf{b}^{out}, \end{aligned}$$

and T-MLP as

$$\begin{aligned} \mathbf{h}_0 & = \mathbf{x}, \ \mathbf{h}_{i} & =\sigma\left(\mathbf{W}_{i} \mathbf{h}_{i-1}+\mathbf{b}_{i}\right),\ \mathbf{t}_{i} & =\mathbf{W}_{i}^{out} \mathbf{h}_{i}+\mathbf{b}_{i}^{out},\ \mathbf{y}_{0} & = \mathbf{0},\ \mathbf{y}_{i} & =\mathbf{y}_{i-1} + \mathbf{t}_{i}, \; i=1, \ldots, k. \end{aligned}$$

Equivalently,

$\mathbf{y}_i = \sum_{j=1}^{i}\mathbf{t}_j.$

Within this formulation, the semantic role of the layers is explicit. Hidden layer 1 / tail 1 learns a coarse approximation of the target signal. Hidden layer 2 / tail 2 learns the residual between the target and the coarse approximation. Later tails continue this residual refinement. The paper states that each layer is designed to focus on learning the residual between two consecutive levels of detail (Yang et al., 26 Aug 2025).

For tails $i>1$, the architecture introduces a multiplicative formulation: $N-1$0 where $N-1$1 is the Hadamard product. In the stated design, tail 1 is linear, while tails 2 through $N-1$2 are products of two affine projections of $N-1$3. The rationale given is that residual magnitudes are often very small, typically smaller than 1, and that a small-magnitude value can be represented as the product of larger-magnitude values (Yang et al., 26 Aug 2025).

For the main experiments, the reported configuration uses 5 hidden layers, 256 hidden features per layer, sine activation, and SIREN initialization for both image fitting and 3D shape representation. The first tail is often not supervised in practice because the subnetwork up to the first tail has too few parameters to produce a useful result (Yang et al., 26 Aug 2025).

3. Supervision, losses, and optimization

The overall training objective is a multi-output supervision objective: $N-1$4 where $N-1$5 is the output at LoD $N-1$6, and $N-1$7 is the loss weight for output $N-1$8. Larger $N-1$9 emphasizes accuracy at LoD $k$0. The paper also identifies this weighting as important and limiting, because improving one LoD can hurt others (Yang et al., 26 Aug 2025).

Task-specific losses are differentiated by signal type. For 3D shape SDF representation, the paper uses an $k$1 loss over SDF values. For image fitting, it uses an $k$2 loss over RGB values. For surface reconstruction from point clouds, it uses the loss function from StEik, and the formula is not reproduced in the provided description (Yang et al., 26 Aug 2025).

The main training strategy in the reported experiments is joint end-to-end training with the weighted sum loss above. The paper does not use staged or cascaded training in its main results. It does mention a progressive training strategy in the limitations and future-work discussion: initially train only the parameters from input to the first output tail, then gradually add more layers as training proceeds. That strategy is described as promising and comparable to the current strategy, but not as the main reported method (Yang et al., 26 Aug 2025).

For both 3D shape representation and image fitting, the practical tail weights are

$k$3

This means that the first tail is not supervised in practice, the final LoD is emphasized heavily, and the intermediate LoDs receive moderate supervision. The stated reason is that the first tail usually yields low-quality results because the subnetwork up to the first hidden layer has too few parameters (Yang et al., 26 Aug 2025).

Optimization details are also task-specific. For 3D shape representation, the network has 5 hidden layers and 256 hidden units each, uses sine activation and SIREN initialization, is trained with Adam, uses initial learning rate $k$4, runs for 10k iterations, and decays the learning rate by multiplying by $k$5 at iterations 7000, 8000, 9000. At each iteration it samples 100k points: 20% random in bounding box, 40% surface points, and 40% near-surface points obtained by Gaussian perturbation of surface points with $k$6. All shapes are normalized to $k$7. For image fitting, the optimizer is again Adam, with initial learning rate $k$8, 10k iterations, the same decay schedule, and the same tail weights (Yang et al., 26 Aug 2025).

4. Empirical evaluation and reported results

The reported baselines are Fourier Features, SIREN, NGLOD, BACON, and BANF. Fourier Features and SIREN do not support LoD, whereas NGLOD, BACON, and BANF do. BANF was reimplemented by the authors for 3D shape representation because code was unavailable (Yang et al., 26 Aug 2025).

The evaluation spans three primary tasks. For 3D shape representation, the datasets are the Thingi32 subset of Thingi10K and the Stanford 3D Scanning Repository, and the task is to fit signed distance functions (SDFs) at multiple LoDs. Meshes are extracted by Marching Cubes at resolution $k$9, 500k points are sampled from each mesh, and the metrics are Chamfer Distance (CD), where lower is better, and Normal Consistency (NC), where higher is better. For surface reconstruction from point clouds, the dataset is again the Stanford 3D Scanning Repository. For image fitting, the dataset is DIV2K, at resolutions $k$0 and $k$1, evaluated with PSNR and SSIM (Yang et al., 26 Aug 2025).

At the highest LoD for 3D shape representation, the reported results are as follows:

Dataset	Method	Result
Thingi10K	Fourier Features	CD 1.871, NC 98.22
Thingi10K	SIREN	CD 1.769, NC 99.19
Thingi10K	NGLOD	CD 1.975, NC 99.02
Thingi10K	BACON	CD 1.787, NC 99.06
Thingi10K	BANF	CD 4.683, NC 96.08
Thingi10K	T-MLP	CD 1.740, NC 99.39
Stanford	Fourier Features	CD 1.763, NC 95.52
Stanford	SIREN	CD 1.613, NC 96.90
Stanford	NGLOD	CD 1.711, NC 96.86
Stanford	BACON	CD 1.638, NC 96.63
Stanford	BANF	CD 1.870, NC 94.82
Stanford	T-MLP	CD 1.513, NC 98.03

The parameter counts are reported as T-MLP: 266k, SIREN: 265k, Fourier Features: 263k, BACON: 264k, NGLOD: 1.35M, and BANF: 2.08M. This indicates that T-MLP adds LoD capability with almost no parameter inflation relative to SIREN. Training time per shape is reported as 0.815 min for Fourier Features, 2.988 min for SIREN, 44.80 min for NGLOD, 6.217 min for BACON, 67.31 min for BANF, and 3.548 min for T-MLP (Yang et al., 26 Aug 2025).

For image fitting, the reported results are:

Resolution	Method	Result
$k$2	Fourier Features	PSNR 29.39, SSIM 90.09
$k$3	SIREN	PSNR 33.39, SSIM 94.18
$k$4	BACON	PSNR 31.73, SSIM 89.81
$k$5	BANF	PSNR 32.46, SSIM 95.40
$k$6	T-MLP	PSNR 37.60, SSIM 96.82
$k$7	Fourier Features	PSNR 25.81, SSIM 77.73
$k$8	SIREN	PSNR 28.02, SSIM 83.83
$k$9	BACON	PSNR 24.43, SSIM 58.20
$\mathbf{y}_1, \mathbf{y}_2, \ldots, \mathbf{y}_k.$0	BANF	PSNR 27.39, SSIM 85.48
$\mathbf{y}_1, \mathbf{y}_2, \ldots, \mathbf{y}_k.$1	T-MLP	PSNR 30.63, SSIM 88.52

The paper states that T-MLP clearly outperforms all baselines on image fitting, and that LoD comparisons for images are in supplementary material (Yang et al., 26 Aug 2025).

Qualitatively, the reported comparisons indicate that standard MLP hidden layers contain progressively finer content, but their intermediate outputs are not faithful LoD representations; T-MLP makes those intermediate outputs more meaningful through direct supervision. For surface reconstruction from point clouds, the paper highlights that, for clean point clouds, higher-detail outputs recover finer geometry, whereas for noisy point clouds, lower-detail outputs suppress noise via underfitting. This suggests a practical denoising utility for multi-LoD inference (Yang et al., 26 Aug 2025).

5. Ablations, interpretation, and relation to MLP modularity

Ablation studies on Stanford 3D shape representation isolate the contributions of supervision, residual design, and multiplicative tails. The reported results are Standard MLP: CD 1.613, NC 96.90; MLP w Res. Conn.: CD 1.540, NC 97.67; T-MLP w/o Res. Des.: CD 1.582, NC 97.52; T-MLP w Res. Conn.: CD 1.517, NC 97.97; and Full T-MLP: CD 1.513, NC 98.03. For the multiplicative design, T-MLP w/o multiplicative design gives CD 1.521, NC 97.94, compared with Full T-MLP: CD 1.513, NC 98.03. The paper interprets these results as showing that layer-wise supervision helps, residual tail design helps further, residual connections from ResNet improve a standard MLP but do not match T-MLP, and adding residual connections to T-MLP has almost no effect (Yang et al., 26 Aug 2025).

The reported explanation is that T-MLP already embodies a residual principle: each hidden representation has its own tail parameters, and the accumulated output

$\mathbf{y}_1, \mathbf{y}_2, \ldots, \mathbf{y}_k.$2

causes later layers to focus on new detail rather than relearning previously represented content. The justification for the depth-wise coarse-to-fine behavior is explicitly empirical rather than formal-theoretical. A probing experiment trains a full $\mathbf{y}_1, \mathbf{y}_2, \ldots, \mathbf{y}_k.$3-layer standard MLP $\mathbf{y}_1, \mathbf{y}_2, \ldots, \mathbf{y}_k.$4, removes its final hidden and output layer to obtain $\mathbf{y}_1, \mathbf{y}_2, \ldots, \mathbf{y}_k.$5, freezes the remaining hidden layers and trains a new output head, and repeats down to $\mathbf{y}_1, \mathbf{y}_2, \ldots, \mathbf{y}_k.$6. This is reported to reveal that standard MLP hidden layers progressively capture higher-frequency content, but still provide poor LoD representations compared with T-MLP’s directly supervised tails (Yang et al., 26 Aug 2025).

A broader conceptual comparison can be made to the network-construction formalism in "Multilayer Perceptron Algebra" (Peng, 2017), but only indirectly. That paper does not introduce a model explicitly called a Tailed Multi-Layer Perceptron (T-MLP). What it does provide is a general algebra of MLP construction and transformation, including O-Product Net for multi-output assembly and Identical Extension-$\mathbf{y}_1, \mathbf{y}_2, \ldots, \mathbf{y}_k.$7 for appending an identity-preserving layer. This does not constitute a T-MLP theory, but it provides a mathematical vocabulary for appended layers, parallel branches, output decomposition, and modular extension. A plausible implication is that T-MLP can be situated within a longer line of work on MLP modularization, even though its specific tail-per-hidden-layer LoD mechanism is introduced separately and directly only in (Yang et al., 26 Aug 2025).

6. Strengths, limitations, and scope of use

The reported strengths are sixfold. First, T-MLP provides native LoD in a single MLP-like network. Second, it supports progressive transmission, because early layers already produce a coarse output and later layers refine it. Third, it provides better supervision of hidden layers by attaching a tail to every hidden layer. Fourth, it achieves competitive or superior final accuracy, so the architecture is not only adding LoD functionality. Fifth, it has low overhead relative to SIREN-like networks. Sixth, for noisy point clouds or noisy images, lower-detail outputs can suppress high-frequency noise (Yang et al., 26 Aug 2025).

The limitations are explicit. The main one is loss-weight sensitivity: the quality at each LoD depends on the choice of $\mathbf{y}_1, \mathbf{y}_2, \ldots, \mathbf{y}_k.$8, and increasing one weight generally improves that LoD at the expense of others. A second limitation is that the first tail is often weak; in practice, the reported setup uses $\mathbf{y}_1, \mathbf{y}_2, \ldots, \mathbf{y}_k.$9, suggesting that the coarsest LoD is not fully satisfactory in the current architecture. A third limitation is the absence of a strong theory for the coarse-to-fine depth behavior; the justification is empirical rather than formally derived. A fourth limitation is that progressive training is mentioned as promising but is not developed as the main method in the paper (Yang et al., 26 Aug 2025).

Within the stated evidence, T-MLP is particularly suitable when a single compact INR is required to support multiple LoDs, progressive transmission, or better utilization of intermediate hidden layers. The reported applications are image fitting, 3D SDF/shape representation, and surface reconstruction from point clouds. The paper also notes additional NeRF experiments in supplementary material, but no further details are provided in the supplied description. If only the final full-resolution output matters and LoD or progressive use is irrelevant, a standard MLP may remain simpler; however, the reported results suggest that T-MLP can also improve highest-LoD fidelity while preserving an MLP-like parameter budget (Yang et al., 26 Aug 2025).