---
title: Thick Outputs in Neural Networks
url: https://www.emergentmind.com/topics/thick-outputs
type: topic
---

# Thick Outputs in Neural Networks

Searching arXiv for recent and directly relevant papers on “thick outputs” and adjacent usages in ML/statistics.
Thick outputs are a neural-network design in which each hidden state or node emits multiple parallel values and then reduces them to a single propagated value by a maximization operation. In the formulation introduced by Thick-Net, thickness is treated as a third architectural dimension, distinct from width and depth, for sequential modeling. The central claim is that expanding along thickness preserves some of the representational richness associated with wider or deeper networks while changing the optimization and regularization behavior through a per-node max selection mechanism [1911.08074]. In adjacent statistical literature, the phrase also appears in other senses, notably for multi-dimensional simulator outputs in tensor-variate Gaussian process regression and for unthinned Markov chain output, but those uses are terminologically distinct from the Thick-Net architecture [2502.10319][2005.03952].

## 1. Architectural definition

In Thick-Net, thickness means that each hidden state or node outputs multiple parallel values of size \(n\), computed using several independent parameter sets. These parallel outputs are then down-sampled at each node by applying a maximum operation, yielding the final single output per node to be passed forward in the network [1911.08074].

This construction differs from ordinary widening. A wide architecture increases the number of units per layer and passes all of them forward. Thick-Net instead keeps the width and depth constant, expands the hidden state in a third dimension, and then applies max-pooling along thickness so that only one value per node is propagated. The design is therefore not simply a re-parameterized wide layer; its defining feature is the coexistence of parallel alternatives and a deterministic selection rule at each node [1911.08074].

The paper positions this mechanism as a response to a familiar tradeoff in sequence learning: wider structures become more prone to overfitting, while deeper structures are difficult to optimize. Thickness is presented as an alternative expansion axis intended to change that tradeoff rather than merely scaling conventional width or depth [1911.08074].

## 2. Formal mechanism

Given an input vector \( \mathbf{x} \in \mathbb{R}^r \) and parameter matrices \( W^{(i)} \in \mathbb{R}^{m\times r} \) for \( i=1,\dots,n \), Thick-Net computes
\[
\mathbf{v}_i = W^{(i)} \mathbf{x} \in \mathbb{R}^m.
\]
The thick output is then defined componentwise by
\[
\mathrm{MAX}(\mathbf{v}_1,\ldots,\mathbf{v}_n)
=
\left[
\max_{i=1..n} v_{1i},
\max_{i=1..n} v_{2i},
\ldots,
\max_{i=1..n} v_{mi}
\right]^T,
\]
where \(v_{ji}\) is the \(j\)-th component of \(\mathbf{v}_i\) [1911.08074].

The formal role of this operator is twofold. First, it allows each node to maintain multiple parameterized alternatives before reduction. Second, it makes the reduction itself nonlinear and selective, rather than averaging or concatenating the parallel outputs. The empirical ablations reported in the paper attribute substantive importance to this exact choice: average or random selection among thick outputs degrades performance compared to maximization [1911.08074].

This mechanism also clarifies why thickness is not identical to an ensemble. The parallel branches are local to each node or hidden transformation, and only the maximum value survives at each step. The resulting architecture remains a single model with an internal selection operation rather than a collection of independent predictors [1911.08074].

## 3. Recurrent instantiations and optimization behavior

The paper develops Thick-Net in the context of sequential modeling, including recurrent neural networks and LSTMs. In the Thick-LSTM variant, the maximization operation replaces the standard linear transform in each gate—input, forget, output, and candidate—so that each gate receives a thick transformation rather than a single affine map [1911.08074].

The optimization claim is that Thick-Net is easier to optimize than vanilla structures due to the large dropout affiliated with it. The summary further states that all parameters participate in backprop at each step, and the max operation acts as a structural regularizer similar to dropout, but without zeroing gradients as in standard dropout. This is used to explain the reported faster convergence relative to both wider and deeper LSTM or RNN baselines [1911.08074].

The regularization argument is likewise architectural rather than external. Because only one value per node survives at each step, thick outputs naturally incorporate large effective dropout while still preserving a parameter-rich hidden transformation. The paper also argues that the maximization operation acts as a nonlinear down-sampler, preventing hypothesis space from growing uncontrollably in the way associated with width, and thereby encouraging better out-of-sample generalization [1911.08074].

## 4. Empirical performance in sequential tasks

Thick-Net is evaluated on four sequential tasks: the adding problem, permuted sequential MNIST, text classification, and language modeling. The reported results are presented as evidence that the model can improve accuracy with faster convergence and facilitate a better generalization ability [1911.08074].

| Task | Baseline or comparison | Thick-Net result |
|---|---|---|
| Adding problem (\(T=500\)) | 10-layer LSTM, wide LSTM | Only Thick-Net converges rapidly |
| Permuted sequential MNIST | Standard 1-layer LSTM: 88.0%; LSTM+BatchNorm: 95.4% | 96.0% accuracy with 1 layer, thickness 10 |
| Text classification (Subj) | BiLSTM: 90.5%; Tree-LSTM: 91.3% | 93.9% |
| PTB language modeling | AWD-LSTM: test perplexity 52.8; single LSTM: 78.4 | 54.7 for 1-layer Thick-Net; 50.2 for 2-layer Thick-Net |

The strongest quantitative contrast in the summary is on permuted sequential MNIST, where a 1-layer Thick-Net with thickness 10 achieves 96.0% accuracy, compared with 88.0% for a standard 1-layer LSTM and 95.4% for LSTM+BatchNorm [1911.08074]. For PTB language modeling, the 2-layer Thick-Net achieves test perplexity 50.2, outperforming AWD-LSTM at 52.8, while the 1-layer Thick-Net reaches 54.7 and the single LSTM baseline is reported at 78.4 [1911.08074].

The text-classification results reported in the summary show Thick-Net at 93.9% on Subj, compared with 90.5% for BiLSTM and 91.3% for Tree-LSTM [1911.08074]. On the adding problem with \(T=500\), the summary states that only Thick-Net converges rapidly, whereas the compared deeper or wider LSTM variants do not exhibit the same behavior [1911.08074].

## 5. Generalization, overfitting, and ablations

A central interpretive theme in the Thick-Net paper is that thick outputs can efficiently avoid overfitting. The mechanism offered for this claim is structural: only the maximum value among the parallel outputs is passed forward, so the model inherits a large dropout-like effect without using conventional stochastic dropout as the primary explanation [1911.08074].

The paper contrasts this with two familiar failure modes. Wide networks increase capacity but also the risk of overfitting and computation. Deep networks add stacked transformations, enabling hierarchical feature extraction but making optimization harder because of issues such as vanishing gradients and longer backpropagation paths. Thick-Net is presented as a way to alter this balance by adding parallel per-node alternatives without forwarding all of them [1911.08074].

The ablation evidence reinforces the importance of the maximization operator itself. Average or random selection among thick outputs degrades performance compared to maximization. The summary also reports that for long sequences or small data, such as the adding problem with \(T=500\), wider or deeper LSTMs overfit or cannot converge, while Thick-Net both converges and generalizes [1911.08074].

This suggests that the term “thick output” in this context denotes more than an increase in parameter count. Its technical content lies in a specific local competition among parallel transformations, combined with a deterministic max reduction that changes both training dynamics and generalization behavior [1911.08074].

## 6. Distinct meanings in adjacent literatures

The expression “thick outputs” is not uniform across fields. In tensor-variate Gaussian process regression, the phrase is used for high-dimensional outputs for each input, such as spatial-temporal arrays, matrices, or tensors. There the central object is not a per-node max selection mechanism but a tensor-variate Gaussian process with separable covariance
\[
\Sigma = \Sigma_1 \otimes \Sigma_2 \otimes \cdots \otimes \Sigma_m,
\]
designed for arbitrary dimensional outputs where a Kronecker product structure is appropriate [2502.10319].

Within that literature, outer product emulators and parallel partial emulators are treated as special cases of tensor-variate Gaussian process regression for matrix-valued outputs. The distinction there concerns regressor and covariance structures, especially whether output dependence is modeled explicitly and whether statistical strength is borrowed across output locations [2502.10319]. This is therefore a different use of “thick outputs” from the Thick-Net notion of multiple parallel values at each hidden state.

A second distinct use appears in MCMC methodology. The paper on Stein thinning contrasts “thick” output with thinning and states that using all MCMC samples, with no thinning, minimizes variance when storage and computation on all samples is possible. When strong compression is required, it proposes retrospective subset selection by greedy minimization of a kernel Stein discrepancy [2005.03952]. Here again, “thick output” refers to retaining the full sample path rather than to a network architecture.

A plausible implication is that “thick outputs” should be interpreted locally within a subfield. In sequential deep learning it denotes the Thick-Net mechanism of multiple parallel outputs followed by maximization [1911.08074]. In emulator methodology it denotes multi-dimensional outputs with structured covariance [2502.10319]. In MCMC it denotes the unthinned chain output [2005.03952].

Source: https://www.emergentmind.com/topics/thick-outputs