Deep Linear Convolutional Networks

• Deep linear convolutional networks are convolutional architectures with only linear hidden layers that exploit Fourier diagonalization to modulate implicit regularization.
• They leverage circular convolution and parameter sharing to transform the optimization geometry, inducing structured frequency sparsity and efficient linear prediction.
• Their analysis via geometric and tensor factorizations reveals how depth and pooling strategies shape sparse representations and model inductive biases.

Searching arXiv for recent and foundational papers on deep linear convolutional networks to ground the article. Deep linear convolutional networks are convolutional architectures whose hidden layers are linear operators and whose output map is linear, so the network is linear in the input even though it is highly non-linear in its parameters. In the full-width circular model, an $L$-layer one-dimensional network with input $x\in\mathbb{R}^D$, $L-1$ convolutional layers, and a final linear readout defines a predictor $f_{\mathbf u}(x)=x^\top \mathbf w_{\mathbf u}$, where the depth and convolutional parameterization do not enlarge the function class beyond linear predictors but do alter the optimization geometry and the induced implicit regularization (Gunasekar et al., 2018). In finite-width Bayesian formulations, multi-channel one-dimensional convolutional hidden layers with periodic boundary conditions and a scalar linear readout lead to exact non-asymptotic mixture-of-Gaussians priors and posteriors over functions, with random kernel renormalization governed by Wishart matrices (Bassetti et al., 2024). Closely related analyses of convolutional arithmetic circuits and piecewise-linear convolutional networks show that convolution, pooling, and hyperplane geometry impose structured inductive biases on correlation modeling and on the shape of preimage manifolds (Cohen et al., 2016, Carlsson, 2019).

1. Architectural forms and linear operator structure

A canonical deep linear convolutional network is the one-dimensional, full-width, linear convolutional network of depth $L$ with a single scalar output. In that model, the input is $x\in\mathbb{R}^D$; the first $L-1$ layers each have exactly $D$ units and use a full-width circular filter $\mathbf u_l\in\mathbb{R}^D$; and the final layer is a fully connected linear map $\mathbf u_L\in\mathbb{R}^D$. The intermediate features satisfy $x\in\mathbb{R}^D$0 and, for $x\in\mathbb{R}^D$1,

$x\in\mathbb{R}^D$2

so the convolution is circular cross-correlation scaled by $x\in\mathbb{R}^D$3. The output is

$x\in\mathbb{R}^D$4

and the paper denotes the parameter-to-predictor map by $x\in\mathbb{R}^D$5 (Gunasekar et al., 2018).

This representation makes explicit a basic fact that is often obscured by the depth of the architecture: in the full-width linear case, the function class is the class of all linear predictors on $x\in\mathbb{R}^D$6, but the parameterization is not the standard direct parameterization by $x\in\mathbb{R}^D$7. That distinction is decisive for optimization and generalization because gradient descent operates in parameter space rather than predictor space (Gunasekar et al., 2018).

A second formulation studies finite-width convolutional deep linear networks with multi-channel one-dimensional inputs $x\in\mathbb{R}^D$8, $x\in\mathbb{R}^D$9, $L-1$0, periodic boundary conditions, unit stride, and the same spatial size $L-1$1 in all layers. Each hidden layer $L-1$2 has $L-1$3 channels and filter size $L-1$4. The pre-activations are

$L-1$5

and for $L-1$6,

$L-1$7

The output is a scalar global linear readout over channels and positions,

$L-1$8

Using the translation operators $L-1$9, each convolutional layer can be viewed as a structured linear operator, equivalently a block-circulant or Toeplitz-type matrix after flattening spatial and channel indices (Bassetti et al., 2024).

A related but distinct family is the convolutional arithmetic circuit, which uses linear point-wise activations and product pooling. There, the hidden layers consist of $f_{\mathbf u}(x)=x^\top \mathbf w_{\mathbf u}$0 convolutions across channels followed by spatial product pooling, and the overall output is multilinear in the representation functions $f_{\mathbf u}(x)=x^\top \mathbf w_{\mathbf u}$1. In that sense, convolutional arithmetic circuits are deep linear convolutional networks in the representation space, with multiplicative interactions introduced only by pooling (Cohen et al., 2016).

2. Fourier diagonalization and multiplicative parameterization

The full-width circular model is naturally analyzed in the Fourier domain. Let $f_{\mathbf u}(x)=x^\top \mathbf w_{\mathbf u}$2 be the unitary DFT matrix,

$f_{\mathbf u}(x)=x^\top \mathbf w_{\mathbf u}$3

and define $f_{\mathbf u}(x)=x^\top \mathbf w_{\mathbf u}$4. Circular convolution becomes pointwise multiplication: $f_{\mathbf u}(x)=x^\top \mathbf w_{\mathbf u}$5 Iterating through the $f_{\mathbf u}(x)=x^\top \mathbf w_{\mathbf u}$6 convolutional layers yields

$f_{\mathbf u}(x)=x^\top \mathbf w_{\mathbf u}$7

Equivalently,

$f_{\mathbf u}(x)=x^\top \mathbf w_{\mathbf u}$8

Thus the deep linear convolutional network is exactly a diagonal linear network in the Fourier domain, with the effective predictor generated by coordinate-wise multiplicative chains on the frequency coefficients (Gunasekar et al., 2018).

This Fourier factorization is the source of the model’s distinctive behavior. In predictor space the network is linear, but in Fourier coordinates its parameterization is a homogeneous polynomial map of degree $f_{\mathbf u}(x)=x^\top \mathbf w_{\mathbf u}$9. The multiplicative structure is coordinate-wise rather than fully dense, so the convolutional architecture changes the geometry relative to a deep linear fully connected network even when both architectures represent the same set of linear functions (Gunasekar et al., 2018).

For local finite-width convolutional networks, the corresponding linear operator viewpoint is expressed through translation operators and covariance transformations rather than an explicit Toeplitz-matrix calculus. The underlying principle is the same: convolutional weight sharing induces a structured linear map whose spectral or local-covariance representation is more informative than its raw parameter tensor (Bassetti et al., 2024).

The tensor formulation of convolutional arithmetic circuits gives an analogous algebraic picture. For each output $L$0,

$L$1

where the coefficient tensor $L$2 is induced by the convolution-and-pooling hierarchy. Deep networks realize a hierarchical tensor decomposition, while shallow networks with global product pooling realize a CP decomposition (Cohen et al., 2016). This suggests a broader algebraic theme: deep linear convolutional architectures are most naturally understood through structured factorizations rather than through the input-output map alone.

3. Optimization dynamics and implicit regularization

For linearly separable binary data $L$3 with $L$4, the full-width model is trained with exponential loss

$L$5

or equivalently with the parameterized loss

$L$6

using discrete-time gradient descent

$L$7

Under assumptions including linearly separable data, asymptotic loss minimization, convergence in direction of predictors and gradients, and coordinatewise phase convergence of the Fourier coefficients, gradient descent converges in direction to a separator characterized by a margin problem in the frequency domain rather than in the original coordinates (Gunasekar et al., 2018).

For depth $L$8, the limit predictor satisfies an $L$9-max-margin characterization in the Fourier domain: $x\in\mathbb{R}^D$0 For general depth $x\in\mathbb{R}^D$1, the limit direction is a positive scaling of a first-order stationary point of

$x\in\mathbb{R}^D$2

where $x\in\mathbb{R}^D$3 is the inverse DFT of $x\in\mathbb{R}^D$4. When $x\in\mathbb{R}^D$5, this is the convex $x\in\mathbb{R}^D$6 norm; when $x\in\mathbb{R}^D$7, $x\in\mathbb{R}^D$8, so the objective is a nonconvex bridge penalty or quasi-norm that is strongly sparsity-inducing (Gunasekar et al., 2018).

The regularizer induced by the convolutional parameterization is

$x\in\mathbb{R}^D$9

up to a monotone positive scalar factor. The frequency-domain origin of this regularizer follows from the coordinatewise factorization problem

$L-1$0

whose optimum equalizes magnitudes across layers: $L-1$1 Summing over frequencies gives the bridge-type penalty on $L-1$2 (Gunasekar et al., 2018).

This contrasts sharply with deep linear fully connected networks. Under analogous assumptions, fully connected depth-$L-1$3 linear networks converge to the classical $L-1$4 hard-margin SVM solution

$L-1$5

independently of depth (Gunasekar et al., 2018). A common misconception is therefore excluded by the analysis: the fact that full-width linear convolutional and fully connected networks realize the same class of linear predictors does not imply the same implicit bias under gradient descent. In the convolutional case, depth acts as a frequency-sparsity dial because the exponent $L-1$6 decreases with $L-1$7, and for $L-1$8 the limiting statement is about first-order stationary points rather than guaranteed global minimizers (Gunasekar et al., 2018).

4. Finite-width Bayesian theory and kernel renormalization

A complementary theory studies deep linear convolutional networks in the Bayesian setting with iid Gaussian weights. In the convolutional case, the hidden-layer kernels satisfy

$L-1$9

and the observation model uses Gaussian likelihood, corresponding to squared error loss (Bassetti et al., 2024).

The principal non-asymptotic result is that the prior over outputs is not Gaussian but an exact mixture of Gaussians. The mixing variables are independent Wishart matrices

$D$0

assembled into the product measure

$D$1

The convolutional architecture enters through the local kernel renormalization map $D$2, defined recursively by

$D$3

and for $D$4,

$D$5

where $D$6, with $D$7 (Bassetti et al., 2024).

Conditioned on $D$8, the outputs are Gaussian with covariance

$D$9

The prior characteristic function is therefore an integral of Gaussian characteristic functions against the Wishart product measure. Under Gaussian likelihood, the posterior predictive remains a mixture of Gaussians, with data-dependent reweighting

$\mathbf u_l\in\mathbb{R}^D$0

where

$\mathbf u_l\in\mathbb{R}^D$1

The predictive density is a mixture of one-dimensional Gaussian regression models whose kernel is determined by the effective local covariance $\mathbf u_l\in\mathbb{R}^D$2 (Bassetti et al., 2024).

This formulation separates two regimes. In the lazy infinite-width limit, the Wishart factors concentrate at identity, the mixture collapses to a single Gaussian process, and there is no feature learning. In the feature-learning infinite-width regime, obtained through a mean-field scaling of the likelihood, the posterior mixing measure satisfies a large deviation principle with data-dependent rate function

$\mathbf u_l\in\mathbb{R}^D$3

so the effective kernel shape becomes data-dependent even at infinite width (Bassetti et al., 2024). In the terminology adopted there, convolutional depth produces local kernel renormalization, while multiple outputs in fully connected models produce kernel shape renormalization.

5. Hyperplane geometry, cones, and preimages

A geometric account of deep convolutional networks, including linear and piecewise-linear cases, is formulated in terms of hyperplanes, dual bases, and polyhedral cones. For a ReLU layer

$\mathbf u_l\in\mathbb{R}^D$4

the affine functions $\mathbf u_l\in\mathbb{R}^D$5 define hyperplanes

$\mathbf u_l\in\mathbb{R}^D$6

which partition the positive orthant into activation cells. For a specified output $\mathbf u_l\in\mathbb{R}^D$7, with active indices $\mathbf u_l\in\mathbb{R}^D$8 and zero indices $\mathbf u_l\in\mathbb{R}^D$9, the preimage is

$\mathbf u_L\in\mathbb{R}^D$0

where

$\mathbf u_L\in\mathbb{R}^D$1

Using the dual basis $\mathbf u_L\in\mathbb{R}^D$2 induced by the hyperplane arrangement, the preimage becomes

$\mathbf u_L\in\mathbb{R}^D$3

so within a fixed activation pattern the preimage is a translated polyhedral cone (Carlsson, 2019).

For convolutional layers approximated by circulant matrices with shared bias, the hyperplanes inherit cyclic symmetry. If each row has common sum $\mathbf u_L\in\mathbb{R}^D$4, then all hyperplanes intersect at

$\mathbf u_L\in\mathbb{R}^D$5

which lies on the identity line $\mathbf u_L\in\mathbb{R}^D$6. With the cyclic shift permutation matrix $\mathbf u_L\in\mathbb{R}^D$7, the relation

$\mathbf u_L\in\mathbb{R}^D$8

shows that the family of hyperplanes forms a regular multidimensional polyhedral cone around the identity line, with apex controlled by $\mathbf u_L\in\mathbb{R}^D$9 and $x\in\mathbb{R}^D$00 and with cyclic rotational symmetry in the coordinates (Carlsson, 2019).

Across layers, these cones can be nested. Complete nesting inside the coordinate cone implies that cells and their faces are mapped into intersection subspaces of non-increasing dimension, yielding contraction. In two and three dimensions the examples show that nested cones contract volumes toward lower-dimensional manifolds, whereas wider or insufficiently nested cones can create regions where the map becomes effectively linear and no further shaping occurs (Carlsson, 2019). For purely linear convolutional networks without ReLU, the network has a single global linear map and the preimage of a given output is a single affine subspace; with ReLU, the global map becomes continuous piecewise-linear and the union of corresponding preimage cones across activation patterns gives the full preimage manifold (Carlsson, 2019).

The explicit three-layer example with circulant transformations illustrates a forward “flattening” interpretation: a highly nonlinear, continuous, piecewise-affine manifold in input space is mapped to a simple affine triangle in the final feature space. This suggests a geometric mechanism by which convolutional stacks can contract and flatten class manifolds before linear readout, although that statement is made in the cited work for deep convolutional networks with ReLU rather than for purely linear models alone (Carlsson, 2019).

6. Tensor expressivity, pooling geometry, and inductive bias

The tensor analysis of convolutional arithmetic circuits isolates how depth and pooling determine correlation structure. For a partition $x\in\mathbb{R}^D$01 of the input indices, the separation rank of a function $x\in\mathbb{R}^D$02 is

$x\in\mathbb{R}^D$03

with $x\in\mathbb{R}^D$04 corresponding to separability. For convolutional arithmetic circuits, if the representation functions $x\in\mathbb{R}^D$05 are linearly independent and square-integrable, then the separation rank equals the rank of the matricized coefficient tensor: $x\in\mathbb{R}^D$06 This converts correlation analysis into matrix-rank analysis of a tensor decomposition (Cohen et al., 2016).

The shallow architecture with global product pooling yields a CP decomposition,

$x\in\mathbb{R}^D$07

so for every partition,

$x\in\mathbb{R}^D$08

By contrast, the deep architecture with pooling window size $x\in\mathbb{R}^D$09 yields a hierarchical tensor decomposition. Its matricization rank depends strongly on how the partition aligns with the pooling hierarchy: the lower bound is exponential in the number of first-layer size-$x\in\mathbb{R}^D$10 groups split by the partition, while the upper bound decreases when larger hierarchical blocks are unbalanced between $x\in\mathbb{R}^D$11 and $x\in\mathbb{R}^D$12 (Cohen et al., 2016).

Two examples summarize the phenomenon. For the interleaved partition

$x\in\mathbb{R}^D$13

every size-$x\in\mathbb{R}^D$14 block is split, so a polynomially sized deep network can realize exponentially large separation rank. For the coarse partition

$x\in\mathbb{R}^D$15

the highest-level blocks are entirely on one side or the other, and the rank is bounded by $x\in\mathbb{R}^D$16, linear in the size of the top hidden layer (Cohen et al., 2016). The benefit of depth is therefore not uniform over all partitions; it is concentrated on those partitions favored by the pooling geometry.

Pooling geometry is the mechanism that selects favored partitions. Contiguous square pooling windows favor interleaved partitions over coarse ones, which the paper interprets as an inductive bias toward the statistics of natural images. Mirror or symmetric pooling schemes favor different long-range partitions, and empirical comparisons show that square pooling performs better on a local “closedness” task, whereas mirror pooling performs better on a “symmetry” task (Cohen et al., 2016). A plausible implication is that, even when the input-output map is linear or multilinear in the relevant representation space, the architectural grouping induced by convolution and pooling governs which correlations can be modeled efficiently.

Taken together, these results establish a recurring theme across deep linear convolutional models and their close variants. The models are often functionally simple at the level of input-output linearity, but the convolutional parameterization, the spectral factorization, the Bayesian kernel renormalization structure, the polyhedral geometry of hyperplanes, and the tensor organization induced by pooling each produce non-trivial inductive bias. In different formalisms, depth changes not the set of realizable linear predictors alone, but the geometry according to which those predictors are selected, regularized, and represented (Gunasekar et al., 2018, Bassetti et al., 2024, Carlsson, 2019, Cohen et al., 2016).