---
title: Depth-2 Linear Circuits
url: https://www.emergentmind.com/topics/depth-2-linear-circuits
type: topic
---

# Depth-2 Linear Circuits

Depth-2 linear circuits compute a linear transform \(Mv\) using one layer of intermediate linear combinations between the input and output layers. For an \(N\times N\) matrix \(M\), this is equivalently a factorization \(M=UV^T\) with \(U,V\in\mathbb{F}^{N\times G}\), or a decomposition \(M=\sum_{i=1}^G U_iV_i^T\) into rank-1 terms. The primary complexity measure is size, usually \(\nnz(U)+\nnz(V)\), and refined analyses also track degree, i.e. the maximal row or column degree on the two outer layers. Recent work has concentrated on Kronecker-power matrices such as Walsh–Hadamard and disjointness, where the classical \(O(N^{1.5})\) construction has been improved through increasingly precise rebalancing frameworks and asymptotic characterizations [2211.05217; 2509.14489].

## 1. Formal model and complexity measures

A depth-2 linear circuit for an \(N\times N\) matrix \(M\) can be viewed as a factorization
\[
M = U V^T
\]
for \(N\times G\) matrices \(U,V\), where \(G\) is the number of middle-layer gates. In this representation, the circuit computes \(Mv\) by first forming the \(G\) intermediate linear combinations \(V^Tv\), then applying \(U\) [2509.14489]. The same model is equivalently expressed as
\[
M=\sum_{i=1}^G U_iV_i^T,
\]
with each summand a rank-1 rectangle [2509.14489].

The standard size measure is
\[
\Size(C)=\nnz(U)+\nnz(V),
\]
or, in the synchronous factorization notation used for Kronecker powers, \(\operatorname{nnz}(M_1)+\operatorname{nnz}(M_2)\) when \(M=M_2M_1\) [2509.14489; 2211.05217]. This makes the model particularly well suited to sparse factorizations and tensor-product constructions. In the disjointness-matrix literature, the same depth-2 viewpoint is described as a decomposition of the target matrix into rank-1 matrices, with complexity measured either by size or by degree [2603.15565].

Degree is a second parameter that becomes central when the circuit is interpreted algorithmically or when the coefficient domain allows cancellations. One formulation defines row and column degrees by the support sizes of the corresponding columns of \(U\) and \(V\), with
\[
\Degree(C)=\max\{r_C(i),c_C(j)\},
\]
and asymptotically
\[
\delta(M)=\lim_{n\to\infty}\Degree(M^{\otimes n})^{1/n}
\]
for Kronecker powers [2509.14489]. In the disjointness setting, size bounds are studied over \(\{0,1\}\), while degree bounds are studied over \(\{0,\pm1\}\), because the latter can exploit sign cancellations and transposition symmetry unavailable in the nonnegative setting [2603.15565].

A recurring source of confusion is the phrase “depth-2 linear circuits.” In this literature it refers to linear-transform circuits for matrices, not to depth-two linear threshold circuits such as \(THR\circ THR\). The two areas interact conceptually through low-depth complexity, but they are distinct models with different gate semantics, measures, and techniques [1401.2444; 1805.10698].

## 2. Kronecker powers and the classical \(O(N^{1.5})\) baseline

The dominant family of examples is formed by Kronecker powers of a fixed base matrix \(A\),
\[
A^{\otimes k},
\]
with input length \(N=n^k\) when \(A\) is \(n\times n\). Important special cases include the Walsh–Hadamard transform
\[
H_k := H_1^{\otimes k},\qquad
H_1=\begin{bmatrix}1&1\\1&-1\end{bmatrix},
\]
and the disjointness matrix
\[
R_k := R_1^{\otimes k},\qquad
R_1=\begin{bmatrix}1&1\\1&0\end{bmatrix}
\]
[2509.14489]. The same disjointness family is denoted \(D^{\otimes N}\) in later work on refined upper bounds [2603.15565].

The classical depth-2 construction is based on the mixed product property
\[
(A\otimes B)(C\otimes D)=(AC)\otimes(BD).
\]
By splitting a Kronecker power into two halves, one obtains a factorization whose two factors have sparsities that multiply to \(N^{1.5}\). In the Walsh–Hadamard case,
\[
H_n=(H_{n/2}\otimes I)(I\otimes H_{n/2}),
\]
and since \(\operatorname{nnz}(H_{n/2}\otimes I)=N\cdot \sqrt N\), this yields a depth-2 circuit of size \(O(N^{1.5})\) [2211.05217]. The same standard argument works for any fixed base matrix and is also described as a generalization of the Fast Walsh-Hadamard transform or Yates’ algorithm [2211.05217; 2509.14489].

For many years this \(O(N^{1.5})\) upper bound was the universal benchmark. One early development was that the disjointness matrix admits better constructions than the generic baseline: the classical mixed-product route gives about \(O(N^{1.293})\), while Jukna and Sergeev improved this to \(O(N^{1.272})\) [2211.05217]. Later work generalized this phenomenon and showed that nontrivial savings over \(N^{1.5}\) are available for every fixed base size \(q\), not only for \(2\times2\) matrices [2211.05217].

This baseline is important because the subsequent literature is largely an attempt to answer two questions. First, how far below \(N^{1.5}\) can one push depth-2 size for concrete Kronecker-power transforms? Second, which balancing obstructions are genuinely inherent to all rebalancing-based constructions, and which merely reflect the limitations of a particular decomposition scheme? The modern answers to these questions are formulated through rebalancing, \(\alpha\)-weighted size, and asymptotic spectral duality [2509.14489].

## 3. Rebalancing, imbalanced decompositions, and asymptotic spectrum

A major step beyond the baseline was the observation that a nontrivial decomposition of a base matrix \(M\) can be lifted to improved depth-2 circuits for \(M^{\otimes n}\). If
\[
M=\sum_{j=1}^J U_jV_j^T,
\]
then
\[
M^{\otimes n}
=
\sum_{x\in[J]^n}
\left(\bigotimes_{k=1}^n U_{x[k]}\right)
\left(\bigotimes_{k=1}^n V_{x[k]}\right)^T,
\]
so the problem becomes one of controlling the imbalance between the two sides of each tensor term [2211.05217]. The 2022 construction introduced multiple update rules—soft balancing via the nontrivial decomposition and hard balancing via the trivial decompositions \(M=I\cdot M\) and \(M=M\cdot I\)—and analyzed the resulting process as a random walk on log-sparsities [2211.05217].

The key quantity in that framework is the imbalance
\[
\gamma(A,B)=\ln\!\left(\frac{\operatorname{nnz}(A)}{\operatorname{nnz}(B)}\right).
\]
The process tracks
\[
S_k=(\log\operatorname{nnz}(A_k),\log\operatorname{nnz}(B_k))
\]
and shows, under an explicit “imbalanced decomposition” condition, that the probability of falling into the hard-balancing stage decays exponentially. This yields the main theorem: if \(M\) has such an imbalanced decomposition, then for every \(n\),
\[
M^{\otimes n}
\]
has a depth-2 linear circuit of size \(\exp(\alpha_1)^{\,n+o(n)}\), where \(\alpha_1\) is defined from the geometric means \(\sqrt{\operatorname{nnz}(U_j)\operatorname{nnz}(V_j)}\) [2211.05217].

The later asymptotic-spectrum perspective recast these balancing phenomena in a more intrinsic form. For a matrix \(M\),
\[
\Size(M)=\min_{UV^T=M}\nnz(U)+\nnz(V),
\qquad
\sigma(M)=\lim_{n\to\infty}\Size(M^{\otimes n})^{1/n},
\]
and, for \(\alpha\in[0,1]\),
\[
\Size_\alpha(M)=\inf \sum_i \nnz(U_i)^\alpha \nnz(V_i)^{1-\alpha},
\qquad
\sigma_\alpha(M)=\lim_{n\to\infty}\Size_\alpha(M^{\otimes n})^{1/n}.
\]
The central duality theorem states
\[
\sigma(M)=\sup_{\alpha\in[0,1]}\sigma_\alpha(M),
\]
showing that the optimal asymptotic depth-2 size is exactly governed by the best \(\alpha\)-weighted relaxation [2509.14489].

This formulation identifies the correct obstruction quantity for rebalancing. Given a decomposition family indexed by \(t\),
\[
\rho_t(\alpha)=\sum_{i\in I_t}\nnz(U_i)^\alpha \nnz(V_i)^{1-\alpha},
\qquad
C(\alpha)=\min_t \rho_t(\alpha).
\]
The theory proves a weak duality lower bound
\[
F_n(\lambda)\ge C(\alpha)^n\lambda^\alpha,
\]
and then a strong duality upper bound
\[
F_n(\lambda)\le 2^{o(n)}\cdot \sup_{\alpha\in[0,1]} C(\alpha)^n\lambda^\alpha.
\]
The paper explicitly interprets this as showing that the obstructions discovered by rebalancing are complete up to subexponential factors; in hindsight, earlier rebalancing arguments were proving special cases of Strassen’s duality theorem in an appropriate semiring [2509.14489].

For the disjointness matrix, this paradigm was sharpened further in 2026. The earlier “wild” rebalancing process was replaced by a tame, discretized process on a finite lattice \(\mathcal H_H\), with recursion
\[
C(n,h):=\sum_{(U_{ij},V_{ij})\in C_i}(U_{ij},V_{ij})\otimes C\bigl(n-K,\clamp_H(h+D_{ij})\bigr).
\]
The asymptotic geometric-average degree growth is then governed by a quenched top Lyapunov exponent of a random matrix product, and the size exponent is certified through the convex-optimization upper bound of Gharavi and Anantharam [2603.15565]. This is a conceptual shift from heuristic balancing to a finite-state dynamical system with rigorous spectral certification.

## 4. Quantitative upper bounds for canonical matrices

The modern literature provides a sequence of explicit improvements for depth-2 circuits computing Kronecker-power transforms. The results are best viewed as a hierarchy: a universal \(O(N^{1.5})\) baseline, general savings for all fixed base dimensions, sharper special-purpose constructions for Walsh–Hadamard and disjointness, and then highly optimized disjointness-specific bounds in both size and degree models.

| Object | Best stated bound in the data | Source |
|---|---|---|
| Any \(q\times q\) Kronecker power | \(O(N^{1.5-a_q})\), \(a_q=\Omega\!\left(\frac{1}{q^2\log q}\right)\) | [2211.05217] |
| Any \(2\times2\) Kronecker power | \(O(N^{1.446})\) | [2211.05217] |
| Walsh–Hadamard transform | \(O(N^{1.443})\) | [2211.05217] |
| Disjointness matrix | \(O(N^{1.258})\) | [2211.05217] |
| Disjointness matrix over any field | \(O(N^{1.249424})\) | [2509.14489] |
| \(D^{\otimes N}\) over \(\{0,1\}\) | \(O\!\left(2^{1.24485N}\right)\) | [2603.15565] |
| \(D^{\otimes N}\) degree over \(\{0,\pm1\}\) | \(O\!\left(2^{0.3199N}\right)\) | [2603.15565] |

The 2022 results established that every fixed base size \(q\) admits a nontrivial power saving over \(N^{1.5}\), with \(a_q=\Omega\!\left(\frac{1}{q^2\log q}\right)\) [2211.05217]. For \(q=2\), the paper gave \(O(N^{1.446})\) for arbitrary \(2\times2\) Kronecker powers, \(O(N^{1.443})\) for Walsh–Hadamard, and \(O(N^{1.258})\) for disjointness. The disjointness improvement used a decomposition of \(R_3\) that was “imbalanced but not one-sided,” precisely the regime where the new theorem was needed [2211.05217].

The 2025 asymptotic-spectrum framework pushed disjointness below the \(1.25\) exponent barrier. The paper proves that, over any field, the \(N\times N\) disjointness matrix has a depth-2 linear circuit of size \(O(N^{1.249424})\), and explicitly stresses that this is the first construction which surpasses exponent \(1.25\) [2509.14489]. The improvement comes from an enhanced partition/decomposition of \(R_{18}\), combining a generalized “take one row or column at a time” construction with a more powerful merging construction and constant-weight binary codes [2509.14489].

The 2026 refinement improves both the size and degree exponents for \(D^{\otimes N}\). Over \(\{0,1\}\), it achieves
\[
O\!\left(2^{1.24485N}\right),
\]
improving the previous bound \(O\!\left(2^{1.249424N}\right)\). Over \(\{0,\pm1\}\), it gives degree
\[
O\!\left(2^{0.3199N}\right),
\]
improving the previous \(O\!\left(2^{N/3}\right)\) [2603.15565]. The degree proof is region-sensitive: it analyzes a cost landscape on the \((p,q)\)-plane and shows that different families—Sonetto, Regulus, and Regulus\(^T\)—dominate in different regions [2603.15565].

These results collectively show that depth-2 linear circuit upper bounds are now driven by fine-grained asymptotic optimization rather than by a single universal construction. A plausible implication is that the field has moved from discovering isolated decompositions to understanding a structured design space, where entropy terms, weighted degrees, and tensor asymptotics all enter the objective explicitly.

## 5. Lower bounds, barriers, and conditional hardness

Despite substantial progress on upper bounds, robust unconditional lower bounds for depth-2 linear circuits remain limited, and the strongest current barriers are conditional. One major line of evidence comes from reductions to fine-grained complexity. For the Walsh–Hadamard matrix \(H_d\) of side-length \(N\), the Orthogonal Vectors Conjecture or the Strong Exponential Time Hypothesis each imply that there is no depth-2 linear circuit of size \(N^{1+o(1)}\) [2509.14489]. A similar result holds for the disjointness matrix \(R_d\), with a restriction that the \(R_d\) circuit is somewhat balanced [2509.14489].

The same paper also argues that proving an \(N^{1+\Omega(1)}\) lower bound for these matrices would have consequences far beyond transform complexity. In particular, if \(R_d\) of side-length \(N\) does not have a depth-2 linear circuit of size \(N^{1+o(1)}\), then Boolean Inner Product on \(n\)-bit vectors does not have \(2^{o(n)}\)-size \(\mathsf{SUM}\circ\mathsf{ETHR}\) circuits, and analogous implications are given for Walsh–Hadamard and Boolean Inner Product mod \(2\) [2509.14489]. This places depth-2 linear-circuit lower bounds close to major open frontiers in threshold-circuit complexity.

A different but related barrier appears in the wire-count model for encoding good codes. There, \(S_d(n)\) denotes the minimum number of wires of a depth-\(d\) unbounded-fan-in linear circuit encoding a code \(C:\{0,1\}^n\to\{0,1\}^{32n}\) with distance at least \(4n\). The main theorem gives
\[
S_d(n)=O(\lambda_d(n)\cdot n)
\quad\text{for all } d\ge 3,
\]
and
\[
S_{\alpha(n)}(n)=O(n),
\]
while any linear-size encoder must have depth at least
\[
\alpha(n)-2
\]
for sufficiently large \(n\) [2402.00378]. For depth \(2\), the paper gives the specific bound
\[
S_2\left(n,\frac{n}{r},n\right)=O(n\log^2 r),
\]
which shows that depth \(2\) is substantially larger than linear in the parameter regimes of interest [2402.00378]. This is a different model from matrix-factorization depth-2 circuits, but it illustrates a recurring phenomenon: depth \(2\) is often too rigid to support simultaneously strong combinatorial properties and linear size.

Taken together, these barriers indicate that upper-bound improvements do not yet amount to a settled structural theory. The existing evidence suggests that unconditional \(N^{1+\Omega(1)}\) lower bounds for depth-2 linear transforms are difficult precisely because they would interact with long-standing open problems in fine-grained complexity and threshold-circuit lower bounds [2509.14489].

## 6. Relation to neighboring low-depth models

Depth-2 linear circuits are part of a broader low-depth landscape, but several adjacent models should be kept separate. The most common source of ambiguity is the threshold model \(THR\circ THR\), where each gate computes a linear threshold function
\[
\sum_i w_i x_i \ge t.
\]
This is a Boolean circuit model rather than a linear-transform factorization model. Nontrivial lower bounds for depth-two linear threshold circuits remain open, and one paper presents an all-input evaluation algorithm for depth-two linear threshold circuits of size \(2^{n/24}\) with exponential weights running in \(2^n\cdot \mathrm{poly}(n)\) time, explicitly describing this as evidence that non-uniform lower bounds for \(THR\circ THR\) are within reach [1401.2444]. A complementary Williams-style study shows that shaving polylogarithmic factors from algorithms for problems such as Hopcroft, \(\ell_2\)-furthest pair, bichromatic \(\ell_2\)-closest pair, or Max-IP would imply that \(NEXP\) has no polynomial-size \(THR\circ THR\) circuits [1805.10698].

Another nearby Boolean regime is \(\{\mathsf{SYM},\mathsf{THR}\}\circ \mathsf{AND}\), which consists of a top symmetric or threshold gate fed by unbounded-fan-in AND gates. For this class, improved correlation bounds lead to a PRG with seed length
\[
2^{O(\sqrt{\log S})}+\polylog(1/\varepsilon),
\]
improving the earlier \(2^{O(\sqrt{\log(S/\varepsilon)})}\) dependence [1803.04553]. These are depth-two circuits, but not depth-2 linear circuits in the transform sense.

Conversely, some recent papers on “linear-size” circuits are only indirectly relevant. Fast adder constructions with depth
\[
\log_2 n+o(\log_2 n),
\qquad
O(n),
\qquad
\text{fan-out }2
\]
concern binary adder circuits, not general depth-2 linear transforms [1503.08659]. Likewise, faster linear-size And-Or path and adder circuits achieve logarithmic depth with iterated-log additive terms, and the paper explicitly states that it is not about depth-2 linear circuits in the usual sense [2405.12765].

These distinctions matter because the methodological overlap can be genuine while the models remain different. Rebalancing, balancing obstructions, fan-out constraints, wire complexity, and Williams-style algorithmic barriers all recur across the literature, but they apply to different notions of “linearity.” In the strict sense used for matrices and Kronecker powers, depth-2 linear circuits are best understood as sparse factorizations of linear transforms, with the current frontier centered on asymptotic size, degree, and the completeness of rebalancing-based design principles [2509.14489; 2603.15565].

Source: https://www.emergentmind.com/topics/depth-2-linear-circuits