Anti-Shallowness in Neural & Language Models

• Anti-shallowness is a critique that shallow architectures and representations can be insufficient when deeper, structured modeling is required.
• It demonstrates that deep networks can achieve exponential efficiency in approximating compositional functions as opposed to shallow counterparts.
• The concept extends to language-model safety and adaptive quantum circuits, advocating for refined alignment and task-aware depth.

Anti-shallowness is a recurring research position according to which shallow architectures, shallow representations, or shallow forms of alignment are insufficient for certain tasks, although the insufficiency is usually conditional rather than absolute. In approximation theory it names efficiency gaps between deep and shallow networks; in alignment research it denotes safety behavior that must persist beyond the first few output tokens or beyond surface-level refusals; and in adaptive quantum circuits it becomes a formal state property measuring distance from constant-depth non-adaptive preparation through $-\log F(\psi)$ (Malach et al., 2019, Mhaskar et al., 2016, Qi et al., 2024, Liu et al., 21 Sep 2025).

1. Conceptual scope

Across the literature, anti-shallowness does not denote a single doctrine so much as a family of objections to solutions that are only locally, superficially, or low-depth adequate. In neural-network approximation, the target of criticism is the assumption that one hidden layer, or modest depth, is enough not merely for universal approximation in principle but for efficient approximation under realistic structural priors. In language-model safety, the target is alignment that changes the model’s behavior mainly at the start of a response while leaving harmful continuations latent. In representational analyses, the target is a system whose visible outputs can be steered at the surface without reflecting a correspondingly rich internal organization.

A common pattern is that anti-shallowness arguments are strongest when they are matched to a structural hypothesis. Depth helps for compositional functions, fractal constructions, or local features; safety depth matters when harmful continuations can already be underway; representational depth matters when latent features must support coherent behavior across interventions. This suggests that anti-shallowness is usually a claim about mismatch between a shallow mechanism and the structure of the problem, rather than a blanket endorsement of depth for its own sake.

For that reason, anti-shallowness results are often paired with explicit caveats. The literature repeatedly rejects unconditional slogans such as “deeper is always better.” Instead it argues that depth, richer latent structure, or nontrivial state complexity matter only under additional assumptions about compositionality, locality, optimization, or robustness to intervention (Mori et al., 2020, Yu et al., 2023, Kabir et al., 29 Aug 2025).

2. Approximation, expressivity, and efficiency

In classical approximation-theoretic form, anti-shallowness is the claim that deep models can realize certain structured functions with exponentially fewer parameters than shallow ones. A central result compares shallow one-hidden-layer networks with hierarchical binary-tree architectures for compositional functions. For generic smooth $d$-variate targets, shallow networks achieve the rate

$\mathsf{dist}(f,\mathcal S_n)=O(n^{-r/d}),$

whereas for hierarchical compositional functions matched to the architecture, deep networks achieve

$\mathsf{dist}(f,\mathcal D_n)=O(n^{-r/2}).$

Accordingly, reaching accuracy $\epsilon$ requires shallow complexity on the order of $O(\epsilon^{-d/r})$, while deep compositional approximation requires only $O(\epsilon^{-2/r})$ per constituent module. The point is not that depth is universally superior, but that it removes the ambient-dimension exponent when the target itself is compositional (Mhaskar et al., 2016).

A more explicit depth-separation construction uses fractal distributions generated by an iterated function system. For binary classification with positive support on the depth-$n$ fractal set $K_n$, the paper proves that there exists a ReLU network of width $5dr$ and depth $d$0 that separates $d$1 from $d$2 exactly. It also proves a shallow lower bound: if a depth-$d$3 network separates the same set, with $d$4, then its width must satisfy

$d$5

The constructive upper bound exploits self-similarity by repeatedly “reversing” one level of the iterated function system, while the lower bound uses linear-region counting. The same paper further introduces the approximation curve

$d$6

which links coarse-to-fine distributional structure to the extent to which shallower networks can still approximate well (Malach et al., 2019).

The same efficiency theme appears in combinatorial form in the travel-maze problem formulated as a product of permutation matrices. An unrestricted shallow Rosenblatt-style perceptron remains universal, but the explicit shallow construction requires $d$7 hidden units, whereas the deep construction that composes permutations stage by stage uses $d$8 neurons and $d$9 connections. Anti-shallowness here is not a denial of shallow expressivity in principle; it is a statement about frugality. The shallow solution enumerates global input configurations, while the deep solution reuses compact intermediate states (Kirdin et al., 2022).

A significant counterpoint is that every finite-depth ReLU network can be rewritten as an exactly equivalent depth-3 ReLU network if weights in the extended reals are allowed. The construction externalizes the network’s piecewise-linear partition into explicit region detectors and affine pieces, with width depending on the number of regions $\mathsf{dist}(f,\mathcal S_n)=O(n^{-r/d}),$0 and half-spaces $\mathsf{dist}(f,\mathcal S_n)=O(n^{-r/d}),$1, and crucially uses $\mathsf{dist}(f,\mathcal S_n)=O(n^{-r/d}),$2 as a hard gating coefficient. A plausible implication is that anti-shallowness results in ordinary deep learning are mainly about efficient representation with finite weights and tractable constructions, not about bare function expressivity in an enlarged mathematical model (Villani et al., 2023).

3. Optimization, generalization, and the limits of depth

A decisive refinement of anti-shallowness is the observation that expressive depth separation does not by itself imply trainability. In fractal classification, the approximation curve $\mathsf{dist}(f,\mathcal S_n)=O(n^{-r/d}),$3 measures whether negative mass is concentrated in coarse or fine detail. If $\mathsf{dist}(f,\mathcal S_n)=O(n^{-r/d}),$4 rises quickly, relatively shallow networks already approximate well, with

$\mathsf{dist}(f,\mathcal S_n)=O(n^{-r/d}),$5

But for Cantor distributions concentrated on the finest scales, the paper proves that gradient information at random initialization becomes tiny. In particular, for an intermediate depth $\mathsf{dist}(f,\mathcal S_n)=O(n^{-r/d}),$6, the population hinge-loss gradients satisfy

$\mathsf{dist}(f,\mathcal S_n)=O(n^{-r/d}),$7

while classification error remains large. In the extreme fine Cantor case, the population gradient is zero at initialization and the error is $\mathsf{dist}(f,\mathcal S_n)=O(n^{-r/d}),$8. This is the sense in which “deeper better only when shallow is good”: learnability by gradient methods appears to require at least partial shallow approximability (Malach et al., 2019).

A distinct theoretical route reaches a related conclusion through margin bounds. For depth-$\mathsf{dist}(f,\mathcal S_n)=O(n^{-r/d}),$9 networks with weight constraints, the Rademacher-average bound

$\mathsf{dist}(f,\mathcal D_n)=O(n^{-r/2}).$0

grows with depth, which is a negative effect on the generalization bound. At the same time, Betti-numbers-based representation power also grows with depth, which can reduce empirical margin error. The resulting thesis is a tradeoff: for a restricted number of hidden units, increasing depth is not always good, because depth helps through representational complexity but hurts through capacity. On that basis the paper proposes margin-penalized objectives,

$\mathsf{dist}(f,\mathcal D_n)=O(n^{-r/2}).$1

and

$\mathsf{dist}(f,\mathcal D_n)=O(n^{-r/2}).$2

as alternatives to merely increasing depth (Sun et al., 2015).

A different anti-shallow argument treats depth as an inductive bias against locally brittle decision surfaces. The proposed unsupervised measure is output sharpness,

$\mathsf{dist}(f,\mathcal D_n)=O(n^{-r/2}).$3

which reduces to the $\mathsf{dist}(f,\mathcal D_n)=O(n^{-r/2}).$4 weight norm in the linear case. The paper argues heuristically that deeper compositions attenuate input-output derivatives and therefore bias networks toward lower sharpness, which correlates strongly with better generalization. A task-structural qualification comes from synthetic local/global labels: deeper fully connected networks generalize better for $\mathsf{dist}(f,\mathcal D_n)=O(n^{-r/2}).$5-local labels, whereas shallower ones are better for $\mathsf{dist}(f,\mathcal D_n)=O(n^{-r/2}).$6-global labels, and the neural tangent kernel does not capture this dependence. Together these results support an optimization-aware and task-aware anti-shallowness rather than a universal one (Sa-Couto et al., 2022, Mori et al., 2020).

4. Adaptive depth, shortcut skepticism, and specialized shallowing

Anti-shallowness can also target the opposite error: treating maximal depth as uniformly beneficial even when earlier representations already suffice. Shallow-Deep Networks instrument standard CNNs with internal classifiers placed near 15%, 30%, 45%, 60%, 75%, and 90% of total inference cost. This makes overthinking measurable. The paper distinguishes wasteful overthinking, where a correct decision is available early but computation continues, from destructive overthinking, where a correct internal prediction becomes a final error. Across tasks, 95% of CIFAR-10 samples, 81% of CIFAR-100 samples, and 69% of Tiny ImageNet samples do not require full depth under an ideal stopping rule; confidence-based early exits reduce average inference cost by more than 50% while preserving accuracy, and destructive overthinking occurs for up to roughly 50% of all misclassifications on natural inputs. Anti-shallowness here becomes anti-unconditional-depth: depth is a resource to deploy selectively, not an unconditional good (Kaya et al., 2018).

A different variant appears in out-of-distribution generalization. The “too-good-to-be-true prior” treats easy solutions to hard tasks as suspicious because they are likely shortcuts. A deliberately weak low-capacity network is used as a shortcut detector, and its confidence defines importance weights

$\mathsf{dist}(f,\mathcal D_n)=O(n^{-r/2}).$7

which are normalized within each mini-batch before training a high-capacity network. The guiding idea is that examples a shallow model can master are more likely to be solvable by superficial cues, so they should be downweighted. This is anti-shallowness not as architectural depth-seeking, but as skepticism toward shallowly solvable evidence (Dagaev et al., 2021).

In natural-language representation learning, the relation can reverse again. Shallow syntax-aware ELMo variants, including mSynC and downstream BIOUL chunk features, add little or nothing over ELMo-transformer baselines on coarse NER, fine-grained NER, constituency parsing, and sentiment. The best reading is that explicit shallow chunk structure becomes largely redundant once deep contextual pretraining already encodes the relevant information. Here anti-shallowness concerns the redundancy of a shallow linguistic signal, not the insufficiency of a shallow network per se (Swayamdipta et al., 2019).

Conversely, deployment-oriented compression shows that shallowing can be rational when the task is intentionally narrow. In the “backyard dog” problem, a pre-trained VGG16 face model is truncated and combined with centralization, spherical projection, and Advanced Supervised PCA. A 5-layer network with 60 ASPCs processes a $\mathsf{dist}(f,\mathcal D_n)=O(n^{-r/2}).$8 image in under 1 second on one core of Raspberry Pi, albeit with worse $\mathsf{dist}(f,\mathcal D_n)=O(n^{-r/2}).$9 error than deeper truncations. This suggests that anti-shallowness claims are task-relative: a deep legacy model may remain necessary for universal feature learning even when a shallowed descendant is acceptable for constrained deployment (Gorban et al., 2018).

5. Alignment depth in LLMs

In language-model safety, anti-shallowness has acquired a specifically token-level meaning. “Shallow safety alignment” denotes the case where post-training primarily alters the generative distribution over only the first few output tokens, enough to elicit a refusal prefix, while leaving later harmful continuations comparatively intact. The paper gives the illustrative factorization

$\epsilon$0

and argues that alignment can become safe-looking simply by driving $\epsilon$1 toward zero. Empirically, Llama-2-7B-Chat begins with either “I cannot” or “I apologize” in 96.1% of harmful HEx-PHI cases, and Gemma-7B-1.1-IT generates “I am unable” in 96.7% of cases. The same work shows that prefilling refusal prefixes makes even base models look safe, while prefilling harmful continuations causes aligned models to resume harmful behavior. Its proposed repair is to train safety recovery from harmful prefixes, i.e. to learn refusal not only at $\epsilon$2 but also for $\epsilon$3 at later depths (Qi et al., 2024).

A more conceptual anti-shallowness critique argues that current alignment methods reinforce “shallow behavioral dispositions” rather than a “genuine capacity for normative deliberation.” The central failure mode is vulnerability to prompts that exploit normative conflicts among helpfulness, honesty, and harmlessness. A normative conflict arises when norms jointly demand that an agent $\epsilon$4 does $\epsilon$5 and that $\epsilon$6 does $\epsilon$7 in contexts where $\epsilon$8 cannot do both. According to this view, current aligned LLMs do not detect and resolve such conflicts in a principled way; they follow whichever learned disposition the prompt makes most salient. A less shallow alternative would require reliable conflict detection, contextual assessment of normative strength, principled resolution, and consistency between the outcome of reasoning and behavior (Millière, 5 Jun 2025).

A direct engineering response is Any-Depth Alignment (ADA). The paper defines generation depth $\epsilon$9 as the number of assistant tokens generated after the user prompt, with $O(\epsilon^{-d/r})$0 immediately after the prompt, and argues that alignment is concentrated in assistant header tokens because those tokens repeatedly anchor refusal training. ADA reintroduces those header tokens mid-stream. In ADA (RK), generation periodically forks, reuses the KV cache, injects assistant-header Safety Tokens, and checks whether the short lookahead produces a refusal. In ADA (LP), the same header injection is used only to extract hidden states from a designated probe token and classify harmfulness with a lightweight logistic probe. Across model families, ADA achieves near-100% refusal against adversarial prefill attacks ranging from dozens to thousands of tokens, reduces the average success rate of GCG, AutoDAN, PAIR, and TAP to below 3%, and does so with minimal over-refusal on benign tasks (Zhang et al., 20 Oct 2025).

6. Formal generalizations and broader variants

The anti-shallowness theme also extends to representation dynamics inside architectures. In classic post-norm Transformers, the depth problem is attributed to token similarity escalation. With

$O(\epsilon^{-d/r})$1

token similarity increases because each row-stochastic attention matrix preserves the all-ones eigenspace while typically exhibiting a large spectral gap. The paper’s proposed anti-shallow intervention is not to damp self-attention wholesale, but to remove the token-uniform component directly through

$O(\epsilon^{-d/r})$2

which becomes token-centering at $O(\epsilon^{-d/r})$3. In this setting, “shallow” means that deeper layers cease to preserve useful inter-token diversity even though they exist nominally (Yu et al., 2023).

In adaptive quantum circuits, anti-shallowness is made fully formal as a state property. For an $O(\epsilon^{-d/r})$4-qubit state $O(\epsilon^{-d/r})$5, the best fidelity to depth-$O(\epsilon^{-d/r})$6 non-adaptive circuits is

$O(\epsilon^{-d/r})$7

and for constant $O(\epsilon^{-d/r})$8 the anti-shallowness of $O(\epsilon^{-d/r})$9 is

$O(\epsilon^{-2/r})$0

This measures distance from the trivial phase. The same paper proves that shallow adaptive circuits with $O(\epsilon^{-2/r})$1 ancillas can realize at most $O(\epsilon^{-2/r})$2 anti-shallowness, and that the bound is tight via good QLDPC code states. It also distinguishes examples by scaling: good QLDPC states have anti-shallowness $O(\epsilon^{-2/r})$3, GHZ states $O(\epsilon^{-2/r})$4, and the symmetric hypergraph state $O(\epsilon^{-2/r})$5. Anti-shallowness here is neither a slogan nor a metaphor; it is a quantitative invariant for approximate state complexity and phase structure (Liu et al., 21 Sep 2025).

A broader representational analogue appears in work on ideological depth in LLMs. There the contrast is between shallow prompt-level ideological behavior and deep internal political structure. The paper studies steerability under prompting and activation intervention, then probes internal representations with sparse autoencoders. It reports that one model can contain 7.3x more political features than another model of similar size, and that targeted ablation of a core political feature in an ideologically “deep” model produces consistent, logical shifts in reasoning across related topics, whereas the same intervention in an ideologically “shallow” model increases refusal outputs. A plausible implication is that anti-shallowness in this setting concerns latent representational richness and robustness under intervention rather than mere response style (Kabir et al., 29 Aug 2025).

Taken together, these lines of work define anti-shallowness less as a universal preference for depth than as a recurrent demand for non-superficial structure. Sometimes that structure is compositional depth, sometimes optimization-compatible coarse structure, sometimes trajectory-wide safety, sometimes resistance to token collapse, and sometimes measurable distance from trivial states. The literature therefore supports a qualified conclusion: shallowness is problematic when it leaves decisive structure unrepresented, untrainable, or only superficially activated, but the remedy is domain-specific and often more subtle than simply adding layers.