Opaque Serial Depth in Neural Architectures

• Opaque serial depth is the measure of the maximum unseen serial computation between interpretable states in models, revealing hidden reasoning capacity.
• It is formalized using circuit complexity to provide rigorous upper bounds that inform analyses of Transformers, RNNs, and Mixture-of-Experts architectures.
• Automated methods compute opaque serial depth to guide model design and enforce externalization of internal computations via chain-of-thought protocols.

Opaque serial depth quantifies the maximum length of hidden, non-interpretable serial computation achievable within a model between externally observable, interpretable states, such as language tokens or exposed memory states. The concept serves as a theoretical and practical measure of a system’s capacity for internal serial reasoning that is not directly externalized, fundamentally constraining the capacity of architectures—particularly LLMs—to hide complex computation from monitoring mechanisms such as chain-of-thought (CoT) supervision. Opaque serial depth is formalized using circuit complexity, yielding rigorous, computable upper bounds on the hidden reasoning depth available to neural architectures. The metric has motivated both new architectural analyses (e.g., of Transformers, Mixture-of-Experts, recurrent neural networks) and theoretical advances in understanding the necessity and sufficiency of explicit reasoning traces for inherently serial problems (Brown-Cohen et al., 10 Mar 2026, Liu et al., 16 Jul 2025, Li et al., 2024).

1. Formal Definition and Complexity-Theoretic Foundations

Opaque serial depth is defined formally in terms of Boolean circuit depth and the associated computational graph of a neural network. Consider a function $f_\theta: \mathbb{R}^n \to \mathbb{R}^m$ implemented by a neural network with parameter count $S$. One expresses $f_\theta$ as a real-arithmetic circuit $C$ of size polynomial in $S$, built from specified basic gates (binary or unary, e.g., addition, ReLU, softmax).

• The (serial) circuit depth $\mathrm{Depth}(C)$ is the length of the longest directed path from input to output in $C$.
• The serial depth of $f_\theta$ is the minimal such depth attainable across all circuits of polynomial size computing $f_\theta$:

$$\mathrm{Depth}(f_\theta) = \min_{\substack{C \text{ s.t. } C \sim f_\theta\ |C| = O(S^k)}} \max_{\text{paths } P\subset C} |P|.$$

To capture interpretability constraints, certain nodes (e.g., token embeddings, output logits, intermediate sampled tokens) are annotated as interpretable. The model’s execution graph can be segmented at these nodes; the opaque serial depth is the maximal serial (circuit) depth computed between successive interpretable states:

$S$0

where $S$1 maps observed states to node $S$2. This metric rigorously captures how much internal, hidden computation may occur within one “observable” step, before any chance for external monitoring (Brown-Cohen et al., 10 Mar 2026).

From a complexity-theoretic perspective, the notion of opaque serial depth closely aligns with the distinction between classes solvable in “shallow” (polylogarithmic-depth) parallel circuits (Nick’s Class $S$3, or threshold classes $S$4) and those requiring deeper, inherently sequential circuits ($S$5) (Liu et al., 16 Jul 2025, Li et al., 2024).

2. Upper Bounds for Transformer and Related Architectures

For modern autoregressive transformers, the forward computation between interpretable states (tokens) traverses a sequence of nonlinear modules (embedding lookup, multihead self-attention, MLP, residuals). Each module’s contribution to circuit depth is:

• Embedding lookup: $S$6 (for vocabulary size $S$7)
• Each multihead attention layer: $S$8 (hidden size $S$9, sequence length $f_\theta$0)
• Each MLP: $f_\theta$1(hidden)$f_\theta$2

The maximal non-interpretable path is $f_\theta$3-times this per-layer cost, where $f_\theta$4 is the number of layers. Explicit upper-bound formulations for the Gemma 3 family are: $f_\theta$5 At maximum sequence length, these yield opaque serial depths of 4,490 (1B), 6,206 (4B), 8,754 (12B), and 11,662 (27B) (Brown-Cohen et al., 10 Mar 2026).

For recurrent neural networks, the serial depth grows with both the number of layers and sequence length: $f_\theta$6. Architectures with persistent black-box memory may achieve unbounded opaque serial depth as hidden states can be passed arbitrarily across time without exposure (Brown-Cohen et al., 10 Mar 2026, Liu et al., 16 Jul 2025).

3. Automated Calculation Methods

Quantifying opaque serial depth analytically for arbitrary architectures is tedious, motivating algorithmic approaches. A general method leverages computational graphs (e.g., JAX’s jaxpr) to traverse nodes and aggregate gate-based depth, segmenting at interpretable points:

• $f_\theta$7
• $f_\theta$8 if the node is interpretable; otherwise, accumulate $f_\theta$9.

The implementation covers $C$075 JAX primitives and scales to large models, yielding upper bounds close to analytic calculations (e.g., automatic bound of 11,268 for Gemma 3 12B vs. by-hand 8,754) (Brown-Cohen et al., 10 Mar 2026).

4. Architectural and Empirical Implications

Opaque serial depth constrains a model’s ability to perform unobservable long-range reasoning. This is particularly evident in empirical analyses:

• Mixture-of-Experts (MoE) architectures demonstrate significantly reduced opaque serial depth compared to dense architectures. For example, a 28-layer MoE Transformer with 11B active/91B total parameters yields depth 4,096—well below the dense Gemma 3 12B at 8,754–11,268 (Brown-Cohen et al., 10 Mar 2026).
• For standard LLMs, tasks demanding serial computation exceeding the opaque serial depth will necessarily pass through interpretable, monitorable states—forcing externalization via mechanisms such as CoT.

In summary, architectures featuring low opaque serial depth, such as dense Transformers with moderate cross-attention span or MoEs, ensure that almost all meaningful serial reasoning must be externalized. Conversely, architectures incorporating recurrence or unconstrained memory can conceal arbitrarily deep reasoning between interpretable bottlenecks (Brown-Cohen et al., 10 Mar 2026, Li et al., 2024).

5. Theoretical Link to Inherently Serial Problems

Opaque serial depth operationalizes key complexity-theoretic distinctions:

• Shallow models (constant depth, large width, e.g., standard transformers) capture only classes $C$1 or $C$2 in the absence of explicit serial unrolling.
• Many reasoning, simulation, and decision problems are $C$3-complete and fundamentally require $C$4 or at least $C$5 sequential steps; such problems have high inherent serial depth and lie outside $C$6 (Liu et al., 16 Jul 2025).

The necessity for chain-of-thought or recurrent architectures becomes evident: only by explicitly increasing the number of observable, interpretable serial steps can models address inherently serial problems. Empirically, chain-of-thought generation in transformers permits simulation of circuits of arbitrary size, lifting expressiveness up to $C$7. Results show that a depth-1 transformer with $C$8 CoT steps can solve $C$9-complete tasks with accuracy approaching 100%, while the same model without CoT remains at chance (Li et al., 2024).

6. Practical Measurement and Applications Across Domains

Opaque serial depth as a metric extends beyond neural LLMs. In computer graphics, depth-peeling methods (e.g., DP-GES) explicitly reconstruct the true front-to-back ordering of opaque surfel layers per pixel, enforcing correct serial compositing without hard global sorting (Ye et al., 25 May 2026). In optical imaging, anisotropic opaque lenses (e.g., Wavelens) enable serial depth scans through scattering media while preserving light-sheet quality, facilitating volumetric imaging by serially stepping through depth layers (“Opaque Serial Depth” imaging) (Mylonakis et al., 16 Sep 2025). In scattering physics, mutual scattering measurements recover axial depth of scatterers in opaque media, with the “susceptivity” metric quantifying depth sensitivity (Truong et al., 2022).

In all cases, the underlying principle is constant: the ability to probe or reconstruct hidden serial structures within otherwise opaque or parallelized systems is determined by formal or physical bottlenecks—encoded mathematically by opaque serial depth.

7. Significance, Limitations, and Directions

Opaque serial depth provides a principled, theoretically robust lens on model interpretability and computational power, exposing the irreducible sequential budget necessary for nontrivial serial computation in models and physical systems. Its computation, both analytic and algorithmic, sets upper bounds for hidden reasoning, justifying monitoring protocols such as chain-of-thought and motivating architectural innovations for inherently serial tasks (Brown-Cohen et al., 10 Mar 2026, Liu et al., 16 Jul 2025, Li et al., 2024).

A plausible implication is that future progress on “hard” AI problems will require a systematic commitment to scaling serial computation—the depth between interpretable states cannot be circumvented by parallelism alone for inherently serial tasks. These insights also reinforce the need for low-latency, high-depth compute architectures and explicit externalization mechanisms for trustworthy and auditable reasoning in machine learning systems.