Deep Linearly Gated Networks

• DLGN is a neural architecture that uses data-dependent gating to partition inputs and select among banks of linear models, achieving powerful nonlinearity.
• It enables local, convex online training with per-unit learning rules that offer provable performance guarantees and universal function approximation.
• DLGN variants support practical applications such as continual learning, contextual bandits, and sequence modeling, while enhancing interpretability and robustness.

A Deep Linearly Gated Network (DLGN) is a neural architecture in which expressive, data-dependent nonlinearity arises from input-dependent gating or context selection, while all learned sub-models remain strictly linear. In contrast to standard deep networks that induce nonlinearity via fixed nonlinear activations (e.g., ReLU, sigmoid), DLGNs partition input or side-information space using gating functions and select among banks of linear models on a per-example basis. This architecture, originally termed the Gated Linear Network (GLN), admits local, convex, per-unit learning rules, enabling online or streaming adaptation with theoretical guarantees and universal approximation properties. The gating principle has been extended across deep feedforward architectures, recurrent models, and sequence-processing frameworks, leading to a spectrum of DLGN variants with differing inductive biases, learning dynamics, and interpretability advantages (Veness et al., 2017, Veness et al., 2019, Lakshminarayanan et al., 2022, Yadav et al., 2024, Sun et al., 21 Apr 2026).

1. Formal Architecture and Forward Computation

At its core, a DLGN is a composition of linear transformations modulated via data-dependent discrete or continuous context functions. In the canonical GLN setting (for density modeling or classification), the network operates recursively as follows:

• Each neuron (indexed by layer $i$ and position $k$) is endowed with a finite set of contexts $C$ and a context function $c_{ik}:\mathcal{Z} \to C$ mapping side information (e.g., input features, history) to a discrete context index.
• For each context $a\in C$, a weight vector $w_{ik,a} \in \mathbb{R}^{K_{i-1}}$ is stored (typically constrained to a box $[-b,b]^{K_{i-1}}$).
• Given previous-layer predictions $p_{i-1}(z)\in[0,1]^{K_{i-1}}$, the neuron selects $w_{ik}(z) := w_{ik,c_{ik}(z)}$ and forms its output by

$$p_{ik}(z) = \sigma\bigl(w_{ik}(z) \cdot \mathrm{logit}(p_{i-1}(z))\bigr)$$

where $k$0 is the sigmoid, and $k$1.

In matrix form, letting $k$2 be the layer-specific context-selected weight matrix, the layerwise update is

$k$3

By induction, the output is a data-dependent composition of linear maps in the logit-space: $k$4 The essential nonlinearity is due entirely to gating—each $k$5 depends on the input, not to per-layer activations.

Context functions $k$6 typically implement half-space splits (via dot-products and thresholds), skip-gram lookups, pooling, or other domain-specific selectors, effectively partitioning $k$7 into a combinatorially large number of regions. The resulting network output is a highly expressive, piecewise-linear (in the logits) function, defined over the input or side-information space (Veness et al., 2017, Veness et al., 2019).

2. Online, Local, and Convex Learning Algorithms

One of the defining properties of DLGNs (contrasting with standard DNNs) is the ability to train each context-specific weight vector via local, log-loss-driven online convex optimization, completely decoupled from the global network structure and without back-propagation.

For each neuron $k$8 and observed example with context $k$9, define its per-context, per-time local log-loss: $C$0 where $C$1. The gradient is: $C$2 With appropriate learning-rate decay (e.g., $C$3) and weight constraint/projection $C$4 to maintain bounded gradients, the weight update for the selected context $C$5 is

$C$6

All other context vectors remain unchanged. This mechanism enables highly parallel, truly online adaptation with per-neuron regret bounds and no need for global gradient computation (Veness et al., 2017, Veness et al., 2019).

3. Theoretical Guarantees and Universality

The DLGN architecture admits precise, constructive approximation theorems. Under mild regularity and "rich" context schemes (i.e., when the set of gating partitions is sufficiently expressive to separate any measurable subset), the main guarantees are:

• Convergence: For each neuron, there exists a deterministic prediction function $C$7 such that, as $C$8, the moving average loss converges to that of $C$9, and the predictions themselves converge pointwise in Cesàro mean.
• Layer-wise improvement: Each layer's optimal predictor achieves strictly decreasing integrated log-loss.
• Universality: If context functions generate a dense Boolean algebra (e.g., all half-spaces, balls with rational centers/radii), the DLGN can approximate any bounded Borel-measurable function on compact domains, paralleling or exceeding formal universal approximation results for classical neural architectures (Veness et al., 2017).
• Regret Bounds: For per-context, per-neuron online-gradient methods, the regret accumulated grows as $c_{ik}:\mathcal{Z} \to C$0; a second-order (Newton-like) method reduces this further to $c_{ik}:\mathcal{Z} \to C$1, guaranteeing vanishing per-step average loss (Veness et al., 2019).

Collectively, these results establish that DLGNs, despite lacking explicit nonlinearity, retain the full representational power of deep architectures via data-driven gating.

4. Inductive Bias, Implicit Regularization, and Dynamics

Recent analysis of DLGNs has revealed that gradient descent on the network implements a group-sparse, context-structured inductive bias:

• The infinite-time GD limit for GLNs solves a margin-maximizing, norm-minimizing problem over context-indexed predictors, encouraging sparsity in context-specific deviations—formally, a "group lasso" in context space (Lippl et al., 2022).
• Explicit "equivariance" constraints induced by the context architecture limit the degrees of freedom in deep configurations, shaping the geometry of the solution manifold.
• Closed-form reductions (via neural race dynamics and exact ODEs over path singular values) show that more-shared routes in the pathway graph converge faster under GD, implying an implicit bias toward representational reuse and abstraction (Saxe et al., 2022).
• Feature learning in DLGNs is achieved by gradient-steered shifting of context partition boundaries (e.g., moving half-spaces), leading to non-kernel behavior even when the overall network is functionally equivalent to a sparse mixture of linear units (Yadav et al., 2024).

Empirical findings corroborate this mathematical structure: generalization improves as gates adapt, freezing the context functions substantially harms performance, and manipulating the degree of context sharing tunes the balance between abstraction and specificity (Veness et al., 2019, Lippl et al., 2022, Saxe et al., 2022, Yadav et al., 2024).

5. Practical Implementation, Flexibility, and Applications

DLGNs support a broad spectrum of practical deployment scenarios:

• Online/continual learning: The local, convex, streaming learning rule provides robustness against catastrophic forgetting and is effective for continual or lifelong learning settings (Veness et al., 2019).
• Interpretability: The model's output is a data-dependent linear transformation of fixed input features (logits), amenable to exact saliency and path-analysis; context hyperplanes in the gating stage can be visualized and analyzed directly (Veness et al., 2019, Rao et al., 20 Feb 2025).
• Contextual bandits: The DLGN/GLN architecture forms the core of the Gated Linear Contextual Bandit (GLCB) algorithm, delivering state-of-the-art exploration/exploitation performance and free uncertainty quantification through context-specific pseudocounts (Sezener et al., 2020).
• Sequence modeling and memory: In recurrent/streaming variants (e.g., Gated DeltaNet, Depth-Gated LSTM), DLGN principles are leveraged to construct linearly-updated memories with input- and context-dependent gating, supporting efficient associative retrieval and long-horizon contexts (Yao et al., 2015, Sun et al., 21 Apr 2026).
• Adversarial robustness: DLGNs facilitate geometric analysis of adversarial robustness, as feature gates correspond directly to input-space hyperplanes; robust training strategies modulate the margin and orientation of these hyperplanes, increasing their resistance to targeted perturbations (Rao et al., 20 Feb 2025).

Limitations include potential storage costs for large context banks, the need for careful gating function design in high-dimensional input spaces, and reduced efficiency in hierarchical feature discovery when compared to classical backpropagation-trained deep nets.

6. Comparative Perspectives, Generalizations, and Future Directions

DLGNs sit at an interpretable midpoint between deep linear and deep nonlinear models. Every neural network with path-wise or affine gating structure—such as highway networks, residual networks, certain LSTM variants, and modern delta-rule long-context transformers—can be seen as a DLGN or a close relative if appropriately refactored (Yao et al., 2015, Lakshminarayanan et al., 2022, Lakshminarayanan et al., 2021).

Active research directions include:

• Generalization to architectures with continuous, probabilistic, or hybrid gating (e.g., combining hard context splits with soft attention scores).
• Characterizing universality and spectral approximation in DLGNs with learnable, deep linear gates (Lakshminarayanan et al., 2021, Lakshminarayanan et al., 2022).
• Exploring the trade-offs between gate richness, resource allocation, and learning speed in modular and multitask environments (Saxe et al., 2022, Li et al., 2022).
• Understanding and benchmarking DLGNs as surrogates for interpretable, robust, and efficient deep learning systems with mathematically tractable inductive biases (Yadav et al., 2024, Rao et al., 20 Feb 2025, Li et al., 2022).
• Integrating DLGNs as explanatory tools for black-box DNNs and as foundations for new continual and online learning benchmarks.

DLGNs offer a rigorous, transparent, and highly adaptable framework that distills nonlinearity to explicit, analyzable gating, while maintaining the scalability and function-approximation guarantees of deep architectures (Veness et al., 2017, Veness et al., 2019, Lippl et al., 2022, Li et al., 2022, Yadav et al., 2024).