Deep Sequential Neural Networks

• Deep Sequential Neural Networks are architectures that use input-conditioned routing to determine dynamic inference paths and specialized transformations.
• They leverage policy gradient methods to effectively sample multiple candidate mappings, resulting in a rich mixture of sub-networks.
• Integrating state space models enables continuous-time evolution of deep features, enhancing long-range memory and scalability.

A deep sequential neural network (DSNN) is a neural architecture in which the sequence of mappings applied to a given input is determined by a series of input‐conditioned routing or aggregation decisions at each layer, resulting in dynamic inference paths or state evolution through depth. This paradigm fundamentally extends classical feedforward models by allowing per‐example specialization among multiple candidate transformations at each stage or, alternatively, by interpreting network depth as a continuous or recurrent dynamical process. Recent research combines both perspectives: the original DSNN is formulated as a directed acyclic graph (DAG) of local mappings with sequentially chosen paths (Denoyer et al., 2014), while contemporary work leverages state space models (SSM) to treat the evolution of deep network features as a continuous‐time process through the network depth (Liu et al., 12 Feb 2025). Both approaches increase representational expressivity and memory compared to standard networks by endowing depth with sequential decision or state dynamics.

1. Architecture and Mathematical Formulation

The DSNN architecture introduced by Denoyer and Gallinari is structured as a directed acyclic graph of nodes ("layers") $\{n_1, ..., n_N\}$, with the root $n_1$ in input space $\mathbb{R}^X$ and the leaves in output space $\mathbb{R}^Y$. Each node $n$ is associated with a set of child nodes, each corresponding to a candidate mapping $$f_{n, n'}: \mathbb{R}^{\dim(n)} \to \mathbb{R}^{\dim(n')}$$ and a selection function $p_n : \mathbb{R}^{\dim(n)} \to \mathbb{R}^{\#n}$, where $\#n$ is the number of children of $n$.

At inference, given an input $x$, the network samples a sequence of actions $H = (a^{(1)}, \ldots, a^{(D-1)})$ that select which child mapping to apply at each node, thus defining a unique path from root to leaf. The output is

$F(x, H) = z^{(D)}\,,$

where intermediate representations are propagated sequentially: $z^{(1)} = x,\quad n^{(1)} = n_1,$

$$n^{(t+1)} = c_{n^{(t)}, a^{(t)}},\quad z^{(t+1)} = f_{n^{(t)}, n^{(t+1)}}(z^{(t)})\,.$$

The selection probability at each step is given by the softmax over $p_n(z)$.

This sequential routing allows the DSNN to implement a mixture of exponentially many sub-networks, unlike conventional feedforward nets where the transformation at each layer is fixed a priori. The DSNN strictly contains the standard multilayer perceptron as the special case where $\#n = 1$ for all nodes (Denoyer et al., 2014).

2. Learning Algorithms

DSNN learning is framed as stochastic optimization of the expected risk over both input data and action trajectories: $$J(\theta, \gamma) = \mathbb{E}_{(x, y)}\,\mathbb{E}_{H \sim P(\cdot|x; \gamma)}\left[\Delta(F(x, H; \theta), y)\right]\,,$$ where $\theta$ (mappings) and $\gamma$ (selectors) parameterize all $f$ and $p$ in the network, and $\Delta$ is a task loss.

The gradient is decomposed using the log-derivative trick: $$\nabla_{\theta, \gamma} J = \mathbb{E}_{x, y, H}\left[ \nabla_{\theta, \gamma} \log P(H | x; \gamma)\, \Delta(F(x, H), y) + \nabla_{\theta, \gamma} \Delta(F(x, H), y) \right],$$ where

$$\log P(H | x) = \sum_{t=1}^{D-1} \log P(a^{(t)} | z^{(t)})$$

enables gradient propagation through both the policy (selectors) and the mappings. Practically, gradients are approximated using averages over $M$ sampled action trajectories per example, with a variance-reducing baseline $b$: $$\nabla_{\theta,\gamma}J \approx \frac{1}{\ell}\sum_{i=1}^\ell\frac{1}{M} \sum_{k=1}^M\left[\nabla \log P(H_{i,k} | x_i)(\Delta_i-b) + \nabla \Delta_i\right].$$ Parameter updates follow standard stochastic gradient descent.

This approach is inspired by policy gradient methods from reinforcement learning, allowing optimization over stochastic routing policies. When the DAG degenerates to a chain ($\#n = 1$ everywhere), DSNN training reduces exactly to backpropagation in classical feedforward architectures (Denoyer et al., 2014).

3. State Space Models and Deep Sequentiality

The sequential perspective on deep networks generalizes further in "From Layers to States" (Liu et al., 12 Feb 2025), which interprets the sequence of layer outputs as discretized samples of a continuous-time state space process: $\dot h(t) = A h(t) + B x(t)\,,$ with latent state $h(t) \in \mathbb{R}^N$ and network input $x(t)$. Under this model, the propagation through network depth becomes state evolution: $h^t = \overline A h^{t-1} + \overline B x^t\,,$ with $\overline A = \exp(\Delta A)$ and a first-order approximation $\overline B \approx \Delta B$. This allows seamless, linear-memory aggregation of features across extremely deep networks, overcoming the memory bottlenecks and rigidities of prior discrete aggregation (concatenation, attention, etc.).

The Selective State Space Model Layer Aggregation (S6LA) module implements this model within existing CNN and Vision Transformer (ViT) backbones by introducing an SSM latent state at each depth, updated via parameterized projections of the backbone outputs. This mechanism maintains long-range memory over depth at $O(N\log N)$ or better computational cost if $A$ has low-rank or diagonal structure.

4. Comparative Expressivity and Memory

DSNNs, both in the DAG-routing (Denoyer et al., 2014) and the SSM aggregation (Liu et al., 12 Feb 2025) formulations, offer enhanced expressivity relative to classical architectures:

• DSNN-DAG (Denoyer & Gallinari): By partitioning the input space and learning specialized transformations per region of feature space, the model can represent a rich mixture of local sub-networks. Empirical results show superior performance on multimodal or highly nonlinear tasks where global affine transformations are insufficient.
• SSM/Deep Sequential (S6LA): The continuous-time SSM abstraction provides theoretically principled long-range memory and smooth aggregation, in contrast to the sharp cutoff or high parameter cost of RNNs or Transformers. S6LA enables information flow across hundreds of layers without vanishing gradients or unmanageable memory scaling. Parameterization is provably efficient for Toeplitz/diagonalizable $A$.

A comparison table encapsulates salient distinctions among approaches:

Architecture	Depth Sequentiality	Memory Mechanism
Standard MLP/CNN	Fixed/global	Per-layer only
DSNN-DAG	Input-conditioned	Path-wise
SSM/S6LA	Continuous-time	State recursion
RNN	Sequential (temporal)	State recursion
Transformer	Fixed blocks, global	Self-attention

5. Empirical Evaluation

DSNN-DAG Results (Denoyer et al., 2014)

• Datasets: UCI (diabetes, fourclass, heart, sonar, splice: $\sim$1000 examples each), MNIST (14×14), "MNIST-negative" (digits inverted), synthetic 2D checkerboards.
• Setup: Compared baseline NNs to DSNN-k (k=2,3,5,10 choices per layer), varying hidden layer size and activation (tanh, ReLU), using classification accuracy as the metric.
• Findings:
  • On UCI tasks, DSNN-2/3 often outperforms NNs, especially for small hidden sizes.
  • For MNIST-negative, DSNN-2 achieves up to 88% accuracy without hidden layers, compared to NN’s 27.7%.
  • On checkerboards, DSNN-3 approaches 99% accuracy on 3×3 tasks, while NNs saturate at 50–53%.

S6LA Results (Liu et al., 12 Feb 2025)

• Tasks: ImageNet-1K classification, MS COCO object detection (Faster R-CNN), Mask R-CNN instance segmentation.
• Sample Results:

Backbone	Method	Top-1	Top-5	$AP^{bb}$ (COCO)
ResNet-50	Vanilla	76.1	92.9	36.4
	MRLA	77.5	93.7	40.1
	S6LA	78.0	94.2	40.3

• Key Observations: S6LA typically confers a consistent improvement (up to $\sim$2% Top-1 accuracy) with minimal parameter/FLOP overhead ($\sim$0.2–0.3M). It outperforms prior aggregation schemes (SE, CBAM, RLA, MRLA) in both image classification and detection.

6. Insights, Limitations, and Future Directions

DSNNs demonstrate pronounced advantages when the data-generating distribution is a composite of regimes, justifying the increased model capacity for distinct patterns. Excessive candidate mappings or dominance by a single regime can cause parameter proliferation or degrade stability (Denoyer et al., 2014). S6LA’s introduction marginally increases parameter and computational cost; further reduction could be achieved by additional factorization or structure in $A$ (Liu et al., 12 Feb 2025).

Proposed research directions include:

• Richer selection functions (deep, convolutional, or non-linear) for both DSNN-DAG and S6LA.
• Explicit regularization to control network size and sparsity.
• Continuous relaxations (e.g., Gumbel-softmax) to enable end-to-end differentiability in DSNN routing.
• Early-stopping and budgeted inference for DSNNs, to facilitate anytime predictions.
• Investigation of nonlinear state dynamics and hybrid depth-time models for SSM-based architectures.

The efficacy of S6LA for modalities beyond vision, such as speech and text, is not yet established. Theoretical analysis concerning stability, memory retention, and behavior as network depth grows arbitrarily large remains an open area (Liu et al., 12 Feb 2025).

7. Relationship to Broader Sequential and Aggregation Approaches

The DSNN framework is conceptually linked to several major trends in deep learning:

• Reinforcement Learning: DSNN-DAG’s use of policy gradient and trajectory sampling is reminiscent of RL-based meta-model selection.
• Memory Models: SSM-based sequentiality shares properties with RNNs and Neural ODEs but emphasizes state propagation over depth rather than over time.
• Layer Aggregation: DenseNet, MRLA, and other aggregation mechanisms can be recast as special cases within the SSM paradigm, but lack the controllable long-range memory and linear scaling of S6LA (Liu et al., 12 Feb 2025).

Overall, deep sequential neural networks unify and extend the sequentiality implicit in depth via explicit, input- or state-adaptive selection, thereby enhancing the flexibility, memory, and specialization potential of deep learning models (Denoyer et al., 2014, Liu et al., 12 Feb 2025).