Random Band Dropout in Neural ODEs

• Random Band Dropout is a stochastic regularization method for continuous-time models, particularly neural ODEs, using random batch sampling to create piecewise-constant vector fields.
• It achieves unbiased estimation and explicit convergence guarantees by balancing computational cost with rigorous error controls.
• RBM-Dropout generalizes classical dropout techniques, enabling principled speed-accuracy trade-offs and efficient adjoint-based training in large neural architectures.

Random Band Dropout (RBM-Dropout) is a stochastic regularization scheme for continuous-time models, particularly neural ordinary differential equations (neural ODEs), where neuron batches are randomly sampled at discrete times to define a piecewise-constant vector field. Formulated as a random-batch method, RBM-Dropout generalizes classical discrete-time dropout and enables unbiased estimation, principled speed-accuracy trade-offs, and theoretical convergence guarantees. The approach provides a mathematically rigorous framework that leverages batch sampling to offer computational efficiency and improved regularization in the training of large neural architectures (Álvarez-López et al., 15 Oct 2025).

1. Construction and Mathematical Formulation

The foundation of RBM-Dropout is the replacement of the standard ODE system

$$\dot{x}_t = F(x_t,\theta_t), \quad F(x,\theta) = \sum_{i=1}^p f_i(x, \theta_i)$$

with a stochastic, piecewise-constant system driven by random batch selection. The parameter index set $[p]$ is covered by $n_b$ batches $\mathcal{B}_j \subset [p]$, each selected independently at uniformly spaced time intervals $[t_{k-1}, t_k)$ of length $h$, with probabilities $q_j > 0$, $\sum_j q_j = 1$. The inclusion probability for neuron $i$ is $\pi_i = \sum_{j: i \in \mathcal{B}_j} q_j > 0$, and Horvitz–Thompson weighting by $[p]$0 is used to ensure unbiasedness.

The RBM-Dropout system is then defined as

$[p]$1

where the batch index $[p]$2 is sampled i.i.d. according to $[p]$3 at each interval, and $[p]$4. This construction ensures the estimator is unbiased: $[p]$5 The solution process $[p]$6 is a pure-jump ODE, piecewise-constant except at the discrete update times.

2. Convergence Analysis

RBM-Dropout admits explicit convergence guarantees for both trajectory-wise and distributional errors.

• Trajectory-wise Convergence: Under uniform Lipschitz and growth conditions on $[p]$7 (with constant $[p]$8), the mean-squared error between the true and dropout trajectories satisfies

$[p]$9

where $n_b$0 depends on $n_b$1, $n_b$2, the total variance $n_b$3, the sampling probabilities, and initial conditions. Therefore, $n_b$4.

• Distributional Stability: For the continuity equations of the induced flows, and with $n_b$5 Lipschitz, the pointwise $n_b$6 error in density is $n_b$7; the total-variation ($n_b$8) error in distribution satisfies

$n_b$9

as $\mathcal{B}_j \subset [p]$0, for $\mathcal{B}_j \subset [p]$1-dimensional state-space and finite $\mathcal{B}_j \subset [p]$2-th moment initial data.

3. Training, Adjoint Analysis, and Control Deviation

RBM-Dropout admits rigorous analysis within optimal control and Pontryagin adjoint frameworks for training:

• The regularized total cost $\mathcal{B}_j \subset [p]$3 to minimize is

$\mathcal{B}_j \subset [p]$4

with corresponding adjoint dynamics and costate analysis.

• Cost Deviation: For optimal controls $\mathcal{B}_j \subset [p]$5 and $\mathcal{B}_j \subset [p]$6 in the full and dropout problems respectively,

  $\mathcal{B}_j \subset [p]$7

• Adjoint-State Deviation: The adjoint error is controlled by

  $\mathcal{B}_j \subset [p]$8

  with $\mathcal{B}_j \subset [p]$9.

• Optimal Control Deviation (Affinely-Parametrized): Under $[t_{k-1}, t_k)$0-strong convexity,

  $[t_{k-1}, t_k)$1

• Gradient Descent Dynamics: For matching learning rates, the gap after $[t_{k-1}, t_k)$2 iterations remains $[t_{k-1}, t_k)$3.

4. Canonical Batch Schemes and Connections to Dropout

RBM-Dropout unifies several sampling schemes:

Scheme	Batches $[t_{k-1}, t_k)$4	Inclusion $[t_{k-1}, t_k)$5	Key S-order ($[t_{k-1}, t_k)$6)
Single-batch	$[t_{k-1}, t_k)$7	$[t_{k-1}, t_k)$8	$[t_{k-1}, t_k)$9
Drop-one	$h$0	$h$1	$h$2
Pick-one	$h$3	$h$4	$h$5
Balanced $h$6-subset	Subsets, $h$7	$h$8	$h$9
Disjoint blocks	Equal-size partitions ($q_j > 0$0)	$q_j > 0$1	$q_j > 0$2
All subsets uniform	$q_j > 0$3 subsets	$q_j > 0$4	$q_j > 0$5
Bernoulli($q_j > 0$6)	All subsets, size-$q_j > 0$7	$q_j > 0$8	$q_j > 0$9

Standard Bernoulli dropout is recovered as the limiting case with batch selection via coin-flips: $\sum_j q_j = 1$0, batch-weighted by $\sum_j q_j = 1$1, with corresponding variance expressions.

5. Cost–Accuracy Trade-Off and Design Optimization

RBM-Dropout enables a principled analysis of computational cost versus solution accuracy:

• Per-interval cost: $\sum_j q_j = 1$2, $\sum_j q_j = 1$3, $\sum_j q_j = 1$4 for stability.
• RMS error decomposition: $\sum_j q_j = 1$5, where $\sum_j q_j = 1$6 is the Euler integration constant.
• Optimal step size minimizing cost for a given error tolerance $\sum_j q_j = 1$7:

$\sum_j q_j = 1$8

with corresponding minimal cost $\sum_j q_j = 1$9.

Two primary regimes guide the step-size and computational resource allocation:

• Integration-limited ($i$0): $i$1, $i$2.
• Variance-limited ($i$3): $i$4, $i$5.

Relative to the full model ($i$6), RBM-Dropout yields substantial speedups in the integration-limited regime for $i$7 and moderate $i$8.

6. Practical Implementation and Applications

Implementation involves sampling a fixed dropout schedule $i$9 before training and retaining it across epochs, resulting in a mask analogous to structured pruning. For single-layer neural ODEs

$\pi_i = \sum_{j: i \in \mathcal{B}_j} q_j > 0$0

forward, transport, and adjoint error bounds apply for activation $\pi_i = \sum_{j: i \in \mathcal{B}_j} q_j > 0$1 of respective regularity ($\pi_i = \sum_{j: i \in \mathcal{B}_j} q_j > 0$2, $\pi_i = \sum_{j: i \in \mathcal{B}_j} q_j > 0$3). Regularization manifests as “fanned-out” trajectories, expanded decision boundaries, and overfitting resistance. Monte-Carlo model averaging of $\pi_i = \sum_{j: i \in \mathcal{B}_j} q_j > 0$4 realizations is used at inference. In high-batch regimes, RBM-Dropout provides up to $\pi_i = \sum_{j: i \in \mathcal{B}_j} q_j > 0$5 speedup and proportional memory reduction, demonstrated in experiments with PyTorch and standard ODE solvers.

The recommended workflow is:

• Determine target error tolerance $\pi_i = \sum_{j: i \in \mathcal{B}_j} q_j > 0$6.
• Compute critical $\pi_i = \sum_{j: i \in \mathcal{B}_j} q_j > 0$7.
• For $\pi_i = \sum_{j: i \in \mathcal{B}_j} q_j > 0$8, operate integration-limited ($\pi_i = \sum_{j: i \in \mathcal{B}_j} q_j > 0$9), else variance-limited ($[p]$00).
• Optimize batch size $[p]$01 to minimize $[p]$02 under this $[p]$03.

7. Summary of Properties and Theoretical Guarantees

Continuous-time dropout as a random-batch method:

• Provides an unbiased ODE solver with $[p]$04 trajectory error and $[p]$05 transport error,
• Preserves stability of training, adjoint states, and controls with cost deviations of $[p]$06,
• Offers design parameters and closed-form formulae to navigate speed-accuracy trade-offs,
• Allows substantial runtime and memory benefits for neural ODE training,
• Enhances regularization by diversifying trajectory flows and effectively widens decision boundaries.

The framework extensive subsumes classical Bernoulli dropout and offers rigorous performance guarantees within the context of continuous-time models (Álvarez-López et al., 15 Oct 2025).