---
title: 'ScheduledDropPath: Adaptive NASNet Regularization'
url: https://www.emergentmind.com/topics/scheduleddroppath
type: topic
---

# ScheduledDropPath: Adaptive NASNet Regularization

ScheduledDropPath is a stochastic regularization technique designed to improve generalization in neural architectures with multi-branch cells, particularly those developed via neural architecture search such as NASNet. It extends the fixed-rate DropPath approach by modulating the probability of dropping computational paths as a function of training progress, thus addressing key deficiencies in traditional stochastic path regularization for deep, over-parameterized, multi-branch structures [1707.07012].

## 1. Motivation for ScheduledDropPath

Standard DropPath regularization independently drops each computational branch in a multi-branch cell with a fixed probability $p$. Early empirical observations in NASNet training revealed that a constant drop rate was inadequate: small $p$ values led to insufficient regularization, while large values disrupted signal propagation during the critical early phases of feature learning. ScheduledDropPath was developed to provide gentle regularization during initial training when low-level filters are forming and progressively stronger regularization as the network's representational motifs mature. This temporal adaptation is essential for stabilizing learning and enhancing generalization in NASNet-style computational graphs [1707.07012].

## 2. Mathematical Formulation

Let $E$ denote the total number of training epochs and $t \in [0, E]$ indicate the current epoch or suitably normalized training index. Two hyperparameters govern the schedule:
- $p_{\min}$: Initial drop probability at the start of training ($t=0$).
- $p_{\max}$: Maximum drop probability at the end of training ($t=E$).

The drop probability $p(t)$ is scheduled via a linear ramp:
\[
p(t) = p_{\min} + (p_{\max} - p_{\min})\,\frac{t}{E}
\]
For each path at training epoch $t$, the branch output $x \in \mathbb{R}^{H \times W \times C}$ is replaced by
\[
\widetilde{x} = 
    \begin{cases}
    0, & \text{with probability } p(t) \\
    \frac{x}{1-p(t)}, & \text{with probability } 1-p(t)
    \end{cases}
\]
The scaling factor $\frac{1}{1-p(t)}$ preserves the expected activation magnitude, following the principle of inverted dropout.

## 3. Scheduling Strategy and Hyperparameters

ScheduledDropPath employs a zero-initialized ramp, setting $p_{\min}=0$ and linearly increasing to $p_{\max}$ at epoch $E$. Published NASNet experiments report the following settings:
- **CIFAR-10**: $E=600$, $p_{\min}=0.0$, $p_{\max}=0.5$
- **ImageNet**: $E=350$, $p_{\min}=0.0$, $p_{\max}=0.5$

A sweep over $p_{\max} \in \{0.3,0.4,0.5\}$ showed $p_{\max}=0.5$ yielded the optimal trade-off between regularization strength and gradient flow for NASNet architectures.

## 4. Implementation in NASNet Cells

ScheduledDropPath is applied to every distinct branch in a NASNet cell during the forward pass, using the same schedule $p(t)$. The procedure operates as follows:

- For each input branch, draw an independent Bernoulli mask with success probability $1-p(t)$. 
- If the branch is dropped, output zero; otherwise, scale activations by $1/(1-p(t))$.
- No dropping occurs at test time.

### NASCellBlock Pseudocode

```python
function NASCellBlock(h_a, h_b, t):
  x_a ← op_a(h_a)
  x_b ← op_b(h_b)
  p ← p_max * (t / E)    # linear ramp schedule
  keep_mask_a ← Bernoulli(1−p)
  keep_mask_b ← Bernoulli(1−p)
  if keep_mask_a == 0:
    x_a ← 0
  else:
    x_a ← x_a / (1−p)
  if keep_mask_b == 0:
    x_b ← 0
  else:
    x_b ← x_b / (1−p)
  h_out ← combine_fn(x_a, x_b)
  return h_out
```
In practice, this process is incorporated into each multi-branch block within the overall cell structure [1707.07012].

## 5. Comparison to Fixed-Rate DropPath

DropPath with a fixed probability $p$ regularizes all training phases equally. If $p$ is set low, NASNet’s over-capacity is insufficiently constrained; if $p$ is high, sensitivity to dropped paths during early training can severely impair feature acquisition. ScheduledDropPath’s zero-to-high ramp allows networks to learn robust low-level filters initially, applying strong regularization only as higher-level representations solidify. Empirically, fixed-rate DropPath yields only marginal improvements or, if poorly tuned, degrades performance. ScheduledDropPath consistently delivers marked generalization gains in experiments across CIFAR-10 and ImageNet [1707.07012].

## 6. Empirical Performance and Ablations

Empirical validation in [1707.07012] demonstrates the impact of ScheduledDropPath:

| Experiment                               | Test Error / Top-1 Acc.      | Regularization           |
|-------------------------------------------|------------------------------|-------------------------|
| CIFAR-10 NASNet-A (7@2304) baseline       | ~3.4% error                  | none                    |
| + Fixed-rate DropPath ($p=0.25$)          | ~3.25% error                 | moderate                |
| + ScheduledDropPath ($p:0 \to 0.5$)       | 2.97% error                  | strong, ramped          |
| + ScheduledDropPath + Cutout              | 2.40% error                  | state-of-the-art        |
| ImageNet NASNet-A (7@1920) baseline       | ~79.5% top-1                 | none                    |
| + Fixed-rate DropPath                     | ~80.0% top-1                 | moderate                |
| + ScheduledDropPath                       | 80.8% top-1                  | strong, ramped          |
| ImageNet NASNet-A (6@4032) + ScheduledDP  | 82.7% top-1 (best published) | strong, ramped          |

ScheduledDropPath led to state-of-the-art CIFAR-10 and ImageNet performances, with significant reductions in computational complexity (FLOPs) relative to previous best models. The compound effect of ScheduledDropPath with other regularizers (e.g., cutout) enabled test error reductions approaching one full percentage point. *A plausible implication is that ramped stochastic regularization synergizes particularly effectively with neural architecture search–based models comprising many parallel computational paths*.

## 7. Broader Implications

ScheduledDropPath’s development was motivated by architectural search–designed convolutional models (NASNet) with deep, multi-branch cells. By providing epoch-dependent stochastic path dropping, it addresses limitations of static regularization in dynamic learning environments. Its efficacy in large-scale image recognition benchmarks established a methodological precedent for time-varying regularization in deep networks. The approach remains significant for future neural architecture search efforts, especially where the learned topologies induce overparameterization and complex inter-branch dependencies [1707.07012].

Source: https://www.emergentmind.com/topics/scheduleddroppath