Occam Gradient Descent

• Occam Gradient Descent is a neural network optimization framework that balances empirical error and model complexity by interleaving gradient steps with structured pruning.
• It dynamically prunes low-magnitude weights using a control loss metric, ensuring that the model remains as compact as possible while reducing overfitting.
• Empirical results demonstrate that OGD improves test accuracy and reduces compute cost, achieving significant compressions without sacrificing performance.

Occam Gradient Descent (OGD) is an algorithmic framework for neural network optimization that jointly minimizes empirical error and model complexity, directly motivated by learning-theoretic generalization bounds. By adaptively alternating between conventional gradient steps and structured model contraction (pruning), OGD aims to maintain the smallest model adequate for a given task, automatically balancing test accuracy and overfitting while substantially reducing compute and storage requirements (Kausik, 2024).

1. Learning-Theoretic Foundations

OGD is rooted in classical Occam-style generalization theory. In supervised learning, given a sample $S = \{(x_i, y_i)\}_{i=1}^m$ from distribution $P$, a neural network class $F_w$ parameterized by weights $w \in W$, and label set $Y=\{1,\ldots,k\}$, the true (test) error is $L_P(f)$, and the empirical error is $L_S(f)$. The approach starts from a standard generalization bound:

$$L_P(f)\;\leq\;L_S(f)\;+\;\sqrt{\frac{\dim(F)+\ln(1/\delta)}{2m}}$$

where $\dim(F)$ is the Natarajan dimension of hypothesis class $F$ and $P$0 is a confidence parameter. For fixed $P$1, this simplifies to:

$P$2

This characterizes the canonical tradeoff: larger networks reduce $P$3 but increase overfitting through their higher complexity term. OGD explicitly operationalizes this tradeoff by dynamically modulating both error and model size.

2. Joint Objective and Optimization Strategy

The implicit surrogate objective minimized by OGD is:

$P$4

Alternatively, by penalizing the count of nonzero parameters, $P$5, the surrogate becomes:

$P$6

Rather than incorporating the complexity term directly into gradient updates (as a regularization penalty), OGD interleaves standard gradient-based optimization with a pruning stage that adaptively reduces $P$7 at each epoch. The method targets overall test error minimization by guiding the model along both the weight and structural dimensions.

3. Algorithmic Implementation and Pruning Mechanism

A single iteration of OGD consists of the following components:

1. Gradient Step: Apply standard stochastic optimization (e.g., SGD or Adam) to update $P$8 by minimizing $P$9.
2. Compute Control Loss: Measure control loss $F_w$0 as training or held-out validation loss.
3. Adaptive Shrink-Rate Update: For $F_w$1 epochs, update pruning fraction $F_w$2 based on the change in control loss:

$F_w$3

1. Model Contraction: For each layer $F_w$4, compute the $F_w$5-quantile $F_w$6 across weights $F_w$7 and prune (set to zero) weights with $F_w$8; these weights are excluded from future optimization updates.

The process continues for a preset number of epochs, after which the remaining nonzero weights constitute the final sparse model. The adaptive $F_w$9 update is derived from a discrete derivative of the generalization bound, ensuring principled model shrinkage.

4. Theoretical Properties

OGD is accompanied by the following learning-theoretic guarantees:

• Generalization Guarantee: With probability at least $w \in W$0 over the draw of the training set $w \in W$1, every $w \in W$2 having effective dimension $w \in W$3 satisfies

$w \in W$4

• Descent Property: Provided $w \in W$5 is sufficiently smooth, the interleaved gradient and pruning steps guarantee a negative discrete directional derivative of the generalization bound. By carefully pruning to offset capacity increases from conventional optimization steps, OGD ensures a consistent decrease in test loss at every epoch:

$w \in W$6

A plausible implication is that OGD not only yields compact models, but also actively resists overfitting throughout training.

5. Empirical Evaluation and Performance

OGD has been empirically evaluated on standard vision and tabular data benchmarks, consistently outperforming conventional training approaches:

MNIST (814K-param linear net, 12 epochs avg over 10 runs):

• Baseline GD (best at epoch 9): Test loss = 0.077, Test acc = 98.1%, Model size = 100%, Compute cost = 9 epochs
• OGD ($w \in W$7, 10% holdback, best at epoch 8): Test loss = 0.052, Test acc = 98.5%, Model size = 20%, Compute cost $w \in W$8 3.05
• OGD ($w \in W$9, train-loss control, best at epoch 7): Test loss = 0.050, Test acc = 98.5%, Model size = 22.1%, Compute cost $Y=\{1,\ldots,k\}$0 2.93
• Post-train pruning (22.1% size) + retrain: Test loss = 0.115, Test acc = 96.8%, Compute cost $Y=\{1,\ldots,k\}$1 10.8

CIFAR-10 (486K-param conv net, $Y=\{1,\ldots,k\}$2, train-loss control):

• GD test loss plateaus $Y=\{1,\ldots,k\}$3 0.8, test acc $Y=\{1,\ldots,k\}$4 70% by epoch 20
• OGD test loss reaches $Y=\{1,\ldots,k\}$5 0.6, test acc $Y=\{1,\ldots,k\}$6 75% by epoch 20, while shrinking model to $Y=\{1,\ldots,k\}$730% of original size

Across tasks, OGD demonstrates:

• Resistance to overfitting (continual decrease in test loss)
• Higher final test accuracy
• Drastic reduction in effective trained model size
• Compute cost reduced to as little as ¼–⅓ that of full GD

6. Practical Applications and Deployment Considerations

OGD operates as a modular, “plug-and-play” algorithmic wrapper for any gradient-based neural network trainer. It requires no changes to underlying architectures and can be seamlessly deployed with standard optimizers. Empirical results show benefits across domains—including image classification, tabular tasks, and transformer-based models—where OGD consistently outperforms traditional SGD/Adam and post-training pruning/retrain regimes in both accuracy and efficiency (Kausik, 2024).

The routine is especially advantageous in scenarios where compute and memory efficiency are critical, enabling the training of smaller yet more generalizable models with less resource expenditure. The adaptive pruning strategy is flexible, with minimal hyperparameter tuning, and is compatible with a range of architectures.

7. Connections to Related Work and Extensions

OGD's theoretical and algorithmic lineage traces directly to Occam-style generalization bounds and the evolution of complexity-aware optimization protocols. The distinguishing feature is OGD’s explicit, epoch-wise alternation between fitting error minimization and model contraction, in contrast to (a) static penalization via regularizers, or (b) a post-hoc sequence of pruning and retraining. Its adaptive, control-loss-based pruning schedule is derived from sharp bounds on discrete generalization error.

By jointly controlling both parameter count and empirical risk, OGD provides a structurally principled and empirically validated path to efficient neural network design and deployment, strengthening the toolkit for scalable and interpretable model development (Kausik, 2024).