---
title: DeepOBS Benchmark Suite
url: https://www.emergentmind.com/topics/deepobs
type: topic
---

# DeepOBS Benchmark Suite

DeepOBS is an open-source Python package that implements a complete, reproducible, and extensible benchmark suite for stochastic optimization in deep learning. It automates the assessment and comparison of deep learning optimizers, addresses major reproducibility and fairness challenges, and provides a collection of realistic, ready-to-use test problems, pre-tuned baseline results, fair evaluation protocols, and publication-ready output. DeepOBS supports major frameworks, such as TensorFlow and PyTorch, and is designed to accelerate empirical optimizer research by establishing common standards for rigorous benchmarking [1903.05499].

## 1. Motivation and Objectives

Deep learning optimizer research has historically lacked a standard, reproducible protocol for fair evaluation. Typical challenges include the stochastic nature of deep learning (random seeds, minibatches), the discrepancy between training and generalization performance, the need to properly tune baselines, and risks of cherry-picking results. DeepOBS was created to automate best practices: ensure unbiased assessment, supply a diverse suite of test problems, provide carefully tuned baselines, and generate publication-ready outputs. The primary objectives are to enable quantitative, reproducible evaluation of stochastic optimizers and support transparent scientific comparison [1903.05499].

## 2. System Architecture

DeepOBS is organized into six principal layers, each encapsulating a key aspect of the benchmarking workflow:

| Layer           | Functionality                                      | Example Components                           |
|-----------------|---------------------------------------------------|----------------------------------------------|
| Data Loading    | Download, preprocess, batch, augment datasets     | MNIST, Fashion-MNIST, CIFAR-10, PTB-char     |
| Models          | Define "test problems" (dataset + architecture)   | MLPs, CNNs (VGG, ResNet), VAE, RNNs, toys    |
| Runners         | Orchestrate training/evaluation, manage seeds     | SingleRunBenchmark, multiple-seed execution  |
| Baselines       | Precomputed baseline results for optimizers       | SGD, SGD+Momentum, Adam                      |
| Runtime Est.    | Measure optimizer performance vs. SGD reference   | Wall-clock time ratio, per-epoch metrics     |
| Visualization   | Generate .tex output for learning curves/tables   | pgfplots code, ready for publication         |

Test problems span four domains: image classification (e.g., MNIST fully connected net, Fashion-MNIST CNN, VGG/ResNet on CIFAR-10, ResNet-50 on ImageNet), generative modeling (VAE on MNIST), natural language modeling (character-level RNN on PTB), and synthetic or toy problems (e.g., 2-D nonconvex "Quadratic Deep") [1903.05499].

## 3. Evaluation Protocol

### Stochasticity and Seeds

Each optimizer–problem pair is run $R \geq 5$ times with different random seeds. For each, all four critical performance curves are tracked: training loss $L(\theta)$, training accuracy $a_{\text{train}}$, test loss $L(\theta)_{\text{test}}$, and test accuracy $a_{\text{test}}$.

### Aggregation and Metrics

Aggregate statistics are computed as follows:
- Mean test error:
  $$
  \bar{e}_{\text{test}} = \frac{1}{R} \sum_{r=1}^R e_{\text{test}}^{(r)}
  $$
- Standard deviation:
  $$
  \sigma_e = \sqrt{\frac{1}{R-1} \sum_{r=1}^R \left(e_{\text{test}}^{(r)} - \bar{e}_{\text{test}}\right)^2}
  $$
- Additional metrics: reached performance thresholds (epochs or gradient evaluations until threshold), runtime (wall-clock and relative to SGD), tunability (spread of best hyperparameters, visualization of error vs. $\log_{10}\alpha$).

### Hyperparameter Tuning

By default, DeepOBS sweeps the learning rate $\alpha$ over a logarithmic grid: $\alpha \in [10^{-5}, 10^{2}]$, using 36 grid points. The optimal $\alpha$ (maximal test accuracy) is selected and reported, typically over $R=10$ runs. Tunability visualization is produced by plotting final test error as a function of $\log_{10}(\alpha)$ [1903.05499].

## 4. Workflow and User Interfaces

DeepOBS supports multiple paradigms for usage:

- **Python API (TensorFlow example):**
  ```python
  from deepobs.tensorflow.testproblems import get_problem
  from deepobs.tensorflow.runners import SingleRunBenchmark
  runner = SingleRunBenchmark(test_problem='CIFAR10SmallCNN',
                              optimizer='Adam',
                              hyperparams={'learning_rate':1e-3})
  runner.run()
  ```

- **Python API (PyTorch example):**
  ```python
  from deepobs.pytorch import test_problems, runners
  runner = runners.SingleRunBenchmark('CIFAR10ResNet', optimizer='Adam')
  runner.run()
  ```

- **Command-Line Interface:**  
  `deepobs-run --problem CIFAR10SmallCNN --opt SGD --lr 0.1 --runs 10`

- **Batch Configuration:**  
  Complex experiments (multiple optimizers and problems) may be specified via JSON or YAML configuration files.

## 5. Output Generation and Publication Integration

DeepOBS's output back-ends generate .tex files containing pgfplots code for learning curves (loss/accuracy vs. epochs) and tabulated summary statistics. Example .tex output for comparing optimizers on CIFAR-10:

```latex
\begin{tabular}{lccc}
\toprule
Optimizer & $\bar{e}_{\mathrm{test}}$ & $\sigma_e$ & Runtime (s) \\
\midrule
SGD       & 0.075 & 0.002 & 120 \\
Adam      & 0.068 & 0.003 &  98 \\
\bottomrule
\end{tabular}
```
These outputs are intended for direct integration in academic manuscripts [1903.05499].

## 6. Baseline Results and Comparative Insights

DeepOBS offers precomputed, hyperparameter-tuned baseline results for commonly used optimizers (SGD, SGD+Momentum, Adam) across both "small" and "large" test problems. Key empirical findings:
- No single optimizer performs best on every problem.
- Adam typically requires less tuning (optimal $\alpha$ in $10^{-4}\dots10^{-3}$); however, on CIFAR-100, Momentum outperforms Adam.
- Ranking by training loss does not reliably predict generalization ranking.
- Trade-offs between speed (epochs or wall-clock time) and generalization performance can be systematically quantified using DeepOBS protocols [1903.05499].

## 7. Extensibility and Fair Evaluation Standards

Researchers may extend DeepOBS with custom optimizers or new test problems:

- **Adding an Optimizer:**  
  Subclass the TensorFlow/PyTorch API or provide a custom update function, register defaults, and (optionally) recompute baselines or submit results.
- **Adding a Test Problem:**  
  Implement a DataSet class and a Model class defining forward pass and loss; register in the testproblems module.

When performing comparative studies, DeepOBS recommends:
- Uniform random seed protocol and equal number of runs.
- Consistent hyperparameter search budgets (e.g., identical grid size or Bayesian optimization limits).
- Runtime always measured relative to SGD on a reference task.
- Reporting all four canonical learning curves and aggregated summary statistics.

A plausible implication is that adherence to these standards, coupled with fast feedback on "small" problems and scalability to "large" tasks, provides an efficient, fair platform for optimizer development and evaluation [1903.05499].

Source: https://www.emergentmind.com/topics/deepobs