Morpho-MNIST: Enhanced Digit Benchmark

• Morpho-MNIST is an enhanced version of MNIST that integrates controlled morphometrics—such as stroke length, thickness, and slant—to provide interpretable shape attributes.
• The dataset computes detailed morphometric measures via upscaling, foreground segmentation, distance transform, and skeletonization, ensuring consistency in quantitative analysis.
• Morpho-MNIST supports robust evaluation of both generative and discriminative models through protocols like two-sample MMD, partial correlation, and outlier detection metrics.

Morpho-MNIST is an augmented version of the classic MNIST handwritten digit dataset, providing quantitative morphometric annotations and parameterized image perturbations. Its objective is to facilitate rigorous, objective assessment of representation learning—enabling measurement of a model’s ability to capture explicit factors of variation in data, discriminate out-of-distribution samples, and quantify sample diversity. The framework specifically addresses limitations in traditional benchmarks by supplying controlled shape measurements and perturbations, offering standardized input for generative and discriminative model benchmarking (Castro et al., 2018).

1. Morphometric Attribute Definition and Extraction

The core extension in Morpho-MNIST is the addition of measured shape attributes—termed "morphometrics"—which capture interpretable structure within each digit image. Morphometric computation proceeds via:

• Rescaling: Each $28 \times 28$ image is upscaled (default factor $f=4$) to $112 \times 112$, applying bicubic interpolation and Gaussian smoothing ($\sigma = 2f/6$).
• Foreground Segmentation: The image is binarised at the midpoint intensity (128), producing a binary mask.
• Distance Transform: Euclidean Distance Transform (EDT) $D(x, y)$ computes the minimal distance from each pixel $p$ to the nearest mask boundary.
• Skeletonization: The medial axis $S$ is extracted from the ridge pixels of the distance transform.

Morphometric attributes are computed as follows:

Attribute	Definition / Formula	Range / Notes
Stroke Length $L$	$\displaystyle L = \sum_{(p, q) \in E(S)} \|p - q\|$	$E(S)$ is the set of unordered skeleton neighbor pairs
Stroke Thickness $T$	$$\displaystyle T = 2 \frac{1}{|S|} \sum_{p \in S} D(p)$$	$T \in \mathbb{R}^+$, mean twice local half-width
Slant Angle $\alpha$	$$\displaystyle \alpha = \arctan\left(- \frac{M_{12}}{M_{22}} \right)$$	$\alpha \in (-\frac{\pi}{2}, \frac{\pi}{2})$
Width $W$, Height $H$	Computed as $W = u_{0.99} - u_{0.01}$, $H = v_{0.99} - v_{0.01}$ (see coordinate rotation by $\alpha$)	1%-99% quantiles define span
(Optional) Eccentricity $e$	$$e = \sqrt{1 - \frac{\lambda_{\min}}{\lambda_{\max}}}$$ where $\lambda_{\max}, \lambda_{\min}$ are eigenvalues of $M$	$e \in (0, 1)$ (not part of original release)

All subsequent analysis utilizes these five primary morphometrics: stroke length, thickness, slant, width, height (Castro et al., 2018).

2. Controlled Image Perturbations

Morpho-MNIST defines four parameterizable perturbations, grouped as global or local, to systematically vary digit appearances and introduce controlled distributional shift:

• Global Perturbations:
  • Thinning (Erosion): Binary mask is morphologically eroded using a disk structuring element of radius $r = \Delta \cdot T$, with $\Delta = -0.7$.
  • Thickening (Dilation): Morphological dilation with $r = \Delta \cdot T$, $\Delta = +1.0$.
• Local Perturbations:
  • Swelling (Elastic Deformation): A skeleton pixel $p_0$ is sampled uniformly; within a radius $R = 1.5 \sqrt{T}$ and strength $\gamma = 7$, each nearby pixel $p$ is transformed as $$p \mapsto p_0 + (p - p_0) (\frac{\|p - p_0\|}{R})^{\gamma - 1}$$.
  • Fractures: Small linear “cuts” are inserted perpendicular to the skeleton, away from branch or endpoint pixels; parameters include location (distance $\geq 2$ px from forks/tips), local orientation (from $5\times5$ window), length $L_f \approx 2$ px, thickness $\approx 1.5$ px.

Perturbed images are provided as $28 \times 28$ uint8 tensors. These operations yield out-of-distribution samples, enabling robust evaluation of domain adaptation and outlier detection algorithms (Castro et al., 2018).

3. Dataset Structure and Access

Morpho-MNIST is released in the original MNIST IDX format (train: 60,000, test: 10,000), with additional annotations:

• CSV Annotation Files:
  • train_morphometrics.csv, test_morphometrics.csv: Contains index and measured morphometrics for each digit.
  • train_perturbations.csv, test_perturbations.csv: Boolean flags for each perturbation (thin, thick, swell, fracture).
• Image Archives: Pickle or NumPy files for perturbed image variants.
• Python Package: The morphomnist package enables seamless data downloading, loading, and on-the-fly perturbation.

Storage design maintains alignment with MNIST conventions while introducing structured access to morphometric measurements and perturbation metadata (Castro et al., 2018).

4. Model Evaluation Methodologies

Morpho-MNIST supports rigorous, quantitative evaluation protocols for both generative and discriminative models.

4.1. Generative Model Assessment

• Sample Diversity (Two-Sample MMD):
  • For $N$ generated samples and the MNIST test set, extract their morphometrics $\{y_i\}$ and $\{x_i\}$.
  • Compute Maximum Mean Discrepancy (MMD) using a Gaussian kernel in the joint 5-dimensional morphometric space. Statistical significance indicates failure to match attribute distributions.
• Disentanglement (Partial Correlation & MIG):
  • Infer latent codes $c \in \mathbb{R}^D$.
  • For each morphometric $y_j$ and code $c_i$, compute the partial correlation coefficient $r_{c_i, y_j \mid c_{-i}}$.
  • Visualize $|r|$ matrix to identify code-attribute associations.
  • Measure Mutual Information Gap (MIG) per attribute to quantify disentanglement.

4.2. Discriminative Model Assessment

• Domain Adaptation (Robustness): Train on plain digits, test on perturbed sets, and report per-class and average accuracy drop $\Delta \text{acc}$ for classifiers (LeNet-5, k-NN, SVM, MLP, CNN).
• Outlier Detection via Stratification: Restrict training to digits with $T < T_0$ (e.g., $T_0=3\,$px), test across all thicknesses, and evaluate using model uncertainty (e.g., MC-dropout mutual information) or auto-encoder reconstruction error. Calculate ROC AUC for “thick” hold-out samples.
• New Prediction Tasks: Binary classification of normal vs. perturbed digits, and regression to predict stroke thickness from image pixels (RMSE metric).

These evaluation protocols enable precise, reproducible comparison of methods’ ability to capture, disentangle, and react to explicit variations and anomalies (Castro et al., 2018).

5. Example Usage and Analysis Workflows

The provided Python package and code snippets support standardized usage:

• Dataset Loading: Using from morphomnist import fetch, users access images, labels, morphometric annotations, and perturbation flags in pandas DataFrames.
• Basic Statistics: Visualization of the marginal or joint distribution of morphometrics (e.g., histogram of thickness, hex scatter of thickness vs. slant) is enabled with seaborn.
• Two-Sample MMD: The two_sample_mmd_test function computes the MMD statistic for two morphometric samples.
• Partial Correlation & MIG Computation: Using scikit-learn for empirical covariance and mutual information, users can assess code-attribute correlations and disentanglement metrics.

These utilities streamline analytic workflows and promote methodological reproducibility (Castro et al., 2018).

6. Protocols and Best Practices

Recommended evaluation protocols include:

• Disentanglement: Compute partial correlations and MIG on both inferred and generated morphometrics; cross-validate attribute consistency with ground-truth codes where applicable.
• Robustness: Train on standard digits, test separately on each perturbation type; report and compare accuracy drops by class and overall.
• Outlier Detection: Define in-distribution subspaces using morphometry (e.g., thickness $< T_0$), train autoencoders or Bayesian classifiers, and score outliers using reconstruction error or predictive uncertainty; evaluate using ROC AUC over morphometric bins.
• Fine-tuning Protocols: Optionally, include adaptation speed via few-shot fine-tuning on perturbed data.

Strict adherence to these practices ensures comparability and quantitative rigor in benchmarking representation learning approaches (Castro et al., 2018).

7. Impact and Applicability

Morpho-MNIST delivers a compact framework for interpretable, quantitative, and systematic benchmarking of generative and discriminative models. The inclusion of controlled morphometric ground truth and distributional perturbations directly addresses the absence of suitable quantitative benchmarks in unsupervised and semi-supervised learning, enabling isolation of specific failure modes in representation learning—particularly for disentanglement, robustness to shape variation, and detection of outliers. The resource is designed to integrate seamlessly into existing machine learning pipelines, and its open-source repository and documentation facilitate broad adoption and extension (Castro et al., 2018).

Repository and further documentation: https://github.com/dccastro/Morpho-MNIST.