Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms (1708.07747v2)

Abstract: We present Fashion-MNIST, a new dataset comprising of 28x28 grayscale images of 70,000 fashion products from 10 categories, with 7,000 images per category. The training set has 60,000 images and the test set has 10,000 images. Fashion-MNIST is intended to serve as a direct drop-in replacement for the original MNIST dataset for benchmarking machine learning algorithms, as it shares the same image size, data format and the structure of training and testing splits. The dataset is freely available at https://github.com/zalandoresearch/fashion-mnist

• The paper introduces a novel dataset of 70,000 fashion images designed as a challenging alternative to MNIST for ML benchmarking.
• It details a preprocessing pipeline that converts high-resolution fashion photos into uniform 28x28 grayscale images.
• Benchmark results show that standard classifiers yield lower accuracy on Fashion-MNIST, highlighting the need for advanced image recognition methods.

Fashion-MNIST: A Novel Image Dataset for Benchmarking Machine Learning Algorithms

The paper "Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms" by Han Xiao, Kashif Rasul, and Roland Vollgraf from Zalando Research introduces a new dataset specifically designed for evaluating machine learning algorithms. Fashion-MNIST aims to provide a more challenging benchmark compared to the widely-used MNIST dataset while maintaining full compatibility with it.

Dataset Overview

Fashion-MNIST contains 70,000 images of fashion products, organized into 10 categories with 7,000 images per category. The dataset is divided into 60,000 training images and 10,000 testing images. Each image is a 28x28 grayscale image, mirroring the structure and format of the original MNIST dataset. This design choice ensures that Fashion-MNIST can be used as a direct drop-in replacement for MNIST, facilitating seamless transition for researchers accustomed to MNIST.

Motivation and Background

The paper highlights the stagnation in the utility of the MNIST dataset for benchmarking, given that modern deep learning methods have achieved near-perfect accuracies on it. Despite its simplicity, MNIST has remained popular due to its small size, easy access, and the ubiquity of pre-built functions in major machine learning libraries and frameworks. However, the simplistic nature of digit classification in MNIST no longer offers a significant challenge to state-of-the-art algorithms, underlining the necessity for a more complex dataset like Fashion-MNIST.

Data Collection and Processing

The images in Fashion-MNIST are derived from Zalando’s extensive catalog of fashion products. The dataset includes a variety of product types, excluding white-colored items due to their low contrast with the background. The raw images, originally captured in a 762x1000 JPEG format, undergo several preprocessing steps:

1. Conversion to PNG format.
2. Trimming edges based on color intensity.
3. Resizing to 28x28 pixels using subsampling.
4. Applying Gaussian sharpening.
5. Centering the image on a 28x28 canvas.
6. Intensity negation.
7. Conversion to 8-bit grayscale.

This preprocessing ensures uniformity and comparability to the MNIST dataset, while preserving the more complex visual features of fashion products.

Benchmarking Results

The paper provides a comprehensive set of benchmark results comparing Fashion-MNIST to MNIST across a variety of machine learning classifiers. Key classifiers include Decision Trees, Random Forests, SVMs, K-Nearest Neighbors, and various neural network architectures. The results indicate that Fashion-MNIST is generally more challenging, with lower accuracy scores than MNIST across most models. For instance, a RandomForestClassifier achieved an accuracy of 87.3% on Fashion-MNIST, compared to 97.0% on MNIST. Similarly, an SVC with a polynomial kernel attained 89.1% accuracy on Fashion-MNIST versus 97.6% on MNIST.

Implications and Future Directions

The introduction of Fashion-MNIST has several important implications. Practically, it provides the machine learning community with a more robust benchmark for model evaluation, encouraging the development of more sophisticated algorithms. Theoretically, it offers a new avenue for research into image classification tasks that exhibit greater variability and complexity than handwritten digits.

Future developments may involve the creation of datasets that incorporate even more diverse and challenging visual features, possibly extending beyond fashion into other domains. Additionally, exploring transfer learning applications using Fashion-MNIST as a pre-training dataset might yield interesting insights, given its balance of complexity and structure.

In conclusion, Fashion-MNIST serves as a valuable resource for advancing the field of machine learning by providing a dataset that is both accessible and demanding. The thorough benchmarking efforts and meticulous data curation underscore its potential to become a standard for evaluating new algorithms, much like its predecessor, MNIST.