MNIST-Gen: Modular MNIST-Style Dataset Generation

• MNIST-Gen is an automated, modular framework for generating MNIST-style datasets from user-specified semantic categories with minimal manual intervention.
• It integrates CLIP-based hierarchical semantic filtering and reinforcement learning to achieve around 85% categorization accuracy and significant annotation time reduction.
• The framework supports domain-specific benchmarking, demonstrated on Tree-MNIST and Food-MNIST, and formalized using category theory for pipeline modularity.

MNIST-Gen denotes an automated framework for constructing MNIST-style datasets from user-specified semantic categories, and more broadly a line of work concerned with generating, transforming, or repurposing MNIST-compatible image distributions for benchmarking and generative modeling. In its specific 2025 formulation, MNIST-Gen is described as an “automated, modular, and adaptive framework” that combines CLIP-based semantic understanding, reinforcement learning, human feedback, and a category-theoretic view of data transformations to produce task-specific grayscale datasets such as Tree-MNIST and Food-MNIST (Shaeri et al., 16 Jul 2025). The name also appears in earlier literature as a shorthand for MNIST generation pipelines, especially GAN-based handwritten-digit synthesis and its evaluation (Düzyel, 2023).

1. Definition and conceptual scope

In the narrow sense established by the paper titled "MNIST-Gen: A Modular MNIST-Style Dataset Generation Using Hierarchical Semantics, Reinforcement Learning, and Category Theory" (Shaeri et al., 16 Jul 2025), MNIST-Gen addresses a specific problem: standard datasets such as MNIST and Fashion-MNIST are accessible but are “limited to generic classes such as digits or clothing items,” whereas many domain-specific tasks require small, standardized benchmarks for categories such as trees or food. The framework therefore starts from user-defined categories, retrieves candidate images, semantically filters and labels them, and converts them into MNIST-style outputs with minimal manual intervention (Shaeri et al., 16 Jul 2025).

The system’s stated design goals are automation, modularity, extensibility, hierarchical semantics, domain specificity, human-in-the-loop efficiency, and formal clarity through category theory (Shaeri et al., 16 Jul 2025). Its proof-of-concept outputs are Tree-MNIST and Food-MNIST, and the paper reports approximately 85% automatic categorization accuracy together with approximately 80% reduction in manual annotation time relative to fully manual approaches (Shaeri et al., 16 Jul 2025).

A broader usage of the term predates this framework. In "A Comparative Study of GAN-Generated Handwriting Images and MNIST Images using t-SNE Visualization" (Düzyel, 2023), “MNIST-Gen” refers to training a GAN on MNIST and evaluating how closely the synthetic samples align with the real MNIST distribution in t-SNE space. This suggests that the term functions both as the name of a specific dataset-generation framework and as a generic label for MNIST-centered generative workflows.

2. Hierarchical semantics and CLIP-based categorization

A defining property of MNIST-Gen is its use of explicit hierarchical semantic structure. Categories are organized as main classes, subcategories, and semantic characteristics. Formally, the hierarchy is written as $$\mathcal{H} = (\mathcal{C}, \mathcal{S}, \mathcal{A})$$, where $\mathcal{C}$ is the set of main categories, $\mathcal{S}$ the associated subcategories, and $\mathcal{A}$ the semantic characteristics attached to each subcategory (Shaeri et al., 16 Jul 2025). In implementation terms, these structures are specified in JSON, so the framework can be reconfigured without changing the rest of the pipeline.

CLIP, specifically OpenAI CLIP (ViT-B/32), is the core semantic engine (Shaeri et al., 16 Jul 2025). For an image $x_i$ and a text prompt $t_j$, image and text embeddings are computed as $\mathbf{v}_i = f_{\text{img}}(x_i)$ and $\mathbf{u}_j = f_{\text{text}}(t_j)$ in $\mathbb{R}^{512}$, and similarity is measured by cosine similarity:

$$\text{sim}(x_i, t_j) = \frac{\mathbf{v}_i \cdot \mathbf{u}_j}{\|\mathbf{v}_i\| \, \|\mathbf{u}_j\|}.$$

The prompt templates used in the implementation include forms such as “A photo of {category}” and “This is a {subcategory}” (Shaeri et al., 16 Jul 2025).

The categorization score is not based on CLIP text similarity alone. Instead, MNIST-Gen defines a combined score

$\mathcal{C}$0

where $\mathcal{C}$1 measures prompt-image compatibility, $\mathcal{C}$2 measures compatibility with semantic characteristics, and $\mathcal{C}$3 uses low-level visual statistics such as brightness, contrast, and edge density (Shaeri et al., 16 Jul 2025). The image is assigned to the highest-scoring subcategory, subject to thresholds and later filtering stages. The paper also notes a CLIP similarity threshold such as 0.3 for filtering clearly irrelevant images (Shaeri et al., 16 Jul 2025).

This hierarchical formulation distinguishes MNIST-Gen from flat prompt-based curation. A plausible implication is that its main novelty is not merely zero-shot classification, but zero-shot classification constrained by user-specified semantic ontologies.

3. Reinforcement learning and human feedback

MNIST-Gen frames dataset curation as a sequential decision problem. A Deep Q-Network decides, for each candidate image, whether to keep it in the predicted category, discard it, or send it to human review (Shaeri et al., 16 Jul 2025). The state representation concatenates semantic features, visual attributes, hierarchical confidence, and dataset-level context such as class frequency and similarity to already selected images (Shaeri et al., 16 Jul 2025).

The three discrete actions are:

• Keep: retain the sample in the predicted category.
• Discard: remove the sample as noise or outlier.
• Send to human review: defer the decision for manual correction (Shaeri et al., 16 Jul 2025).

The reward is explicitly defined as

$\mathcal{C}$4

with reported weights $\mathcal{C}$5, $\mathcal{C}$6, $\mathcal{C}$7, and $\mathcal{C}$8 (Shaeri et al., 16 Jul 2025). Here, semantic confidence rewards CLIP-consistent samples, entropy rewards class balance, model accuracy rewards downstream utility, and redundancy penalizes near-duplicate selections.

The DQN itself is a 3-layer MLP with hidden sizes 256, 128, and 64, trained with Adam at learning rate 0.001, replay buffer size 10,000, batch size 32, target network update every 100 steps, and $\mathcal{C}$9-greedy exploration with $\mathcal{S}$0 and decay 0.995 (Shaeri et al., 16 Jul 2025). The temporal-difference loss is the standard DQN objective:

$\mathcal{S}$1

Empirically, the paper reports approximately 89.3% decision accuracy with average reward about 0.67 after 1000 episodes on Tree-MNIST, and approximately 85.1% decision accuracy with average reward about 0.54 on Food-MNIST (Shaeri et al., 16 Jul 2025). Human feedback enters through three processing modes. Individual review maximizes control; smart batch processing automatically accepts high-confidence samples, reviews medium-confidence ones, and removes low-confidence ones; fast batch processing clusters CLIP embeddings so users can label or reject entire clusters (Shaeri et al., 16 Jul 2025). This architecture places manual review where semantic uncertainty is highest rather than distributing it uniformly across the dataset.

4. Transformation pipeline and category-theoretic formalization

MNIST-Gen models the conversion from raw images to MNIST-style outputs as a composition of transformations,

$\mathcal{S}$2

with stages for semantic analysis, hierarchical categorization, resizing, background removal, center cropping, grayscale conversion, and binarization or normalization (Shaeri et al., 16 Jul 2025). The paper gives explicit forms for several of these operations:

$\mathcal{S}$3

$\mathcal{S}$4

$\mathcal{S}$5

$\mathcal{S}$6

CLIP preprocessing uses 224×224 inputs, while final outputs are typically 28×28 grayscale, with 64×64 listed as a configurable alternative (Shaeri et al., 16 Jul 2025). Background removal uses U²-Net (Shaeri et al., 16 Jul 2025).

The category-theoretic apparatus is used to formalize modularity rather than to alter the numerics. Raw image datasets with hierarchical annotations form a category $\mathcal{S}$7, MNIST-style datasets form a category $\mathcal{S}$8, and the full pipeline is represented as a functor $\mathcal{S}$9 satisfying

$\mathcal{A}$0

Augmentations that remain within the MNIST-style space are modeled as endofunctors $\mathcal{A}$1, while alternative pipelines, such as one with RL filtering and one without, are compared through natural transformations $\mathcal{A}$2 satisfying

$\mathcal{A}$3

for each morphism $\mathcal{A}$4 (Shaeri et al., 16 Jul 2025).

This formalization is unusual in the MNIST literature. Earlier work on MNIST generation tends to specify optimization objectives or architectural components directly, such as GAN minimax losses (Düzyel, 2023), differentially private WGAN-GP training (Schwabedal et al., 2020), or successive subspace learning with Saab, ICA, and LLE (Lei et al., 2022), rather than modeling the full data-processing pipeline as a compositional category.

5. Demonstration datasets and empirical behavior

The framework is demonstrated on two novel datasets. Tree-MNIST contains four main categories—Broadleaf Tree, Cactus, Coniferous Tree, and Palm—each split into three subcategories, with approximately 400 images per main class retrieved and a final balanced dataset of 1,500 labeled grayscale 28×28 images split 80/20 into train and validation (Shaeri et al., 16 Jul 2025). Food-MNIST contains ten main categories, each with three subcategories, built from 5,000 images selected from Kaggle Food-11 and other sources, then converted to 28×28 grayscale MNIST-style images with an 80/20 train/validation split (Shaeri et al., 16 Jul 2025).

Dataset	Main classes	Output format	Best reported classifier
Tree-MNIST	4	28×28 grayscale	CNN (2 conv layers), 86.51%
Food-MNIST	10	28×28 grayscale	CNN (2 conv layers), 73.84%

The benchmark suite includes CNN, Random Forest, Gradient Boosting, KNN, a 2-layer fully connected neural network, Decision Tree, MLP, AdaBoost, SVM, Logistic Regression, and Gaussian NB (Shaeri et al., 16 Jul 2025). On Tree-MNIST, the top model is a 2-conv-layer CNN at 86.51% accuracy, followed by Random Forest at 82.73% and Gradient Boosting at 81.95% (Shaeri et al., 16 Jul 2025). On Food-MNIST, the same CNN reaches 73.84%, followed by Random Forest at 68.92% and the 2-layer fully connected network at 67.45% (Shaeri et al., 16 Jul 2025). Food-MNIST is explicitly harder, which the paper associates with larger class count and higher visual variability (Shaeri et al., 16 Jul 2025).

The annotation-efficiency results are equally central. For 5,000 Food-MNIST images, the reported comparison is: manual labeling, 25.0 hours and 92.3% accuracy; individual mode, 18.5 hours and 91.1%; smart batch, 5.2 hours and 89.7%; fast batch, 3.1 hours and 86.4% (Shaeri et al., 16 Jul 2025). These numbers support the claim that smart batch processing gives substantial time savings with a comparatively small accuracy loss.

6. Relation to the broader MNIST ecosystem

MNIST-Gen belongs to a larger ecosystem of MNIST-compatible datasets and MNIST-based generation methods. On the dataset side, Fashion-MNIST was explicitly designed as a direct drop-in replacement for MNIST while remaining harder than the original digit task (Xiao et al., 2017). Oracle-MNIST provides 30,222 images of oracle bone characters in 28×28 grayscale MNIST format and is described as a more realistic and more challenging classification task than MNIST (Wang et al., 2022). Typography-MNIST extends the format to 565,292 glyph images over 1,812 glyphs and 1,355 Google fonts (Magre et al., 2022). Afro-MNIST shows that a full 60,000/10,000 MNIST-style dataset can be synthesized for low-resource numeral systems from a single exemplar per class using elastic deformation with $\mathcal{A}$5 and $\mathcal{A}$6 (Wu et al., 2020). In that sense, MNIST-Gen extends an established tradition of treating “MNIST-style” as a reusable interface rather than as a fixed digit dataset.

On the generative-model side, the term also intersects with several distinct paradigms. A standard GAN+t-SNE evaluation pipeline for MNIST was presented in 2023, using 4,800 synthetic images and 4,800 real test images to study overlap in embedded space (Düzyel, 2023). Differentially private MNIST generation with WGAN-GP and a DP critic was studied through $\mathcal{A}$7-DP and Inception Score, with useful models reported at $\mathcal{A}$8 (Schwabedal et al., 2020). Interpretable feedforward synthesis through multi-stage PCA and random-forest AC prediction was proposed as an alternative to GANs and VAEs (Zhu et al., 2018). A single-hidden-layer Simple Generative Network optimized via a kNN-based KL objective was reported to outperform plain GMMN and exceed a GAN baseline on Parzen-estimated mean log-likelihood (Nissani, 2021). GenHop, based on successive subspace learning with Saab, ICA, histogram matching, and LLE, reported FID 5.1 on MNIST, lower than the compared WGAN value of 6.7 and VAE value of 23.8 (Lei et al., 2022). A hybrid quantum-classical QC-AAN with an 8-qubit ion-trap prior and multi-basis measurements achieved Inception Scores above 9.4 and 9.5 on hardware, surpassing a classical 16-dimensional DCGAN baseline (Rudolph et al., 2020). There are also nonstandard formulations such as using a trained classifier directly as a generator by optimizing the input image under a classifier loss plus a prior term (Li, 2022).

A common misconception is that “MNIST-Gen” refers only to digit synthesis with GANs. The literature does not support such a narrow reading. It can denote a specific modular dataset-construction framework (Shaeri et al., 16 Jul 2025), a generic MNIST generation-and-evaluation workflow (Düzyel, 2023), or, more loosely, an MNIST-compatible paradigm for domain transfer, privacy-preserving synthesis, interpretable generation, or task-specific benchmark construction. The unifying idea is not a single architecture but the preservation of the MNIST interface—small grayscale images, compact labels, and low-friction benchmarking—while changing either the data source, the semantic space, or the generative mechanism.