Food-MNIST: Automated Food Image Benchmark

• Food-MNIST is a 10-class food recognition benchmark built using automated, hierarchical semantic data generation for domain-specific image classification.
• It employs a modular CLIP-based pipeline enhanced by reinforcement learning and human review to refine image retrieval and label accuracy.
• The dataset maintains MNIST-style simplicity while introducing semantic richness to address food presentation nuances and inter-class similarity challenges.

Food-MNIST is a 10-class, 30-subcategory MNIST-style food benchmark generated from 5,000 images by the MNIST-Gen framework, which was introduced as an automated, modular, and adaptive system for producing domain-specific image datasets from hierarchical semantic categorization, CLIP-based semantic understanding, reinforcement learning, and human feedback. In that framework, Food-MNIST serves as a proof-of-concept dataset for compact food classification in a format intended to preserve the simplicity of MNIST while encoding richer semantic structure through food categories and subcategories. The benchmark is positioned as a lightweight, publishable dataset for food recognition rather than digits or clothing, and it is explicitly described as more challenging than standard MNIST-style datasets because food classes often exhibit high inter-class similarity and depend on preparation, presentation, and context (Shaeri et al., 16 Jul 2025, Peng et al., 2023).

1. Definition and benchmark scope

Food-MNIST was introduced as one of the two proof-of-concept datasets created with MNIST-Gen, alongside Tree-MNIST. Its stated purpose is to provide a small, clean, task-relevant image dataset for a domain-specific problem—food recognition—under circumstances where manual dataset construction is slow, legally constrained, or impractical. The benchmark is therefore framed not merely as a collection of low-resolution images, but as an example of automated dataset generation for specialized visual classification tasks (Shaeri et al., 16 Jul 2025).

The dataset is explicitly a 10-class classification benchmark. It was produced from 5,000 images retrieved from the Kaggle Food-11 dataset and processed into $28\times 28$ grayscale format while validating subcategory labels through a hierarchical semantic system. The main categories are Bread, Dairy Product, Dessert, Egg, Fried Food, Meat, Noodles-Pasta, Rice, Seafood, and Vegetable-Fruit. Each main category contains exactly three subcategories, giving 30 subcategories total. This design is intended to make the label space semantically richer than flat MNIST-style taxonomies while retaining compactness and compatibility with standard workflows (Shaeri et al., 16 Jul 2025).

A common misconception is that an MNIST-style representation necessarily implies a trivial recognition problem. The benchmark description argues the opposite for food data: compared with standard MNIST-style datasets, Food-MNIST is far more domain-specific and semantically rich, but also more challenging, because food categories often overlap visually and depend on preparation, presentation, and context. This suggests that the “MNIST-style” designation refers primarily to format and workflow convenience rather than to low intrinsic task difficulty (Shaeri et al., 16 Jul 2025).

2. Generative framework and formal structure

Food-MNIST is generated by a pipeline that turns keyword prompts or existing image collections into an MNIST-style dataset through retrieval, semantic understanding, hierarchical categorization, preprocessing, and filtering. The framework is explicitly modular and adaptive. Its principal components are a hierarchical semantic model, a CLIP-based similarity engine, a reinforcement-learning controller, and human-in-the-loop review. In this architecture, dataset construction is treated as a structured decision process rather than as simple keyword-based scraping and resizing (Shaeri et al., 16 Jul 2025).

The framework is also described in category-theoretic language. Raw-image datasets and MNIST-style datasets are treated as categories, and preprocessing steps such as resizing, grayscale conversion, background removal, cropping, and binarization are treated as composable morphisms. The full workflow is formalized as a functor

$F: \mathcal{I} \to \mathcal{M},$

with the functor laws

$$F(\text{id}_X)=\text{id}_{F(X)}, \qquad F(f\circ g)=F(f)\circ F(g).$$

This formulation is presented to emphasize modularity, composability, and the possibility of swapping preprocessing components without changing the overall dataset-generation logic (Shaeri et al., 16 Jul 2025).

The hierarchy itself is defined as

$\mathcal{H}=(\mathcal{C},\mathcal{S},\mathcal{A}),$

where $\mathcal{C}$ denotes main categories, $\mathcal{S}_i$ the subcategories for each main category, and $\mathcal{A}_{i,j}$ the semantic characteristics attached to each subcategory. In Food-MNIST, this hierarchy is not only a labeling convenience. It functions as a semantic scaffold for downstream matching, filtering, and review, allowing the dataset-generation system to reconcile image content with human-defined food structure (Shaeri et al., 16 Jul 2025).

3. Hierarchical semantics and CLIP-based categorization

The Food-MNIST hierarchy centers on 10 broad food categories, each split into three subcategories. The paper explicitly gives examples: Bread is split into Sliced Bread, Whole Loaves, and Rolls and Buns; Dairy Product into Milk and Liquid Dairy, Cheese, and Yogurt and Cream; Dessert into Cakes and Pastries, Ice Cream and Frozen, and Cookies and Small Sweets. The appendix further describes intended semantics such as shape, texture, preparation style, and visual cues. “Milk and Liquid Dairy” is characterized by white liquid containers, “Cheese” by yellow/white blocks or slices, “Yogurt and Cream” by thick consistency, “Fried Food” categories by golden coating and crispy texture, and “Egg” categories by oval shape, shells, or cooked yolks. These descriptors are used operationally rather than illustratively (Shaeri et al., 16 Jul 2025).

CLIP provides the main semantic understanding mechanism. For each image $I$, the framework extracts a CLIP embedding $\mathbf{f}_{clip}\in\mathbb{R}^{512}$, along with visual features such as brightness, contrast, and edge density, plus object-presence scores from zero-shot CLIP classification. Assignment to a subcategory $s_{i,j}$ is based on the combined score

$F: \mathcal{I} \to \mathcal{M},$0

Here, text similarity compares image and prompt embeddings, characteristic similarity checks agreement with the semantic attributes of the subcategory, and visual similarity checks compatibility with expected appearance (Shaeri et al., 16 Jul 2025).

The implementation notes specify CLIP ViT-B/32 with 512-dimensional embeddings, prompt templates such as “A photo of {category}” and “This is a {subcategory},” and a similarity threshold of 0.3 for filtering. The resulting labels are therefore not assigned purely by keyword matching. Instead, the system attempts to align the retrieved image with the hierarchical semantic description of a food class. A plausible implication is that Food-MNIST encodes a stronger prior over label meaning than flat prompt-only labeling schemes (Shaeri et al., 16 Jul 2025).

Subcategory-level appendix results illustrate the behavior of this semantic layer. In Bread, Sliced Bread had a CLIP score of 0.782 and 91.2% categorization accuracy, Whole Loaves 0.756 and 87.8%, and Rolls and Buns 0.734 and 85.3%. In Dairy, Milk and Liquid Dairy had 0.812 and 93.5%, Cheese 0.789 and 89.7%, and Yogurt and Cream 0.723 and 82.1%. These numbers suggest that visually distinctive and semantically coherent subcategories are easier for CLIP-based semantic matching than more ambiguous ones (Shaeri et al., 16 Jul 2025).

4. Preprocessing, filtering, and human review

Food-MNIST was constructed from 5,000 images retrieved from the Kaggle Food-11 dataset. Ambiguous images containing multiple food types in one photo—such as eggs and fries on the same plate—were removed to avoid label conflicts. Each image was then processed into $F: \mathcal{I} \to \mathcal{M},$1 grayscale format, and the preprocessing chain was described explicitly through resizing, center cropping, grayscale conversion, and binarization. The appendix additionally states that grayscale conversion can use weighted RGB averaging $F: \mathcal{I} \to \mathcal{M},$2, binarization can use Otsu’s method or an RL-learned threshold, and normalization scales values to $F: \mathcal{I} \to \mathcal{M},$3 with $F: \mathcal{I} \to \mathcal{M},$4 and $F: \mathcal{I} \to \mathcal{M},$5. Background removal with $F: \mathcal{I} \to \mathcal{M},$6-Net is part of the general pipeline, although the Food-MNIST description emphasizes the $F: \mathcal{I} \to \mathcal{M},$7 grayscale output and semantic filtering more than any single preprocessing stage. The final dataset is stored in NumPy or IDX format with hierarchical label metadata, making it compatible with standard PyTorch and TensorFlow workflows (Shaeri et al., 16 Jul 2025).

Reinforcement learning adds an adaptive sample-selection stage. Each image is treated as a state $F: \mathcal{I} \to \mathcal{M},$8 with features including semantic confidence, class balance, redundancy, and similarity to already selected samples. The agent can keep the sample, discard it, or send it to human review. The reward is defined as

$F: \mathcal{I} \to \mathcal{M},$9

with appendix weights $$F(\text{id}_X)=\text{id}_{F(X)}, \qquad F(f\circ g)=F(f)\circ F(g).$$0, $$F(\text{id}_X)=\text{id}_{F(X)}, \qquad F(f\circ g)=F(f)\circ F(g).$$1, $$F(\text{id}_X)=\text{id}_{F(X)}, \qquad F(f\circ g)=F(f)\circ F(g).$$2, and $$F(\text{id}_X)=\text{id}_{F(X)}, \qquad F(f\circ g)=F(f)\circ F(g).$$3. The RL agent is implemented as a DQN with a 3-layer MLP of 256, 128, and 64 hidden units, Adam optimization, epsilon-greedy exploration, a replay buffer of 10,000 samples, batch size 32, and target-network updates every 100 steps (Shaeri et al., 16 Jul 2025).

The paper evaluates three processing modes for Food-MNIST construction: Individual Review, Smart Batch, and Fast Batch. Individual Review presents AI predictions and confidence scores image by image. Smart Batch automatically categorizes highly confident samples, routes medium-confidence samples to humans, and discards low-confidence samples. Fast Batch clusters similar images and allows users to label clusters rather than individual images.

Mode	Time	Accuracy
Manual annotation	25.0 hours	92.3%
Individual Review	18.5 hours	91.1%
Smart Batch	5.2 hours	89.7%
Fast Batch	3.1 hours	86.4%

On the 5,000-image Food-MNIST workload, Smart Batch achieved 85% automatic categorization while reducing annotation time by about 79% relative to manual processing. The paper’s broader efficiency claim is that MNIST-Gen can provide around 80% time savings while maintaining high enough annotation quality for practical dataset generation. This suggests that Food-MNIST is also an experiment in annotation-policy design, not only a benchmark release (Shaeri et al., 16 Jul 2025).

5. Benchmark behavior and downstream performance

Food-MNIST is evaluated not only as a dataset artifact but also through downstream classification. The paper compares three dataset versions: randomly sampled images, semantically filtered images using CLIP, and images filtered using both hierarchical semantic analysis and DQN reinforcement learning. The RL + semantic version improved downstream classification accuracy by about 5–8% over the simpler versions, while also improving class-distribution entropy and lowering redundancy. This is presented as evidence that the quality of the dataset-generation pipeline measurably affects benchmark quality (Shaeri et al., 16 Jul 2025).

For downstream evaluation, the paper trains several classical and neural models. The CNN used for Food-MNIST has two $$F(\text{id}_X)=\text{id}_{F(X)}, \qquad F(f\circ g)=F(f)\circ F(g).$$4 convolutional layers with 16 and 32 filters respectively, max pooling after each, a 64-unit fully connected layer, and a softmax output over $$F(\text{id}_X)=\text{id}_{F(X)}, \qquad F(f\circ g)=F(f)\circ F(g).$$5 classes. It is trained with Adam at learning rate 0.001, batch size 32, for 10 epochs, with an 80/20 train-validation split. On Food-MNIST, the CNN achieved the best reported performance at 73.84% accuracy, with precision 0.7521, recall 0.7384, and F1 0.7441 (Shaeri et al., 16 Jul 2025).

Model	Accuracy
CNN	73.84%
Random Forest	68.92%
Two-layer fully connected model	67.45%
Gradient Boosting	64.73%
KNN	62.18%
AdaBoost	58.35%
Decision Tree	56.92%
Shallow MLP	54.67%
SVM	51.23%
Logistic Regression	49.86%
Gaussian Naive Bayes	42.35%

The paper interprets the CNN result as evidence that convolutional models are best at capturing the local patterns and spatial arrangements needed for food images, while classical methods still benefit from the dataset’s structured feature content. It also states that Food-MNIST is harder than Tree-MNIST, which is reflected in lower scores across all methods and is explicitly attributed to greater inter-class similarity among food categories (Shaeri et al., 16 Jul 2025).

Appendix-level error analysis further characterizes benchmark quality. Semantic filtering substantially reduces wrong-object, poor-quality, context-mismatch, ambiguous-category, and scale-related errors; total error rate drops from 80.0% before filtering to 30.7% after filtering, a 61.6% improvement. This suggests that in Food-MNIST, label-space engineering and sample filtering are integral to benchmark construction rather than merely ancillary preprocessing (Shaeri et al., 16 Jul 2025).

6. Relation to broader food-recognition research and methodological significance

Food-MNIST occupies a recognizable position within food-image recognition research. An earlier study constructed a small-scale dataset of 5,822 color food images drawn from ImageNet across 10 classes—apple, banana, broccoli, burger, egg, french fry, hotdog, pizza, rice, and strawberry—and evaluated a bag-of-features plus linear SVM baseline against a custom five-layer CNN. All images were down-sampled to $$F(\text{id}_X)=\text{id}_{F(X)}, \qquad F(f\circ g)=F(f)\circ F(g).$$6 and randomly split into training and test with a 4:1 ratio, producing 4,654 training and 1,168 test images. That setup was described as conceptually similar to later food benchmarks such as Food-MNIST because it involved a multi-class food recognition task with modest scale, class imbalance, and image variability (Lu, 2016).

In that earlier benchmark-style study, the BoF + SVM pipeline achieved 68% training accuracy and 56% test accuracy overall, while the CNN trained on raw data reached a best test accuracy of 74% with 95% training accuracy, indicating overfitting. After geometric data augmentation based on rotation, translation, and scaling, CNN test accuracy improved to about 87% within 100 epochs and then to more than 90% when trained longer; the best reported model, trained for 400 epochs with augmented data, achieved over 90% test accuracy. The study concluded that small food recognition datasets benefit strongly from invariance-inducing preprocessing and augmentation, and that a simple CNN can outperform a BoF + SVM baseline by a large margin (Lu, 2016).

A separate line of work on self-supervised learning reinforces the claim that food imagery is intrinsically difficult. On Food-101, a benchmark with 101 classes, 1,000 images per class, and 101,000 images total, self-supervised methods were evaluated by linear classification after 100 epochs of pretraining. DINO achieved 61.4% top-1 linear evaluation accuracy, followed by SwAV at 54.7%, MoCo v2 at 53.9%, SimCLR at 51.0%, BYOL at 47.7%, and SimSiam at 44.5%. That study explicitly argued that food images are harder than generic object images because of high inter-class similarity, high intra-class variance, fewer total images than ImageNet, and noisier examples. It also showed that batch size strongly affects performance, with SimCLR improving from 41.8% at batch 64 to 48.0% at batch 128 and 51.0% at batch 256 (Peng et al., 2023).

These related results help situate Food-MNIST. A plausible implication is that the benchmark’s low-resolution grayscale format reduces superficial complexity while preserving a problem class that remains nontrivial because food categories are semantically overlapping and visually ambiguous. At the same time, the Food-MNIST paper states clear limitations: the grayscale format loses color and fine detail, keyword search can still introduce semantic noise, and some ambiguity is unavoidable, especially for classes like pizza, sushi, or mixed plates. Human review remains important for edge cases, and because the dataset is built from public image sources rather than a curated ground-truth corpus, performance evaluation relies on downstream accuracy and semantic consistency instead of absolute labeling certainty (Shaeri et al., 16 Jul 2025).