MM-Food-100K: Multimodal Food Dataset

• MM-Food-100K is a multimodal food intelligence dataset combining real-world images with detailed, evidence-linked JSON annotations for diverse food analysis tasks.
• It features a progressive L1-L5 annotation schema and wallet-linked provenance to ensure traceability and quality control, addressing scale and diversity limitations.
• Benchmark results indicate that fine-tuning vision-language models with this dataset enhances both classification and regression tasks, especially for homemade and street-food items.

MM-Food-100K is a public 100,000-sample multimodal food intelligence dataset with verifiable provenance, introduced as the approximately 10% open, research-focused subset of an original approximately 1.2 million quality-accepted corpus. It pairs real-world food photos with structured annotations intended for fine-tuning and benchmarking vision-LLMs on practical food tasks, including dish recognition, ingredient extraction, portion and calorie estimation, cooking-method reasoning, and authenticity checks. The dataset is explicitly positioned against recurring limitations in earlier public food resources: insufficient diversity and scale, shallow or monolithic labels, and curated imagery that differs from real-world capture conditions (Dong et al., 14 Aug 2025).

1. Scope, motivation, and design objectives

MM-Food-100K is defined as a multimodal dataset in which each record couples an image link with a JSON metadata block. Its stated purpose is fine-grained food intelligence rather than single-task classification, and its annotation design therefore spans identity, ingredients, portion, nutrition, preparation, and authenticity. This multi-axis structure is intended to support both supervised fine-tuning and benchmark construction for large vision-LLMs operating on practical food analysis tasks (Dong et al., 14 Aug 2025).

The dataset is also framed as a provenance-aware data resource. Collection was conducted over six weeks from more than 87,000 contributors via a Binance Wallet × Codatta campaign, and each submission is tied to a wallet address for attribution and traceability. The open 100,000-sample release is therefore not merely a sample in the statistical sense; it is a release boundary within a larger data-governance and commercialization framework that separates a free research subset from a retained commercial subset with prospective royalty sharing.

A central design goal is the combination of real-world imagery with evidence-linked annotations and verifiable provenance. In the terminology of the paper, evidence links may include label photos, menu screenshots, URLs, and receipts, particularly for nutrition claims. This suggests that MM-Food-100K should be understood not only as a food-recognition corpus but also as an attempt to formalize evidentiary support and contributor attribution within a dataset release.

2. Record structure, schema, and taxonomy

Each MM-Food-100K sample contains an image link and a JSON metadata block with core fields such as dish_name, food_type, ingredients, portion_size, nutritional_profile, cooking_method, and authenticity indicators including camera_or_phone_prob, online_download_prob, and food_prob. The food_type field uses values such as homemade, restaurant, packaged, raw, and other; ingredients[] is multi-label; and portion_size and calories are regression targets. Mixed annotation sources are permitted, so fields may be human-labeled and/or AI-predicted, with evidence-linked entries included where available (Dong et al., 14 Aug 2025).

The schema is organized through a progressive L1-L5 taxonomy intended to support staged enrichment. L1 comprises image plus name; L2 adds nutrition with evidence; L3 adds visible ingredients; L4 adds portion and geometry cues; and L5 adds per-ingredient localization, enabling detection or segmentation. This layered design makes explicit that annotation depth is not uniform across all fields and that the corpus supports incremental expansion from coarse recognition to fine-grained reasoning.

Food type	Count
Homemade	46,555
Restaurant	35,461
Raw vegetables and fruits	9,357
Packaged	8,354
Other	273

The reported open-subset snapshot shows that homemade and restaurant images dominate the release. Authenticity statistics are reported through example camera_or_phone_prob bins: 0.9: 200, 0.85: 161, 0.8: 47,879, 0.7: 51,629, and 0.6: 131. The paper emphasizes that these high-probability bins total approximately 99,000 images, indicating a strong skew toward user-captured photos. Exact global label distributions beyond food_type and authenticity are not reported (Dong et al., 14 Aug 2025).

3. Data sources, provenance, and quality assurance

The collection design distinguishes among packaged food, chain restaurant, and homemade/street sources. Packaged items are associated with supermarket packs or barcode pages and nutrition labels, described as high verifiability. Chain-restaurant items draw on brand websites or menus and menu nutrition, described as medium verifiability. Homemade and street-food items are associated with personal photos and estimated nutrition with portion cues, described as having lower external ground truth. Contributors were invited with zero upfront payment, while attribution and prospective royalty sharing were emphasized for the commercial portion of the corpus (Dong et al., 14 Aug 2025).

The provenance system is wallet-linked and ledger-backed. Each submission is tied to a contributor wallet address, and a blockchain-anchored contribution ledger records sourcing, annotation, verification, and adoption events as immutable, hashed attestations tied to wallets and timestamps. Compact cryptographic commitments to event batches are periodically anchored to a public chain, while payloads such as images, evidence, and review notes are stored in a hybrid encrypted, access-controlled architecture. The paper describes a roadmap toward an end-state with contract-mediated revenue pools, automated royalty distribution, and community governance, but states that the current release uses verifiable off-chain records with public attestations.

Quality control follows a two-stage, AI-augmented workflow. Contributors first submit image and annotations under the L1-L5 schema. An automated LVM review then performs rapid vetting with near-real-time feedback and a configurable rejection/resubmission loop. A final programmed LVM review conducts more detailed checks for accuracy and integrity. Accepted data are deduplicated and divided into public and commercial subsets. Additional QA measures include perceptual hashing for near-duplicate removal, OCR validation, outlier flags, consensus sampling, quick human checks for packaged and chain items, and human-led review assisted by models for homemade and street-food items. Inter-annotator agreement, explicit noise rates, and detailed validation statistics are not reported (Dong et al., 14 Aug 2025).

4. Experimental protocol and benchmark results

The paper validates dataset utility through fine-tuning experiments on image-based nutrition prediction and through textual prediction tasks for dish name, ingredients, and cooking method. Experiments use an 80/10/10 train/validation/test split stratified by cuisine and source type, with near-duplicates removed via perceptual hashing and a frozen release index used to fix schema and check-logic versions for reproducibility. In the detailed benchmark section, the reported models are GPT-4o and Qwen-Max, each evaluated in base and supervised fine-tuned variants (Dong et al., 14 Aug 2025).

For kilocalorie regression, evaluation uses Mean Absolute Error, Root Mean Square Error, and the coefficient of determination:

$$\mathrm{MAE} = \frac{1}{n}\sum_{i=1}^{n}\left|y_i-\hat{y}_i\right|$$

$$\mathrm{RMSE} = \sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_i-\hat{y}_i)^2}$$

$$R^2 = 1 - \frac{\sum_{i=1}^{n}(y_i-\hat{y}_i)^2}{\sum_{i=1}^{n}(y_i-\bar{y})^2}$$

The fine-tuning setup is described as a simple regression head on a shared VLM backbone, trained with a Huber/MSE objective, mixed precision, and fixed augmentations, with early stopping based on validation MAE. Seeds, batch sizes, and learning rates were held constant across runs, but exact values are not provided. For categorical outputs, evaluation is a text-similarity-based win rate; the exact similarity measure and judge procedure are not detailed.

Model	Kilocalorie regression	Dish / Ingredients / Cooking win rate
Qwen-Max (Base)	MAE 126.5; RMSE 185.3; $R^2$ 0.521	42.1% / 39.8% / 41.3%
Qwen-Max (SFT)	MAE 104.2; RMSE 154.5; $R^2$ 0.638	57.9% / 60.2% / 58.7%
GPT-4o (Base)	MAE 98.7; RMSE 148.1; $R^2$ 0.685	48.9% / 48.6% / 48.8%
GPT-4o (SFT)	MAE 95.8; RMSE 144.3; $R^2$ 0.706	51.1% / 51.4% / 51.2%

The paper’s stated observation is that fine-tuning on MM-Food-100K improves both categorical and regression performance. Qwen-Max exhibits larger relative gains, including MAE $-17.6\%$, RMSE $-16.6\%$, and $R^2$ $$\mathrm{RMSE} = \sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_i-\hat{y}_i)^2}$$0, while GPT-4o shows smaller but consistent improvements, with MAE approximately $$\mathrm{RMSE} = \sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_i-\hat{y}_i)^2}$$1, RMSE approximately $$\mathrm{RMSE} = \sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_i-\hat{y}_i)^2}$$2, and $$\mathrm{RMSE} = \sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_i-\hat{y}_i)^2}$$3 approximately $$\mathrm{RMSE} = \sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_i-\hat{y}_i)^2}$$4. The largest relative gains are reported for homemade and street-food items, which the paper associates with structured portion cues at L4 and evidence-backed nutrition at L2. Statistical significance testing and ablations are not reported (Dong et al., 14 Aug 2025).

5. Position within multimodal food-dataset research

MM-Food-100K is positioned against earlier public food datasets through qualitative rather than numerical comparison. The paper argues that prior public resources often suffer from limited scale and diversity, shallow labels, and curated imagery unlike real-world photos. MM-Food-100K responds by combining image data with structured JSON, progressive schema depth from L1 to L5, evidence-linked annotations for verifiable claims, authenticity indicators favoring user-captured images, wallet-linked provenance, and a dual-access release model in which the open 10% subset coexists with a retained 90% commercial subset subject to contributor royalties (Dong et al., 14 Aug 2025).

A useful point of contrast is Uni-Food, introduced in the RoDE paper. Uni-Food is a different 100,000-sample dataset whose official name is “Uni-Food”; the name “MM-Food-100K” does not appear in that paper. Uni-Food provides category labels, ingredient lists with quantities, recipes, and nutrition for all 100,000 samples, with nutrition generated using ChatGPT-4-vision and cross-referenced against USDA for test-set quality control. By contrast, MM-Food-100K is defined by image-plus-JSON records, staged schema enrichment, evidence-linked annotations, authenticity indicators, and a provenance architecture built around wallet linkage and blockchain-anchored attestations (Jiao et al., 2024).

This distinction matters because the two datasets address overlapping research space with different design emphases. Uni-Food is presented as a unified multi-task dataset for food large multimodal models, whereas MM-Food-100K foregrounds provenance, attribution, and evidence linkage alongside task coverage. A common misconception is therefore to collapse them into a single resource because both operate at 100,000-sample scale; the cited papers treat them as separate datasets with separate naming, construction pipelines, and evaluation goals.

6. Access, limitations, ethics, applications, and planned extensions

MM-Food-100K is released for public research use, with a public dataset card and README hosted at https://huggingface.co/datasets/Codatta/MM-Food-100K. The paper does not state specific license terms and directs readers to the dataset card for licensing details. The release model is economically asymmetric by design: the open 10% subset is free for research, while the retained 90% commercial subset is licensable and linked to royalty distribution through wallet-linked provenance and quality/usage-weighted splits (Dong et al., 14 Aug 2025).

The paper acknowledges several limitations. Under-represented cuisines and domain imbalance are noted, but precise regional distributions are not reported. Homemade and street-food items have higher variance and lower external ground truth, which increases uncertainty in portion and calorie labels. Menu drift and seasonal changes can affect chain-restaurant validity. Field-level fill rates vary, and missing annotations can constrain downstream tasks. Image resolution ranges, inter-annotator agreement statistics, explicit noise rates, detailed error analyses, and exact optimization hyperparameters are not specified.

Ethical and governance claims center on attribution, privacy, and selective disclosure. Contributor records are wallet-linked, the ledger is described as off-chain and privacy-preserving with periodic on-chain attestations, payload storage is encrypted and access-controlled, and selective disclosure is envisioned for audits. The paper presents this arrangement as preserving both public research utility and contributor economic value.

The stated application space includes diet tracking and logging, image-based calorie estimation, recipe recommendation, cooking-method reasoning, health monitoring, nutritional advisory systems, cultural food studies, and menu analytics. Planned extensions include maturation of the on-chain protocol, expanded benchmarks for dish and cuisine classification, ingredient extraction and portion estimation with task-specific heads, scaling studies across additional model families and training sizes, per-domain reporting for packaged, chain, and homemade/street subsets, schema-depth ablations, QA-score weighting, and weekly instrumentation to tune gates and prompts for improved cost-quality trade-offs (Dong et al., 14 Aug 2025).