January Food Benchmark (JFB)

• January Food Benchmark (JFB) is a public dataset and evaluation suite for multimodal food analysis, designed to standardize meal name recognition, ingredient extraction, and macronutrient estimation.
• It compiles 1,000 real-world food images with detailed, human-validated annotations that reflect complex meal compositions and operational challenges in mobile food logging.
• The benchmark evaluates models on meal-name similarity, ingredient accuracy, macronutrient estimation, latency, and cost, combining these into a composite Overall Score for balanced performance.

Searching arXiv for the named benchmark and closely related food-analysis benchmark papers. arXiv search query: "January Food Benchmark JFB multimodal food analysis" January Food Benchmark (JFB) is a public benchmark dataset and evaluation suite for multimodal food analysis, introduced to address two bottlenecks in automated nutritional analysis: the lack of standardized evaluation methodologies and the lack of high-quality, real-world, human-validated benchmark data. JFB is organized around a practical food-logging task in which a model takes a meal photo and is evaluated on meal-name understanding, ingredient extraction, and macronutrient estimation, while also being assessed on latency and cost. The released benchmark contains 1,000 food images with human-validated annotations for meal name, ingredient list, ingredient quantities, and macronutrients, and its evaluation framework aggregates these dimensions into a single Overall Score on a 0–100 scale (Hosseinian et al., 13 Aug 2025).

1. Problem setting and benchmark identity

JFB targets the practical setting faced by food-logging and nutritional-analysis systems: given a mobile photograph of a meal, a model should identify what the meal is, which ingredients are present, and what its macronutrient content is. The benchmark is explicitly motivated by the claim that this problem is substantially harder than conventional food recognition, because fine-grained food analysis involves visually similar dishes, hidden or occluded ingredients, uncertain cooking methods, and non-visual quantities such as portion size (Hosseinian et al., 13 Aug 2025).

The benchmark is framed as a response to two specific deficiencies in the existing evaluation ecosystem: there had been no standardized metrics for end-to-end food analysis, and no reliable public benchmark with sufficiently high-quality ground truth for nuanced meal understanding. JFB therefore contributes a public dataset of 1,000 food images, human-validated labels for meal names, ingredients, quantities, and macronutrients, and a benchmarking framework covering semantic meal recognition, ingredient extraction, nutrient estimation, latency, cost, and a composite overall score (Hosseinian et al., 13 Aug 2025).

Its intended applications are not restricted to academic recognition tasks. The benchmark is presented as relevant to dietary tracking, nutrition coaching, and health applications. That emphasis matters because JFB does not treat food understanding as a closed-set image classification problem; it treats it as meal-level analysis under realistic deployment constraints.

2. Dataset construction and annotation protocol

JFB contains 1,000 real-world food images sourced from users of a mobile health application who had consented to research use. The collection process exploited the app workflow itself: users interacted with a baseline AI system identified as GPT-4o and could accept or correct its suggested food log. Sampling was then stratified over a “Liked Cohort,” in which the AI output was accepted, and a “Disliked Cohort,” in which the AI output was corrected. This ensured inclusion of both relatively easy examples and cases where the baseline model had failed (Hosseinian et al., 13 Aug 2025).

Two additional curation filters were applied. First, only meals with more than two ingredients were retained, so the benchmark emphasizes complex dishes rather than trivial single-item examples. Second, the entire dataset underwent expert human review, and a human annotator validated and corrected all labels. The final annotation set includes meal name, ingredient list, ingredient quantities, and macronutrients. The benchmarked nutrient targets are calories, carbohydrates, protein, and fat (Hosseinian et al., 13 Aug 2025).

The benchmark repeatedly stresses that it is human validated under a rigorous protocol and strict annotation protocol. Operationally, the process was: initial meal data from user-logged app data and AI-generated logs; users’ accept/correct behavior as a curation signal; then human validation and correction of all annotations. The paper does not report inter-annotator agreement, number of annotators per image, adjudication details, or a formal annotation handbook, but it treats the human validation step as the central reason JFB should be regarded as a trustworthy benchmark (Hosseinian et al., 13 Aug 2025).

The images are explicitly described as real-life examples logged by users in day-to-day environments, with varied lighting, varied camera angles, and complex backgrounds. This realism is core to the benchmark’s rationale. JFB is not built from curated web imagery or controlled studio captures; it is meant to reflect the operational regime of mobile food logging (Hosseinian et al., 13 Aug 2025).

The dataset size is 1,000 images. The authors justify that size on four grounds: the labels are high quality due to human validation, the benchmark focuses on complex meal-level analysis, the real-world images increase variance and difficulty, and a power analysis suggested that 1,000 images is sufficient to detect meaningful model differences with statistical power greater than 0.80. The reported cuisine distribution includes American at 31.7% and Other or unclassified or mixed cuisine at 29.8% (Hosseinian et al., 13 Aug 2025).

The paper also states that the dataset was collected with informed consent under January AI’s terms and privacy policy, and that personally identifying information, faces, and EXIF metadata were removed (Hosseinian et al., 13 Aug 2025).

3. Label structure and benchmark protocol

JFB is not framed as a closed-set classification dataset with a finite class vocabulary. Its outputs are free-form semantic predictions: a meal-name string, a list of ingredient strings, and four numeric macronutrient values. This design decision explains why the evaluation relies on semantic similarity and assignment-based matching rather than exact lexical match or class-wise accuracy (Hosseinian et al., 13 Aug 2025).

The practical protocol is an evaluation-only benchmark rather than a conventional supervised benchmark with public train, validation, and test splits. The main experiments evaluate all models on the full 1,000-image JFB dataset. A separate held-out validation set of 100 images is used for metric-design choices, specifically for selecting the text embedding model used in meal-name similarity and for tuning the ingredient-matching threshold $T = 0.75$. The public release includes images and JSON annotations, metric scripts, and prompts, and the license is CC-BY-4.0 (Hosseinian et al., 13 Aug 2025).

The annotation schema can be summarized as follows:

Component	Ground truth
Meal identity	Meal name
Composition	Ingredient list and ingredient quantities
Nutrition	Calories, carbohydrates, protein, fat

This structure makes JFB closer to a meal-understanding benchmark than to a dish-classification benchmark. A plausible implication is that it can expose failure modes that are invisible under category-only evaluation, such as semantically close but nutritionally distinct meal names, partially correct ingredient lists, or accurate recognition with poor macro estimation.

4. Evaluation framework and Overall Score

JFB evaluates five dimensions: meal-name similarity, ingredient recognition, macronutrient estimation, inference latency, and inference cost. These are combined into a single Overall Score intended to reward balanced systems and penalize severe weakness on any one axis (Hosseinian et al., 13 Aug 2025).

Meal-name prediction is scored semantically rather than by exact match. The metric is cosine similarity between embeddings of the predicted and ground-truth meal names,

$\text{score} = \cos(E(\text{pred}), E(\text{gt})).$

The embedding model used is OpenAI text-embedding-3-small. On the 100-image validation set, the reported correlation with human judgments was $r = 0.82$, compared with $r = 0.71$ for BERT-base and $r = 0.76$ for Sentence-BERT, while model rankings remained stable across embedding choices with Kendall’s $\tau > 0.85$ (Hosseinian et al., 13 Aug 2025).

Ingredient evaluation uses semantic matching plus standard precision, recall, and F1. Given a ground-truth ingredient list $G$ and a predicted list $P$, the benchmark constructs a cost matrix

$C_{ij} = 1 - \cos(E(g_i), E(p_j)),$

then uses a Hungarian algorithm to find the optimal one-to-one alignment. A matched pair is counted as correct only if cosine similarity exceeds $T = 0.75$. On the 100-image validation set, this threshold gave F1 $\text{score} = \cos(E(\text{pred}), E(\text{gt})).$0, precision $\text{score} = \cos(E(\text{pred}), E(\text{gt})).$1, and recall $\text{score} = \cos(E(\text{pred}), E(\text{gt})).$2. The paper notes that lower thresholds increased false positives such as matching “butter” and “oil,” while higher thresholds missed valid semantic equivalences (Hosseinian et al., 13 Aug 2025).

Macronutrient estimation is scored with Weighted Mean Absolute Percentage Error over the four targets

$\text{score} = \cos(E(\text{pred}), E(\text{gt})).$3

using

$\text{score} = \cos(E(\text{pred}), E(\text{gt})).$4

Lower WMAPE is better. In the composite score, the benchmark uses $\text{score} = \cos(E(\text{pred}), E(\text{gt})).$5 so that all components are aligned in a higher-is-better direction (Hosseinian et al., 13 Aug 2025).

Latency and cost are also benchmarked. Latency is reported in seconds per call, and cost in USD per call, computed from provider pricing as of July 2025. For oracle Best-of-4 prompt settings, latency and cost are still reported per call, while the paper notes that per-image totals are approximately four times larger (Hosseinian et al., 13 Aug 2025).

The Overall Score is a weighted geometric mean over normalized metrics, with weight vector

$\text{score} = \cos(E(\text{pred}), E(\text{gt})).$6

corresponding to meal-name similarity, ingredient accuracy, macronutrient accuracy, cost, and latency. The weights were derived from consultation with 12 domain experts—4 nutritionists, 4 machine-learning researchers, and 4 product managers—using the SMART multi-criteria method. Sensitivity analysis with $\text{score} = \cos(E(\text{pred}), E(\text{gt})).$7 perturbations yielded stable rankings with Spearman’s $\text{score} = \cos(E(\text{pred}), E(\text{gt})).$8. The use of a geometric mean is justified as a way to penalize systems that perform well on one dimension while failing badly on another (Hosseinian et al., 13 Aug 2025).

5. Models, prompting regimes, and empirical findings

The benchmark compares four general-purpose multimodal VLM families—GPT-4o, GPT-4o-mini, Gemini 2.5 Pro, and Gemini 2.5 Flash—against a specialized model, january/food-vision-v1. The specialized model is described as proprietary and commercial, based on a Vision Transformer backbone, operating at $\text{score} = \cos(E(\text{pred}), E(\text{gt})).$9 image resolution, and trained on a private food dataset on the order of $r = 0.82$0 images with no overlap with JFB (Hosseinian et al., 13 Aug 2025).

General-purpose VLMs are tested in two zero-shot regimes. In the Avg setting, performance is averaged across four distinct zero-shot prompt variants. In the Best-of-4 (Oracle) setting, all four prompts are run and the highest-scoring output against ground truth is selected, which provides an oracle upper bound on prompt engineering but requires four calls per image. All VLMs use temperature $r = 0.82$1 and structured JSON outputs enforced via a Pydantic schema. By contrast, january/food-vision-v1 uses a single deterministic call per image with a 30-second timeout (Hosseinian et al., 13 Aug 2025).

Configuration	Key metrics	Overall Score
january/food-vision-v1	Meal 0.886; Ingredient F1 0.883; Macro WMAPE 14.2%; 10.5 s; \$0.0100	86.2
GPT-4o (Best)	Meal 0.705; Ingredient F1 0.737; Macro WMAPE 23.5%; 10.3 s; \$0.0065	74.1
GPT-4o (Avg)	Meal 0.682; Ingredient F1 0.712; Macro WMAPE 26.8%; 10.3 s; \$0.0060	70.6
GPT-4o-mini (Best)	Meal 0.665; Ingredient F1 0.667; Macro WMAPE 36.1%; 8.1 s; \$0.0058	66.4
Gemini 2.5 Pro (Best)	Meal 0.583; Ingredient F1 0.579; Macro WMAPE 28.5%; 28.1 s; \$0.0225	60.7
Gemini 2.5 Flash (Best)	Meal 0.602; Ingredient F1 0.621; Macro WMAPE 38.5%; 13.5 s; \$0.0014	60.7

The headline result is that january/food-vision-v1 achieves an Overall Score of 86.2, outperforming the strongest general-purpose baseline, GPT-4o (Best), at 74.1 by 12.1 points. The paper attributes this gain to superior performance on all core tasks, especially ingredient recognition: 0.883 F1 for january/food-vision-v1 versus 0.737 for GPT-4o (Best). The gap is reported as statistically significant under a paired $r = 0.82$2-test on per-image scores with $r = 0.82$3 and $r = 0.82$4 (Hosseinian et al., 13 Aug 2025).

The oracle prompt-selection setting improves GPT-4o from 70.6 to 74.1, a 3.5-point gain, but that setting is explicitly described as not deployable as-is without additional cost and latency. The paper also reports that january/food-vision-v1 exhibits a tighter interquartile range, indicating more consistent performance across examples (Hosseinian et al., 13 Aug 2025).

6. Relation to adjacent food benchmarks

Within its own framing, JFB is contrasted most directly with Food-101, Recipe1M+, and MEAL. Food-101 provides meal names only, is not real-world in the sense claimed for JFB, and lacks ingredient or macro labels; Recipe1M+ provides meal name and ingredients but no macronutrients; MEAL is described as closer to JFB, with meal, ingredients, and nutrition, but only partial real-world coverage and partial validation. The paper’s claim is that JFB is unique in combining meal name, ingredient list, and macronutrients with real-world mobile photos and full human validation (Hosseinian et al., 13 Aug 2025).

Placed in the broader literature, JFB is adjacent to several benchmark traditions but not reducible to any of them. The AIcrowd Food Recognition Benchmark built around MyFoodRepo-273 is an instance-segmentation benchmark with 24,119 images, 39,325 segmented polygons, and 273 classes, evaluated on private test sets using mAP and mAR at mask IoU $r = 0.82$5 (Mohanty et al., 2021). AI4Food-NutritionDB is a hierarchical nutrition-oriented recognition benchmark with 558,676 food images, 893 final food products, 73 subcategories, and 19 main categories (Romero-Tapiador et al., 2022). FoodSeg103 and FoodSeg154 focus on ingredient-level semantic segmentation, with FoodSeg103 providing 7,118 images, 103 ingredient classes, and 42,097 masks (Wu et al., 2021). SFOOD extends the benchmark space beyond RGB to spectral sensing, reporting 3,266 food categories and 2.351 million data points across 17 main food categories (Xu et al., 6 Jul 2025). CalorieBench-80K, introduced with Food-R1, contributes a calorie-centered benchmark with approximately 80,000 food images, 15,768 dishes, 3,209 unique ingredients, and Chain-of-Thought annotations for calorie reasoning (Zhu et al., 3 Jun 2026).

This suggests that JFB occupies a specific niche: meal-level, image-grounded, human-validated nutritional analysis under deployment-aware evaluation. It is broader than classification-only benchmarks, narrower than multi-task suites spanning segmentation, counting, VQA, and hyperspectral analysis, and more operational than benchmarks that ignore cost and latency.

7. Limitations, biases, and significance

The paper does not provide a standalone limitations section, but several limitations and biases are explicit or implicit. JFB has only 1,000 images, much smaller than web-scale food corpora. The cuisine distribution is not uniformly balanced, with American food at 31.7% and Other at 29.8%. All images come from users of a specific mobile health application, which likely introduces demographic, geographic, and behavioral biases tied to that user base. The benchmark excludes meals with two or fewer ingredients, which sharpens its focus on complex meal understanding but makes it less suitable for simpler food-item recognition. The human validation process is emphasized, yet the paper does not report inter-annotator agreement, disagreement rates, adjudication methodology, or whether multiple annotators reviewed each image (Hosseinian et al., 13 Aug 2025).

A further limitation concerns reproducibility. The benchmark itself is public, but the strongest model, january/food-vision-v1, is proprietary; only limited details are disclosed. In addition, the comparison with general-purpose VLMs is conducted in zero-shot settings rather than after food-domain fine-tuning, and the paper explicitly identifies fine-tuning open VLMs on comparable data as an important future direction. The benchmark is also motivated partly by concerns that general VLMs may hallucinate or behave unreliably in health-relevant settings, but JFB is not itself a full safety benchmark (Hosseinian et al., 13 Aug 2025).

Its significance lies in standardization. JFB does not merely contribute another food dataset; it specifies a meal-level evaluation protocol with human-validated labels, semantic scoring for free-form outputs, application-aware operational metrics, and a composite score that can support both research benchmarking and production-oriented model selection. Within that framework, its central empirical conclusion is that specialization matters: the domain-specific model january/food-vision-v1 achieves an Overall Score of 86.2, outperforming the best general-purpose zero-shot configuration by 12.1 points, with the largest gains concentrated in ingredient recognition and the resulting improvement in macronutrient estimation (Hosseinian et al., 13 Aug 2025).