---
title: 'AgMMU Benchmark: Multimodal Agriculture QA'
url: https://www.emergentmind.com/topics/agmmu-benchmark
type: topic
---

# AgMMU Benchmark: Multimodal Agriculture QA

Searching arXiv for the cited AgMMU papers to ground the article with current paper metadata.
{"query":"2504.10568 AgMMU benchmark 2509.24350", "max_results": 5}
{"query":"AgMMU: A Comprehensive Agricultural Multimodal Understanding and Reasoning Benchmark", "max_results": 10}
AgMMU, short for Agricultural Multimodal Understanding and Reasoning Benchmark, is a large-scale multimodal benchmark for evaluating and developing vision-language models (VLMs) in agriculture, with an explicit focus on factually accurate answering in a knowledge-intensive expert domain. It is designed around tasks that require connecting detailed visual observation with precise agricultural knowledge and reasoning, such as pest identification, disease diagnosis, management instructions, species recognition, and symptom description. The benchmark is curated from 116,231 conversations between real-world users and authorized agricultural experts, and comprises an evaluation set of 5,460 multiple-choice questions (MCQs) and open-ended questions (OEQs), together with a development set containing 205,399 pieces of agricultural knowledge information [2504.10568].

## 1. Scope, composition, and domain coverage

AgMMU is organized around two principal components: an evaluation benchmark and a development corpus. The evaluation set contains 5,460 questions in both MCQ and OEQ formats, paired with real-world user images and context. The development set, referred to as “AgBase-200K,” contains 205,399 facts and is non-overlapping with the evaluation set. The benchmark covers five core agricultural knowledge types: disease/issue identification, insect/pest identification, species recognition, management instruction, and symptom/visual description [2504.10568].

| Component | Contents | Role |
|---|---|---|
| Evaluation set | 5,460 MCQs and OEQs | Benchmarking |
| Development set (“AgBase-200K”) | 205,399 facts | Fine-tuning |
| Image base | 12,481 unique images | Multimodal grounding |

The underlying data are real-world rather than studio-curated. Images are user-submitted, often of varying quality and number, and multi-image scenarios are common. The source material spans 2013–2024 and reflects practical farm, garden, turf, and pest contexts rather than standardized photographic conditions. This design places AgMMU closer to operational agricultural support settings than to conventional image classification corpora [2504.10568].

A central property of the benchmark is that its facts, questions, and answers are extracted from real-world user–expert exchanges. This distinguishes AgMMU from synthetic or crowd-authored QA resources and suggests a stronger coupling between visual evidence, domain-specific language, and actionable expert knowledge.

## 2. Data source and curation pipeline

The primary source of AgMMU is 116,231 real-world user–expert conversations from platforms such as AskExtension.org, involving cooperative extension or university experts and public users throughout the United States over 11 years. Personally identifying information is stripped before inclusion, no human faces are retained in images, and the shared dataset is described as de-identified and ethically compliant [2504.10568].

Dataset construction proceeds through a three-step curation pipeline using LLaMA-70B, LLaMA-405B, GPT-4o, and human verification. In the first step, categorization, LLaMA-70B tags each conversation with an agriculture sub-domain and the main organism of interest. The seven main sub-domains are disease advice, weed management, pest control, growing advice, environmental stress, nutrient deficiency, generic identification, and other; only image-relevant sub-domains are retained.

In the second step, knowledge extraction, LLaMA-405B uses domain-specific prompt templates with in-context, manually annotated examples to extract atomic pieces of relevant agricultural knowledge into 5–6 categories, including species identification, disease identification, symptom/visual description, management instructions, and insect/pest identification. The process includes heavy filtering for relevance and avoidance of hallucination.

In the third step, QA generation, GPT-4o converts extracted facts into MCQs and OEQs, after which human verification is applied. For MCQs, the prompting procedure generates three wrong answers while anchoring the correct answer in the extracted fact, and can incorporate background information and time or location as required. For OEQs, the resulting questions are direct and knowledge-intensive, requiring concise and factual answers. The evaluation set is created through balanced sampling across sub-domains and knowledge types, while the remaining material is assigned to the development set [2504.10568].

The benchmark’s final distribution is explicitly balanced. The raw data are described as highly imbalanced, dominated by some sub-domains, whereas the final AgMMU benchmark enforces equitable representation across sub-domains and knowledge tasks. A plausible implication is that reported performance is intended to reflect model competence across agricultural task types rather than overfitting to the most common real-world query categories.

## 3. Task design and evaluation protocol

AgMMU supports both MCQ and OEQ evaluation. The MCQ setting is similar to benchmarks such as MMMU, requiring selection of a correct answer from several options. The OEQ setting is inspired by SimpleQA and requires models to generate precise, knowledge-intensive answers without answer choices. Because both modalities share the same underlying agricultural knowledge types, the benchmark separates answer recognition from answer generation and thereby probes different failure modes in VLMs [2504.10568].

MCQs are scored by accuracy after format preprocessing. OEQs are evaluated with an “LLM-as-judge” approach in which Qwen-7B grades answers using the rubric “correct,” “incorrect,” “partially correct,” and “not attempted.” Short-form OEQs use direct n-gram or phrase matching. Long multi-statement OEQs, such as management recommendations, are graded by splitting answers into unique statements and judging relevance and correctness per statement. The final score is the harmonic mean of correctness across all aspects, following SimpleQA.

The benchmark also incorporates a design decision about MCQ option count. Supplementary experiments report that, for GPT-4o, moving from 4 choices to 5 choices reduces accuracy by 5% to 7%, and moving to 6 choices reduces accuracy by up to 10%. Four-option MCQs were therefore selected for the main benchmark. This indicates that the MCQ setting is not treated as a trivial proxy for knowledge recall; option construction materially changes measured difficulty.

Reproducibility is supported through the use of public APIs or models for closed-source systems such as GPT-4o, Gemini, and Claude 3, and official open-source checkpoints for LLaMA, LLaVA, Cambrian, InternVL, Qwen, and VILA. This makes AgMMU simultaneously a benchmark for model comparison and a development suite for continued training and domain adaptation [2504.10568].

## 4. Position relative to related benchmarks

AgMMU is positioned against four representative datasets: iNat21, TreeOfLife, SimpleQA, and MMMU. In the comparison provided in the source paper, iNat21 and TreeOfLife are multimodal and expert-oriented but are classification datasets rather than QA benchmarks; SimpleQA emphasizes factuality and OEQ but is not multimodal and has no training set; MMMU is multimodal and MCQ-based but is neither expert-domain-specific nor factuality-centered. AgMMU combines MCQ and OEQ, is multimodal, provides both evaluation and training data, targets an expert domain, and emphasizes factuality [2504.10568].

This comparison clarifies a common misreading of agricultural multimodal evaluation. AgMMU is not merely a species recognition benchmark with agricultural images. Its task space includes disease and issue identification, insect and pest identification, management instruction, and symptom or visual description, in addition to species recognition. It therefore spans recognition, diagnosis, and recommendation.

The benchmark is described as the first to combine knowledge-intensive vision-language MCQ and OEQ in agriculture with both evaluation and fine-tuning sets and with strong factuality and expert-domain emphasis. This suggests that AgMMU is intended not only to rank models, but also to study how multimodal systems incorporate specialized factual knowledge under realistic visual variability.

## 5. Empirical findings and diagnostic value

AgMMU is explicitly reported to be challenging for existing VLMs. On OEQs, proprietary models remain low-scoring in absolute terms: GPT-4o reaches an average of 17.79, Gemini 1.5 Pro 21.98, and Claude 3 Haiku 13.26. On MCQs, the corresponding averages are higher: GPT-4o 85.25, Gemini 1.5 Pro 80.42, and Claude 3 Haiku 62.00. Open-source models score still lower, with Cambrian-8B at 9.92 OEQ and 65.81 MCQ, LLaMA-3.2 at 15.91 OEQ and 73.32 MCQ, and InternVL2-8B at 7.27 OEQ and 60.17 MCQ [2504.10568].

| Model | OEQ average | MCQ average |
|---|---:|---:|
| GPT-4o | 17.79 | 85.25 |
| Gemini 1.5 Pro | 21.98 | 80.42 |
| Claude 3 Haiku | 13.26 | 62.00 |
| LLaMA-3.2 | 15.91 | 73.32 |
| Cambrian-8B | 9.92 | 65.81 |

Several benchmark-level findings are highlighted. First, there is a large gap between MCQs and OEQs: models can use elimination and answer-choice constraints in MCQ settings, but OEQs more directly measure factual knowledge production. Second, open-source VLMs still demonstrate a substantial performance gap compared to proprietary models. Third, disease or pest identification and species recognition are reported as especially difficult, particularly for OEQs. Symptom descriptions and management are somewhat easier in MCQ form, but remain difficult in OEQ form.

The error analysis further identifies lack of knowledge as the main failure reason at 56.5%, followed by perceptual errors at 18.5%; some failures are occasionally fixed by chain-of-thought prompting at 22.8%. This supports the benchmark’s framing as a test of both visual grounding and factual knowledge access. Fine-tuning on AgBase-200K is reported to improve LLaVA-1.5 evaluation accuracy by up to 3.1%, indicating that the development set is operationally useful for domain adaptation rather than merely descriptive [2504.10568].

## 6. Adoption, multi-image evaluation, and broader implications

AgMMU has already been used as an evaluation substrate in subsequent work on agricultural visual question answering. “Dynamic Orchestration of Multi-Agent System for Real-World Multi-Image Agricultural VQA” evaluates a multi-agent framework on the public AgMMU multiple-choice subset and characterizes the benchmark as a realistic multi-image agricultural VQA resource derived from 116,000 user–expert conversations. In that study, each question is associated with one or more images, specifically 1–10 per question in the public subset, together with contextual information such as time and location; the subset used for experiments contains 1,094 images, with image-count distribution of 38% for one image, 28% for two images, 30% for three images, and 4% for four or more images [2509.24350].

That downstream study also illustrates the benchmark’s diagnostic utility for multi-image reasoning. Using accuracy as the evaluation metric, the reported average performance on the public subset is 88.72 for GPT-4o, 88.23 for Gemini-1.5-Pro, 76.03 for Claude-3-Haiku, 81.32 for Qwen2.5-VL-7B, 69.77 for LLaVA-1.5-7B, 56.57 for InternVL2-8B, and 90.78 for the proposed multi-agent system. The accompanying qualitative analysis attributes gains to better integration of complementary evidence from multiple images and to iterative retrieval and answer refinement [2509.24350].

The intended uses stated for AgMMU include robust evaluation and diagnostics for VLMs in agriculture and allied expert domains, domain adaptation for real-world agricultural decision support, and broader research on bridging the gap between image understanding and knowledge grounding. The source paper also presents the benchmark in terms of social good, including support for timely automated farmer or gardener assistance, improved resource use, sustainability, and food security. Its pipeline is described as potentially applicable to other specialist domains such as medicine and biology, although that extension remains prospective rather than part of the benchmark itself [2504.10568].

AgMMU’s limitations are framed primarily through task difficulty rather than annotation validity. The dataset is restricted to agriculture, and the paper states that no societal biases are present because questions are restricted to that domain. This suggests that the principal research challenge is not demographic fairness in the conventional sense, but reliable multimodal grounding and factual expert reasoning under realistic image quality, multi-image evidence, and open-ended answer generation.

Source: https://www.emergentmind.com/topics/agmmu-benchmark