---
title: 'AgriMM: Multimodal Benchmark for Crop Diagnosis'
url: https://www.emergentmind.com/topics/agrimm
type: topic
---

# AgriMM: Multimodal Benchmark for Crop Diagnosis

AgriMM is a large-scale, multimodal agricultural benchmark designed to support fine-grained multimodal learning for crop disease diagnosis and agricultural knowledge reasoning. Introduced together with AgriDoctor, it comprises **400000 annotated disease images**, **831 expert-curated knowledge entries**, and **300000 bilingual prompts for intent-driven tool selection**. Its role is not limited to passive evaluation: the benchmark is constructed so that different subsets directly supervise a router, classifier, detector, and knowledge retriever within a modular multimodal system for intelligent crop disease diagnosis and agricultural knowledge interaction [2509.17044].

## 1. Definition and scope

AgriMM is organized around three primary data components. The image component targets disease recognition and lesion localization; the knowledge component targets crop disease and pest reasoning; and the prompt component targets intent inference for automated module selection. This design makes AgriMM simultaneously a training resource and an evaluation substrate for multimodal agricultural agents [2509.17044].

The benchmark is explicitly bilingual in Chinese and English for the knowledge entries and prompt data. It is also explicitly task-typed: prompts are annotated for **Agricultural Knowledge Retrieval**, **Disease Classification**, or **Disease Detection**. This gives AgriMM a narrower and more operationally structured scope than benchmarks that focus only on question answering or only on visual recognition [2509.17044].

A plausible implication is that AgriMM was constructed not merely as a static dataset, but as a benchmark aligned with a concrete agentic workflow in which multimodal input must be routed to specialized tools before response synthesis. That interpretation is consistent with the architecture and evaluation protocol of AgriDoctor [2509.17044].

## 2. Internal composition

AgriMM contains approximately **400,000 images** spanning **29 crop categories** and **138 distinct types of agricultural diseases**. Each image is labeled with crop species and disease type, curated by agricultural experts. A substantial portion of the image set also includes **bounding box** annotations for lesion localization in YOLO format, written as `<class_id> <x_center> <y_center> <width> <height>`, normalized. The image sources combine **public datasets** such as Kaggle and Roboflow, web-crawled agricultural imagery, and **images captured in real-world agricultural environments during field surveys**. The distribution is long-tailed but still relatively large-scale: **128 disease categories** have **≥1,000 images each**, while **10 categories** have **200–1,000 images** [2509.17044].

The knowledge base contains **831 entries** with expert-level information on **crop diseases and pests**. The entries cover **symptoms, transmission, physiological impacts, integrated management, and more**. Every entry is provided in **Chinese and English**, and the content is derived from **peer-reviewed publications, expert manuals, and domain-specific curated databases**. The entries are categorized and standardized to support efficient retrieval and downstream integration [2509.17044].

The prompt corpus contains approximately **300,000 prompts** in **Chinese and English**. Each prompt is labeled with one of three intents: knowledge retrieval, disease classification, or disease detection. The prompt set was generated with **GPT-4o** as a seed corpus and then subjected to **rigorous manual filtering and verification** to ensure quality and diversity [2509.17044].

| Component | Quantity | Role |
|---|---:|---|
| Disease images | ~400,000 | Vision, detection, classification |
| Knowledge entries | 831 | Knowledge QA, context retrieval |
| Bilingual tool prompts | ~300,000 | Tool routing, intent disambiguation |

## 3. Construction methodology and annotation regime

The construction pipeline is defined by **expert supervision and verification**. Domain experts oversaw curation, annotation, and labeling to ensure **semantic and spatial accuracy**. The benchmark’s diversity is attributed to the joint use of **web scraping and field campaigns**, which broaden representation across species, diseases, and environments [2509.17044].

For images, experts labeled crop and disease classes and annotated lesions with bounding boxes where appropriate. For knowledge entries, experts extracted, distilled, and translated content, then structured it so that it aligns with disease and image categories. For prompts, the generation process aimed to cover realistic and varied user intents and linguistic formulations, after which the data were manually refined [2509.17044].

This construction strategy differs in emphasis from other agricultural multimodal datasets. AgMMU, for example, extracts all facts, questions, and answers from **116,231 real-world, one-to-one user–expert conversations** and provides an evaluation set of **5,460** MCQs and OEQs together with **205,399** pieces of agricultural knowledge information in its development set [2504.10568]. AgroMind, by contrast, is organized around agricultural remote sensing and contains **25,026 QA pairs** and **15,556 images** across **four core task dimensions** and **13 task types** [2505.12207]. This suggests that AgriMM is distinguished less by conversational provenance or remote-sensing breadth than by its explicit support for disease diagnosis, lesion localization, bilingual knowledge access, and intent-conditioned tool routing [2509.17044].

## 4. Intent-driven tool selection and system integration

AgriMM is tightly coupled to an intent-driven modular architecture. Its bilingual prompts and annotated intent labels are used to train a **Router module**, described as a **BERT-based classifier**, which infers which downstream module should process a multimodal query. A query routed to **classification** is sent to a **classifier (CLIP+ViT head)**; a query routed to **detection** is sent to a **YOLOv12-based detector**; and a query routed to **knowledge retrieval** is sent to a **knowledge retriever**, which uses **BERT and vector retrieval (FAISS)** over the knowledge base [2509.17044].

Each module is trained on the relevant subset of AgriMM. The **Router** is trained on bilingual intent-labeled prompts and achieves **99.0%/98.82% accuracy on English/Chinese**. The **Classifier** is trained with supervised fine-tuning on image labels using **weighted cross-entropy for class imbalance**. The class weight for ground-truth class $y$ is defined as
$$
w_y = \min\left(\frac{N}{N_y}, 10\right),
$$
where $N$ is the total number of samples and $N_y$ is the number of samples of class $y$. The **Detector** is trained on bounding-box-annotated disease regions. The **Knowledge Retriever** is trained and tested with structured knowledge entries and supports dense vector retrieval. At inference time, **only the selected tool's output is fed, with the original user query, to the LLM for final response synthesis** [2509.17044].

The operational significance of this design is that AgriMM does not treat multimodality as undifferentiated fusion. Instead, it supervises a dispatch mechanism over heterogeneous subproblems: recognition, localization, and knowledge-grounded explanation. That separation is central to the benchmark’s design logic [2509.17044].

## 5. Evaluation protocol and reported effectiveness

AgriMM supports both module-level evaluation and end-to-end evaluation through AgriDoctor. The benchmark uses a **balanced evaluation set** of **300 expert-verified samples**, with **100 each for classification, detection, and knowledge QA**, all excluded from training. For the end-to-end system, **DeepSeek-V3** is used to assess **Semantic Consistency (SC)** and **Information Completeness (IC)** for each task, each scored on a **0–1** range. The **Task Score** is defined as **Avg(SC, IC)**, and the **Overall Score** is the normalized average across all tasks [2509.17044].

On the end-to-end evaluation, **Qwen2.5-VL** reports **Class-Avg 0.72**, **Detect-Avg 0.42**, **QA-Avg 0.83**, and **Overall 0.659**. **GPT-4o-mini** reports **0.74**, **0.52**, **0.86**, and **0.704** on the same fields. **AgriDoctor**, trained on AgriMM, reports **Class-Avg 0.81**, **Detect-Avg 0.81**, **QA-Avg 0.98**, and **Overall 0.86** [2509.17044].

Module-specific results are likewise strong within the paper’s evaluation. The **classification tool** achieves **96.2% accuracy** on the disease classification task. The **detection tool** reports **Precision 0.876**, **Recall 0.868**, **mAP@50 0.893**, and **mAP@50–95 0.761**. On the knowledge QA task, AgriDoctor scores **0.98** compared with **0.69/0.92** for the cited baselines. The paper summarizes these results by stating that AgriDoctor, trained on AgriMM, **outperforms state-of-the-art LVLMs on fine-grained agricultural tasks** [2509.17044].

These results also speak to a broader issue in the literature: domain transfer remains difficult for generic multimodal systems. AgMMU reports that existing VLMs face **significant challenges** on questions requiring both detailed perception and factual knowledge, and that open-source VLMs show a **substantial performance gap** compared with proprietary ones [2504.10568]. AgroMind likewise reports that only GPT-4o consistently outperforms humans overall, while **16/21** evaluated LMMs fall below human level and all models perform poorly on key spatial tasks such as SL and SR [2505.12207].

## 6. Position within agricultural multimodal research

AgriMM belongs to a rapidly expanding cluster of agricultural multimodal resources, but it addresses a specific problem profile. It is centered on **crop disease diagnosis**, **lesion localization**, **knowledge retrieval**, and **bilingual intent routing**. AgMMU addresses **knowledge-intensive multimodal understanding** through MCQs and OEQs derived from real-world expert conversations [2504.10568]. AgroMind targets **agricultural remote sensing** across **spatial perception, object understanding, scene understanding, and scene reasoning** [2505.12207]. AgroGPT addresses the scarcity of agricultural vision-language data by constructing **AgroInstruct**, a **70k** expert-tuning dataset from vision-only agricultural datasets, and uses **AgroEvals** to assess fine-grained agricultural VQA [2410.08405]. AgroNVILA is built on **AgroOmni**, a **288,831 QA pair** multi-view training corpus spanning **Ground, UAV, Satellite** views, and emphasizes cross-scale spatial planning through **Perception-Reasoning Decoupling**, **VCMN**, and **ARPO** [2603.14342].

A plausible implication is that these resources occupy complementary strata of the same research program. AgriMM concentrates on tool-oriented disease workflows; AgMMU and AgReason emphasize knowledge-intensive and open-ended reasoning; AgroMind and AgroNVILA emphasize spatial reasoning and remote sensing; and AgroGPT emphasizes expert conversational ability under vision-only data constraints [2509.17044; 2505.19259; 2505.12207; 2410.08405; 2603.14342].

This broader landscape also shows that agricultural multimodality is not reducible to one modality pairing or one task family. MMST-ViT, for instance, couples **Sentinel-2 satellite images** with **short-term and long-term meteorological data** for county-level crop yield prediction and reports results across **corn, cotton, soybean, and winter wheat** on **over 200 counties** in the United States [2309.09067]. Other work extends multimodal and reasoning-oriented approaches to agricultural machinery management through **multi-round prompt engineering with GPT-4** [2407.20588] and to profit-aware crop recommendation through a hybrid system combining a **Random Forest classifier** and an **LSTM network** with a **voice-based interface in the local Kannada language** [2507.08832]. Within that larger ecosystem, AgriMM is best understood as a benchmark specifically engineered to make multimodal agricultural agents operationally selective, bilingual, and disease-centric rather than merely visually competent.

Source: https://www.emergentmind.com/topics/agrimm