AgriMM: Multimodal Benchmark for Crop Diagnosis

• AgriMM is a multimodal benchmark designed for fine-grained crop disease diagnosis, lesion localization, and bilingual agricultural knowledge retrieval.
• It integrates 400,000 annotated images, 831 expert-curated knowledge entries, and 300,000 bilingual prompts to support intent-driven tool routing and precise module supervision.
• The benchmark’s modular architecture and rigorous evaluation protocol significantly enhance accuracy in disease classification, detection, and knowledge reasoning tasks.

AgriMM is a large-scale, multimodal agricultural benchmark designed to support fine-grained multimodal learning for crop disease diagnosis and agricultural knowledge reasoning. Introduced together with AgriDoctor, it comprises 400000 annotated disease images, 831 expert-curated knowledge entries, and 300000 bilingual prompts for intent-driven tool selection. Its role is not limited to passive evaluation: the benchmark is constructed so that different subsets directly supervise a router, classifier, detector, and knowledge retriever within a modular multimodal system for intelligent crop disease diagnosis and agricultural knowledge interaction (Zhang et al., 21 Sep 2025).

1. Definition and scope

AgriMM is organized around three primary data components. The image component targets disease recognition and lesion localization; the knowledge component targets crop disease and pest reasoning; and the prompt component targets intent inference for automated module selection. This design makes AgriMM simultaneously a training resource and an evaluation substrate for multimodal agricultural agents (Zhang et al., 21 Sep 2025).

The benchmark is explicitly bilingual in Chinese and English for the knowledge entries and prompt data. It is also explicitly task-typed: prompts are annotated for Agricultural Knowledge Retrieval, Disease Classification, or Disease Detection. This gives AgriMM a narrower and more operationally structured scope than benchmarks that focus only on question answering or only on visual recognition (Zhang et al., 21 Sep 2025).

A plausible implication is that AgriMM was constructed not merely as a static dataset, but as a benchmark aligned with a concrete agentic workflow in which multimodal input must be routed to specialized tools before response synthesis. That interpretation is consistent with the architecture and evaluation protocol of AgriDoctor (Zhang et al., 21 Sep 2025).

2. Internal composition

AgriMM contains approximately 400,000 images spanning 29 crop categories and 138 distinct types of agricultural diseases. Each image is labeled with crop species and disease type, curated by agricultural experts. A substantial portion of the image set also includes bounding box annotations for lesion localization in YOLO format, written as <class_id> <x_center> <y_center> <width> <height>, normalized. The image sources combine public datasets such as Kaggle and Roboflow, web-crawled agricultural imagery, and images captured in real-world agricultural environments during field surveys. The distribution is long-tailed but still relatively large-scale: 128 disease categories have ≥1,000 images each, while 10 categories have 200–1,000 images (Zhang et al., 21 Sep 2025).

The knowledge base contains 831 entries with expert-level information on crop diseases and pests. The entries cover symptoms, transmission, physiological impacts, integrated management, and more. Every entry is provided in Chinese and English, and the content is derived from peer-reviewed publications, expert manuals, and domain-specific curated databases. The entries are categorized and standardized to support efficient retrieval and downstream integration (Zhang et al., 21 Sep 2025).

The prompt corpus contains approximately 300,000 prompts in Chinese and English. Each prompt is labeled with one of three intents: knowledge retrieval, disease classification, or disease detection. The prompt set was generated with GPT-4o as a seed corpus and then subjected to rigorous manual filtering and verification to ensure quality and diversity (Zhang et al., 21 Sep 2025).

Component	Quantity	Role
Disease images	~400,000	Vision, detection, classification
Knowledge entries	831	Knowledge QA, context retrieval
Bilingual tool prompts	~300,000	Tool routing, intent disambiguation

3. Construction methodology and annotation regime

The construction pipeline is defined by expert supervision and verification. Domain experts oversaw curation, annotation, and labeling to ensure semantic and spatial accuracy. The benchmark’s diversity is attributed to the joint use of web scraping and field campaigns, which broaden representation across species, diseases, and environments (Zhang et al., 21 Sep 2025).

For images, experts labeled crop and disease classes and annotated lesions with bounding boxes where appropriate. For knowledge entries, experts extracted, distilled, and translated content, then structured it so that it aligns with disease and image categories. For prompts, the generation process aimed to cover realistic and varied user intents and linguistic formulations, after which the data were manually refined (Zhang et al., 21 Sep 2025).

This construction strategy differs in emphasis from other agricultural multimodal datasets. AgMMU, for example, extracts all facts, questions, and answers from 116,231 real-world, one-to-one user–expert conversations and provides an evaluation set of 5,460 MCQs and OEQs together with 205,399 pieces of agricultural knowledge information in its development set (Gauba et al., 14 Apr 2025). AgroMind, by contrast, is organized around agricultural remote sensing and contains 25,026 QA pairs and 15,556 images across four core task dimensions and 13 task types (Li et al., 18 May 2025). This suggests that AgriMM is distinguished less by conversational provenance or remote-sensing breadth than by its explicit support for disease diagnosis, lesion localization, bilingual knowledge access, and intent-conditioned tool routing (Zhang et al., 21 Sep 2025).

4. Intent-driven tool selection and system integration

AgriMM is tightly coupled to an intent-driven modular architecture. Its bilingual prompts and annotated intent labels are used to train a Router module, described as a BERT-based classifier, which infers which downstream module should process a multimodal query. A query routed to classification is sent to a classifier (CLIP+ViT head); a query routed to detection is sent to a YOLOv12-based detector; and a query routed to knowledge retrieval is sent to a knowledge retriever, which uses BERT and vector retrieval (FAISS) over the knowledge base (Zhang et al., 21 Sep 2025).

Each module is trained on the relevant subset of AgriMM. The Router is trained on bilingual intent-labeled prompts and achieves 99.0%/98.82% accuracy on English/Chinese. The Classifier is trained with supervised fine-tuning on image labels using weighted cross-entropy for class imbalance. The class weight for ground-truth class $y$ is defined as

$w_y = \min\left(\frac{N}{N_y}, 10\right),$

where $N$ is the total number of samples and $N_y$ is the number of samples of class $y$. The Detector is trained on bounding-box-annotated disease regions. The Knowledge Retriever is trained and tested with structured knowledge entries and supports dense vector retrieval. At inference time, only the selected tool's output is fed, with the original user query, to the LLM for final response synthesis (Zhang et al., 21 Sep 2025).

The operational significance of this design is that AgriMM does not treat multimodality as undifferentiated fusion. Instead, it supervises a dispatch mechanism over heterogeneous subproblems: recognition, localization, and knowledge-grounded explanation. That separation is central to the benchmark’s design logic (Zhang et al., 21 Sep 2025).

5. Evaluation protocol and reported effectiveness

AgriMM supports both module-level evaluation and end-to-end evaluation through AgriDoctor. The benchmark uses a balanced evaluation set of 300 expert-verified samples, with 100 each for classification, detection, and knowledge QA, all excluded from training. For the end-to-end system, DeepSeek-V3 is used to assess Semantic Consistency (SC) and Information Completeness (IC) for each task, each scored on a 0–1 range. The Task Score is defined as Avg(SC, IC), and the Overall Score is the normalized average across all tasks (Zhang et al., 21 Sep 2025).

On the end-to-end evaluation, Qwen2.5-VL reports Class-Avg 0.72, Detect-Avg 0.42, QA-Avg 0.83, and Overall 0.659. GPT-4o-mini reports 0.74, 0.52, 0.86, and 0.704 on the same fields. AgriDoctor, trained on AgriMM, reports Class-Avg 0.81, Detect-Avg 0.81, QA-Avg 0.98, and Overall 0.86 (Zhang et al., 21 Sep 2025).

Module-specific results are likewise strong within the paper’s evaluation. The classification tool achieves 96.2% accuracy on the disease classification task. The detection tool reports Precision 0.876, Recall 0.868, mAP@50 0.893, and mAP@50–95 0.761. On the knowledge QA task, AgriDoctor scores 0.98 compared with 0.69/0.92 for the cited baselines. The paper summarizes these results by stating that AgriDoctor, trained on AgriMM, outperforms state-of-the-art LVLMs on fine-grained agricultural tasks (Zhang et al., 21 Sep 2025).

These results also speak to a broader issue in the literature: domain transfer remains difficult for generic multimodal systems. AgMMU reports that existing VLMs face significant challenges on questions requiring both detailed perception and factual knowledge, and that open-source VLMs show a substantial performance gap compared with proprietary ones (Gauba et al., 14 Apr 2025). AgroMind likewise reports that only GPT-4o consistently outperforms humans overall, while 16/21 evaluated LMMs fall below human level and all models perform poorly on key spatial tasks such as SL and SR (Li et al., 18 May 2025).

6. Position within agricultural multimodal research

AgriMM belongs to a rapidly expanding cluster of agricultural multimodal resources, but it addresses a specific problem profile. It is centered on crop disease diagnosis, lesion localization, knowledge retrieval, and bilingual intent routing. AgMMU addresses knowledge-intensive multimodal understanding through MCQs and OEQs derived from real-world expert conversations (Gauba et al., 14 Apr 2025). AgroMind targets agricultural remote sensing across spatial perception, object understanding, scene understanding, and scene reasoning (Li et al., 18 May 2025). AgroGPT addresses the scarcity of agricultural vision-language data by constructing AgroInstruct, a 70k expert-tuning dataset from vision-only agricultural datasets, and uses AgroEvals to assess fine-grained agricultural VQA (Awais et al., 2024). AgroNVILA is built on AgroOmni, a 288,831 QA pair multi-view training corpus spanning Ground, UAV, Satellite views, and emphasizes cross-scale spatial planning through Perception-Reasoning Decoupling, VCMN, and ARPO (Zhang et al., 15 Mar 2026).

A plausible implication is that these resources occupy complementary strata of the same research program. AgriMM concentrates on tool-oriented disease workflows; AgMMU and AgReason emphasize knowledge-intensive and open-ended reasoning; AgroMind and AgroNVILA emphasize spatial reasoning and remote sensing; and AgroGPT emphasizes expert conversational ability under vision-only data constraints (Zhang et al., 21 Sep 2025, Zaremehrjerdi et al., 25 May 2025, Li et al., 18 May 2025, Awais et al., 2024, Zhang et al., 15 Mar 2026).

This broader landscape also shows that agricultural multimodality is not reducible to one modality pairing or one task family. MMST-ViT, for instance, couples Sentinel-2 satellite images with short-term and long-term meteorological data for county-level crop yield prediction and reports results across corn, cotton, soybean, and winter wheat on over 200 counties in the United States (Lin et al., 2023). Other work extends multimodal and reasoning-oriented approaches to agricultural machinery management through multi-round prompt engineering with GPT-4 (Johnson et al., 2024) and to profit-aware crop recommendation through a hybrid system combining a Random Forest classifier and an LSTM network with a voice-based interface in the local Kannada language (Sindhur et al., 6 Jul 2025). Within that larger ecosystem, AgriMM is best understood as a benchmark specifically engineered to make multimodal agricultural agents operationally selective, bilingual, and disease-centric rather than merely visually competent.