Agri-342K: Agricultural Multimodal Data Overview

• Agri-342K is an informal multimodal agricultural data suite combining 400K feature alignment samples and 6K instruction-tuning conversations focused on pests and diseases.
• The dataset leverages expert labels and GPT-4 to transform agricultural knowledge into structured image-text dialogues, enhancing model training and evaluation.
• It plays a key role in Agri-LLaVA’s two-stage training pipeline and contrasts with benchmarks like AgrI Challenge that focus on cross-domain validation.

Agri-342K is not introduced as a formal dataset name in the cited arXiv literature. In that literature, the closest referent is the large agricultural multimodal instruction-following corpus constructed for Agri-LLaVA: a resource covering over 221 types of pests and diseases, with approximately 400,000 data entries, operationalized as a feature alignment dataset of approximately 400,000 samples based on 391,785 images and a curated instruction-tuning dataset of 6,000 high-quality agricultural multimodal conversations (Wang et al., 2024). It is methodologically distinct from the AgrI Challenge benchmark, which addresses tree species classification through heterogeneous field data collected by multiple teams and evaluates generalization under Cross-Team Validation rather than instruction-following or multimodal dialogue (Brahimi et al., 7 Mar 2026).

1. Nomenclature and corpus identity

A recurring source of ambiguity is the name itself. In the Agri-LLaVA paper, “Agri-342K” does not appear as a named dataset or resource. The paper instead defines four explicitly named resources: agricultural pests and diseases feature alignment data, agricultural pests and diseases instruction-tuning data, Agri-LLaVA-Chatbot-Bench, and Agri-LLaVA-VQA-Bench (Wang et al., 2024).

For that reason, “Agri-342K” is best understood as an informal or external designation for the multimodal agricultural data ecosystem surrounding Agri-LLaVA rather than a canonical title used in the paper. A common misconception is that the paper contains a dataset section, table, or appendix entry defining a corpus of exactly 342K samples. It does not. The scales actually reported are 391,785 images, approximately 400,000 feature alignment samples, and 6,000 instruction-tuning conversations.

Resource	Scale or composition	Function
Agricultural pests and diseases feature alignment data	approximately 400,000 samples; based on 391,785 images	Stage-1 feature alignment
Agricultural pests and diseases instruction-tuning data	6,000 high-quality multimodal conversation data	Stage-2 instruction tuning
Agri-LLaVA-Chatbot-Bench	30 images; 151 conversational rounds	Conversational evaluation
Agri-LLaVA-VQA-Bench	482 images; 2,268 QA pairs	VQA and visual reasoning evaluation

This naming ambiguity matters because the resource is often discussed as if it were a single monolithic benchmark. The paper instead presents a training-and-evaluation suite with distinct components, each serving a different role in multimodal agricultural modeling.

2. Data sources, scope, and taxonomic coverage

The core training corpus is centered on agricultural pests and diseases rather than general agricultural scenes. The paper states that it covers over 221 types of pests and diseases with approximately 400,000 data entries, and that 391,785 images were downloaded and preprocessed from 16 datasets (Wang et al., 2024).

The source material combines public imaging datasets and web-based agricultural knowledge sources. The listed image sources include IP102, PlantVillage and other plant disease datasets, Baidu PaddlePaddle datasets for crops such as rice, cabbage, pepper, and wheat, AppleLeaf9, grape disease datasets, wheat leaf datasets, and related collections. The knowledge sources include the Chinese Academy of Agricultural Sciences Pests and Diseases Database and other agricultural websites used to retrieve category names and detailed symptom descriptions.

The underlying taxonomy spans crop species, pest species, and disease types. The paper gives examples including wheat, rice, apple, grape, banana, soybean, peach, potato, sugarcane, beet, citrus, blueberry, cassava, tomato, corn, tea, pepper, cowpea, cashew, lemon, ash gourd, bitter gourd, and alfalfa; insect pests identified with scientific names; and diseases such as wheat powdery mildew, rice sheath blight, apple powdery mildew, grape downy mildew, tomato canker, and potato soft rot disease. The knowledge associated with each class is organized around symptoms, pathogen, transmission conditions, and prevention and control methods.

Preprocessing is part of the dataset definition. Images with multiple sub-images are split to increase sample count, abstract or non-diagnostic images are removed, and filenames are standardized as cropCategory_pestOrDiseaseName_index.jpg, for example mango_sternochetus frigidus_1.jpg. These steps indicate that the dataset is not merely aggregated from existing sources but normalized for multimodal supervision.

3. Knowledge construction and annotation format

The dataset construction pipeline is knowledge-infused rather than manually re-annotated from scratch. Existing expert labels in the source datasets provide crop and pest-or-disease names, while agricultural knowledge is collected from expert-curated web sources. GPT-4 is then used to transform these inputs into multimodal conversations (Wang et al., 2024).

For the feature alignment data, the model builders attach category labels and symptom descriptions to each image and generate two-round image-text conversations. The first question asks for a simple description of the visual features, and the second asks for the category and detailed symptoms. In the paper’s notation, each sample is represented as

$$\begin{align*} \text{Human: } X_{q1}, X_i \textcolor{Green}{<STOP>} \ \text{Assistant: } X_k \textcolor{Green}{<STOP>} \ \text{Human: } X_{q2}, X_i \textcolor{Green}{<STOP>} \ \text{Assistant: } X_k \textcolor{Green}{<STOP>} \end{align*}$$

where $X_i$ is the image, $X_{q1}$ and $X_{q2}$ are the two questions, and $X_k$ is the associated knowledge. The corresponding JSON format contains an "image" field and a "conversations" list.

For the instruction-tuning data, the paper reports 6,000 high-quality agricultural multimodal conversation instances derived from 5,813 images plus associated structured knowledge. These samples are multi-turn and knowledge-intensive, covering symptoms, pathogen, transmission, control, and related agronomic details. Structured knowledge is extracted from web pages by segmenting text according to keywords such as “symptoms”, “pathogens”, “transmission conditions”, and “control methods”, then organizing the output into a structured JSON knowledge base.

Quality control is explicitly described. For feature alignment data, the first question is required to include the word “image”; prompt templates were refined when GPT-4 violated this constraint. When GPT-4 answered from its own prior knowledge rather than the supplied agricultural knowledge, the prompt was adjusted, and rare errors were manually fixed. For Chatbot-Bench, some GPT-4 answers were manually refined for detail, and redundant questions were removed to improve diversity.

This suggests that the defining characteristic of the corpus is not only scale, but the deliberate coupling of images with constrained, domain-grounded conversational supervision.

4. Role in Agri-LLaVA training

The dataset suite is used to train Agri-LLaVA through a two-stage knowledge-infused pipeline built on LLaVA-1.5 (Wang et al., 2024). The architecture follows the standard LLaVA pattern: a visual encoder, a projection matrix mapping visual features into the LLM token space, and a LLaMA-style language backbone.

Stage 1 is pre-training for feature alignment. The data are the 400K feature alignment samples. In this stage, the visual encoder is frozen, the LLM backbone is frozen, and only the projection matrix is trained. The reported hyperparameters are a learning rate of $1\times10^{-3}$, batch size 256, and 1 epoch, with training time of 10 hours and 40 minutes on 8×A800 GPUs.

Stage 2 is end-to-end instruction tuning. The data are the 6K multimodal conversations. In this stage, the visual encoder remains frozen, the projection matrix is updated, and the LLM is fully fine-tuned. The reported hyperparameters are a learning rate of $2\times10^{-5}$, batch size 128, and 3 epochs, with training time of 40 minutes. The authors also examine LoRA and full fine-tuning variants.

The paper does not introduce a separate symbolic knowledge module. Knowledge infusion occurs at data construction time: category names, symptom descriptions, and structured narratives about pathogen, transmission, and control are encoded into the supervision signal. A plausible implication is that the corpus is designed to specialize a general-domain LMM through domain-grounded supervised fine-tuning rather than through explicit retrieval or knowledge graph integration.

5. Evaluation benchmarks and empirical performance

The Agri-LLaVA data suite includes two held-out evaluation benchmarks that are distinct from the training datasets: Agri-LLaVA-Chatbot-Bench and Agri-LLaVA-VQA-Bench (Wang et al., 2024).

Agri-LLaVA-Chatbot-Bench evaluates multimodal conversational ability in agriculture. It contains 30 images, comprising 6 pests and 24 diseases, with 4–6 turns per image and 151 conversational rounds in total. The paper states that 25 out of 30 types are not seen during training. Evaluation uses GPT-4 as a relative scorer: GPT-4 first answers the question using only the knowledge text as an upper-bound reference, and then scores candidate model answers on a 1–10 scale for helpfulness, relevance, accuracy, and detail. In the reported results, baseline LLaVA achieves 52.2 on Disease, 43.6 on Pest, and 50.6 Overall, while the best Agri-LLaVA configuration with 6K instruction data plus extra 3K simple single-round data reaches 54.0 on Disease, 61.4 on Pest, and 55.4 Overall. The paper summarizes this as Agri-LLaVA achieving 55.4% of GPT-4’s performance.

Agri-LLaVA-VQA-Bench evaluates visual question answering and visual reasoning. It contains 482 images, including 476 pest/disease images and 6 healthy plant images, and 2,268 QA pairs. The benchmark covers 49 disease types and 50 pest types. Questions span 9 themes, including organ damage, abnormal symptoms, related attributes, potential hazards, disease or pest name, causes or conditions, prevention methods, transmission routes, and other relevant topics. The paper states that 21 disease types and 3 pest types do not appear in training, and that the benchmark images themselves do not appear in the training data.

The VQA metrics are explicitly defined. For closed-set questions, Accuracy is used:

$$\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}.$$

For open-set questions, the benchmark uses $F_1$-score:

$$F_{1} = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}},$$

with

$X_i$0

The best Agri-LLaVA variant reported in the paper reaches Open-set $X_i$1, Closed-set Accuracy $X_i$2, and Average $X_i$3, compared with 55.18 average for baseline LLaVA. The paper characterizes this as a 4.87% higher comprehensive ability than LLaVA.

These results position the corpus not as a passive archive but as a training substrate linked to explicit agricultural evaluation tasks.

6. Relation to agricultural benchmark design and to AgrI Challenge

Within agricultural AI, the interpreted “Agri-342K” resource and AgrI Challenge occupy different methodological niches. The former is a multimodal instruction-following corpus focused on pests and diseases, knowledge-infused dialogue construction, and LMM specialization. AgrI Challenge is a field-collected benchmark for tree species classification with six tree species, 50,673 raw field images, and a final clean dataset of 47,367 images from 11 teams across 6 classes; its central innovation is Cross-Team Validation, comprising Train-on-One-Team-Only and Leave-One-Team-Out protocols for measuring domain shift across independently collected datasets (Brahimi et al., 7 Mar 2026).

The contrast is informative. Agri-LLaVA operationalizes agricultural intelligence through multimodal conversations, structured knowledge, and downstream evaluation on chatbot and VQA tasks. AgrI Challenge operationalizes agricultural robustness through explicit domain partitioning, cross-domain generalization, and validation–test gap analysis. In AgrI Challenge, single-source training produces validation–test gaps of up to 16.20% for DenseNet121 and 11.37% for Swin Transformer, while collaborative multi-source training reduces these gaps to 2.82% and 1.78%, respectively. That framework addresses data collection practices and distribution shift in outdoor field imagery rather than knowledge-intensive conversation.

A common conceptual error is to treat all large agricultural datasets as interchangeable “benchmarks.” The cited literature instead suggests at least two distinct benchmark philosophies. One is multimodal instruction-following with domain knowledge, exemplified by the Agri-LLaVA corpus. The other is multi-source domain generalization under heterogeneous acquisition conditions, exemplified by AgrI Challenge. These are complementary rather than competing formulations of agricultural vision and multimodal learning.

7. Limitations, release status, and interpretive cautions

The Agri-LLaVA paper identifies several limitations of its data and model ecosystem (Wang et al., 2024). Despite aggregation from many sources, some pest and disease types remain underrepresented, producing data scarcity and imbalance. The model can still misdiagnose conditions, especially when visual symptoms resemble other diseases or when deeper causal or pathogen-level detail is requested. Hallucinations persist even with knowledge injection. The authors also note generalization limitations arising from uneven agricultural data quality and variability, as well as broader concerns involving data privacy, security, and the risk of misleading recommendations.

The resource is intended for open research use. The paper states that all resources can be found at https://github.com/Kki2Eve/Agri-LLaVA. It describes images stored as .jpg files and labels or conversations stored in JSON, with four principal JSON datasets corresponding to feature alignment data, instruction-tuning data, Chatbot-Bench, and VQA-Bench.

Interpretively, the most precise characterization is that “Agri-342K” is not a formally defined benchmark in the cited papers, but an informal label that aligns most closely with the Agri-LLaVA multimodal agricultural data suite. A plausible implication is that external references using the term are pointing to a processed subset, companion release, or shorthand for the larger approximately 400K-scale corpus. What is firmly established in the literature is the existence of a large agricultural instruction-following dataset for pests and diseases, built from 391,785 images and 6,000 knowledge-rich conversations, together with dedicated conversational and VQA benchmarks for evaluating agricultural large multimodal models (Wang et al., 2024).