Agri-3M-VL: Agricultural Vision–Language Data
- Agri-3M-VL is a large-scale agriculture-specific vision–language training corpus comprising 1M image-caption pairs, 2M VQA pairs, 50K expert QAs, and 15K GRPO samples.
- It underpins the AgriGPT-VL suite by enabling progressive training from shallow alignment to deep multimodal reasoning and reinforcement learning refinement.
- A multi-agent pipeline generates, refines, and filters the data, ensuring factual consistency, expert-level instruction quality, and robust image-text grounding.
Searching arXiv for Agri-3M-VL and closely related agricultural multimodal resources. Search results (relevant):
- (Yang et al., 5 Oct 2025) — "AgriGPT-VL: Agricultural Vision-Language Understanding Suite"
- (Gauba et al., 14 Apr 2025) — "AgMMU: A Comprehensive Agricultural Multimodal Understanding and Reasoning Benchmark"
- (Wang et al., 2024) — "Agri-LLaVA: Knowledge-Infused Large Multimodal Assistant on Agricultural Pests and Diseases"
- (Zhao et al., 16 Oct 2025) — "SUM-AgriVLN: Spatial Understanding Memory for Agricultural Vision-and-Language Navigation"
- (Zhang et al., 15 Mar 2026) — "AgroNVILA: Perception-Reasoning Decoupling for Multi-view Agricultural Multimodal LLMs" Agri-3M-VL is a large-scale agriculture-specific vision–language training corpus created for the AgriGPT-VL suite. Within that suite, it functions as the foundational multimodal dataset used to train AgriGPT-VL, while AgriBench-VL-4K serves as the evaluation benchmark. The corpus is designed to convert heterogeneous agricultural image datasets into multimodal supervision for captioning, image-grounded question answering, expert agronomic reasoning, and reinforcement-learning-based refinement. In the AgriGPT-VL paper, it is presented as the largest vision-language corpus for agriculture to the authors’ knowledge, with 1M image-caption pairs, 2M image-grounded VQA pairs, 50K expert-level VQA instances, and 15K GRPO reinforcement learning samples (Yang et al., 5 Oct 2025).
1. Position within the AgriGPT-VL suite
Agri-3M-VL is the dataset component of the AgriGPT-VL agricultural vision–language understanding suite. The suite has three major parts: Agri-3M-VL as the training corpus, AgriGPT-VL as the agriculture-specialized vision-LLM, and AgriBench-VL-4K as the evaluation benchmark. This division is methodologically important because Agri-3M-VL is used mainly as training data rather than as the primary benchmark (Yang et al., 5 Oct 2025).
The corpus is motivated by three deficits the AgriGPT-VL paper attributes to prior agricultural multimodal work: prior agricultural LLM efforts such as AgriBERT, AgriLLM, AgroLLM, and earlier AgriGPT primarily handled text rather than visual grounding; agricultural image datasets such as PlantVillage and IP102 were useful for recognition or classification but were not instruction datasets; and previous work offered isolated datasets, models, or benchmarks rather than a coherent training-evaluation stack. Agri-3M-VL is therefore positioned between general-domain vision–language corpora, which are large but weak in agricultural expertise and grounding, and traditional agricultural datasets, which are visually relevant but not in multimodal instruction-following form (Yang et al., 5 Oct 2025).
A recurrent misunderstanding is to treat Agri-3M-VL as a benchmark. The paper does not do so. AgriBench-VL-4K is the compact evaluation suite, whereas Agri-3M-VL is the training corpus that supports shallow alignment, deep alignment, expert supervised fine-tuning, and GRPO refinement (Yang et al., 5 Oct 2025).
2. Scale, composition, and source coverage
The paper defines Agri-3M-VL through four headline components.
| Component | Scale |
|---|---|
| Image-caption pairs | 1M |
| Image-grounded VQA pairs | 2M |
| Expert-level VQA instances | 50K |
| GRPO reinforcement learning samples | 15K |
The 15K GRPO dataset is further described as 10K VQA and 5K single-choice questions, and is characterized as a rewarded GRPO reinforcement learning dataset. The paper does not fully specify exact overlap relationships among all four components. It states that the 1M image-caption pairs are generated first, that the 2M VQA pairs are synthesized building upon the image-caption pairs, that the 50K expert-level VQA are an additional 50K subset verified or polished with GPT-4o and retained after filtering and manual verification as expert instructions, and that the 15K GRPO data are additionally constructed for reward modeling and preference optimization. This suggests that the caption and VQA components are sequentially related, that the expert VQA component is likely a filtered or polished subset, and that exact overlap with the GRPO set is not stated (Yang et al., 5 Oct 2025).
The raw visual pool comes from multiple agricultural datasets. The paper provides concrete source counts: PlantVillage with 54,305 images and 38 classes; Species196 with 6,878 images and 166 fine-grained insect species; Insect Foundation with 317,128 images and 38,867 fine-grained insect classes; SelectDataset with 558,930 images and 2,958 categories; Fruits-360 with 97,255 images and 206 categories; and Fresh-Rotten Fruit with 30,357 images and 18 categories. Aggregated totals are given as 324,006 insect images across 39,033 classes, 157,969 fruit images across 224 classes, and 1,064,853 images across 42,253 fine-grained categories overall (Yang et al., 5 Oct 2025).
The paper explicitly describes coverage across pests and diseases, insects, crops, weeds, and fruits. Caption content is said to include crop growth stage, leaf color, fruit maturity, pest morphology, lesion color and spread, external texture, and ripeness stage. VQA examples and task descriptions additionally indicate species recognition, symptom and mechanism analysis, disease diagnosis, management recommendations, multi-step agronomic reasoning, attribute recognition, and basic multi-hop reasoning. However, the paper does not provide a full taxonomy of crop types or disease categories beyond source-dataset class counts, and languages are not specified for Agri-3M-VL (Yang et al., 5 Oct 2025).
3. Multi-agent generation and curation pipeline
Agri-3M-VL is constructed through a four-stage Data Generator pipeline: Caption Generation, Instruction Synthesis, Multi-Agent Refinement, and Instruction Filtering. The paper describes this as a multi-agent data generator for transforming raw agricultural images into instruction-ready corpora, and also as a transferable paradigm that could be reused in other scientific domains (Yang et al., 5 Oct 2025).
In Caption Generation, raw images from the collected agricultural datasets are converted into structured visual captions describing observable image content. The examples given include diseased tomato leaves described by lesion color and spread and fruit images described by ripeness stage and external texture. This stage produces about 1 million image-caption pairs.
In Instruction Synthesis, the image-caption pairs are used by Qwen2.5-VL 72B and GPT-4o to generate diverse question-answer style supervision. The paper says this stage produces high-quality factual queries, expert-level reasoning tasks, and interactive multimodal dialogues. An example pair is given for a weed image: “What species of weed is shown?” for recognition and “What is the likely impact of this weed on crop yield?” for reasoning. The output is approximately 2 million VQA samples covering both open-ended and single-choice formats.
Multi-Agent Refinement is the main quality-control stage. The paper states that generated data are processed through a loop with a feedback agent, an evaluation agent, and a rethinking agent. The evaluation agent scores samples along dimensions such as factual consistency and image grounding, and the rethinking agent rewrites samples with self-consistency checks. Samples iterate among these agents until preset quality thresholds are met. Additional curation in this stage includes GPT-4o verification and polishing of an additional 50K subset for supervised fine-tuning and construction of the 15K GRPO reinforcement learning dataset.
Instruction Filtering then removes irrelevant instructions, hallucinated instructions, and generic questions not actually grounded in the image. The example of a removed question is “How many legs does a seven-spotted ladybug have?” and the example of a retained question is “What kind of insect is in the picture?” After filtering and manual verification, about 50K expert instructions remain (Yang et al., 5 Oct 2025).
Several details are explicitly not stated for Agri-3M-VL: prompt templates for caption or VQA generation, exact heuristics used by each agent, scoring thresholds, pseudocode, inter-annotator agreement, exact human annotation workforce details, and the exact deduplication recipe for the training corpus. The paper does describe de-duplication for AgriBench-VL-4K, but it does not explicitly transfer those procedures to Agri-3M-VL (Yang et al., 5 Oct 2025).
4. Data structure and role in the training curriculum
Agri-3M-VL supports multiple multimodal training tasks: image captioning, image-grounded VQA, expert-level agronomic reasoning VQA, instruction-following multimodal dialogue, single-choice image-grounded QA, and reward-based RL or GRPO refinement. The paper does not provide a formal JSON schema, but it gives the implied structures of image-caption pairs, image-question-answer triples, expert VQA instances, and GRPO refinement data (Yang et al., 5 Oct 2025).
The corpus is integrated into a progressive training curriculum for AgriGPT-VL. Stage 1 is text-only domain grounding and is not Agri-3M-VL proper; it uses approximately 200K documents and about 2.2B tokens, together with supervised instruction tuning on Agri-342K. Agri-3M-VL is used primarily in later stages. In multimodal shallow alignment, the 1M image-caption pairs are used while the vision encoder and LLM are frozen and only connector and adapter layers are trained. In multimodal deep alignment, the 2M image-QA samples are used with LoRA while the vision encoder and LLM are gradually unfrozen. In the later refinement stage, 50K GPT-4o-polished expert samples are used for supervised fine-tuning and 15K GRPO samples are used for reinforcement learning. The paper summarizes the role of the four components as follows: the 1M captions teach image-to-text mapping, the 2M VQA teach image-conditioned instruction following and reasoning, the 50K expert VQA provide high-trust supervised signals, and the 15K GRPO data refine output quality toward image-text consistency, internal logic, and terminology correctness (Yang et al., 5 Oct 2025).
The paper does not give LaTeX training objectives for multimodal alignment, supervised fine-tuning, GRPO, or reward modeling in the provided text. It says only that details are in appendices not reproduced there. It also does not specify whether explicit rationale or chain-of-thought fields are stored in Agri-3M-VL. The text refers to expert-level reasoning, internal logic, self-consistency checks, and multi-step reasoning, but does not explicitly state that rationales are included as supervision targets (Yang et al., 5 Oct 2025).
5. Empirical role in downstream performance
The evidence for Agri-3M-VL is primarily downstream and ablation-based. On AgriBench-VL-4K, AgriGPT-VL achieves Acc 85.84%, Acc 74.17%, BLEU 26.27, Meteor 47.55, Rouge-1-f 46.52, Rouge-2-f 20.09, and Rouge-L-f 43.81. The paper reports that these outperform all reported baselines in the table, including Qwen2.5-VL-72B-Instruct, which had Acc 81.70%, Acc 67.49%, BLEU 15.41, Meteor 41.38, Rouge-1-f 39.14, Rouge-2-f 13.25, and Rouge-L-f 36.63. The paper interprets these gains as improvements in visual evidence grounding, agronomic multi-step reasoning, and professional expression (Yang et al., 5 Oct 2025).
The alignment-stage ablation gives the clearest evidence of what the Agri-3M-VL subcomponents contribute. Starting from Qwen2.5-VL-7B, the base system scores Acc 77.20%, Acc 60.32%, BLEU 13.42, Meteor 38.24, Rouge-1-f 35.52, Rouge-2-f 10.78, and Rouge-L-f 32.73. Adding Shallow Alignment, which uses the 1M image-caption pairs, raises the model to Acc 78.23%, Acc 62.47%, BLEU 15.54, Meteor 36.47, Rouge-1-f 40.76, Rouge-2-f 14.07, and Rouge-L-f 37.95. Adding Deep Alignment, which uses the caption data plus the 2M VQA pairs, yields Acc 81.18%, Acc 66.67%, BLEU 21.68, Meteor 44.38, Rouge-1-f 43.04, Rouge-2-f 15.62, and Rouge-L-f 40.37. The full pipeline with GRPO reaches the final 85.84% Acc and 74.17% Acc. The paper summarizes these roles as follows: Shallow Alignment improves lexical and descriptive consistency, Deep Alignment is the main driver of cross-modal understanding and reasoning, and GRPO improves factual faithfulness and robustness, especially strict image-level cross-consistency (Yang et al., 5 Oct 2025).
A plausible implication is that Agri-3M-VL is not merely a large corpus in a scaling-law sense; it is also explicitly partitioned to support different adaptation regimes. The paper’s own interpretation, however, is grounded in the staged gains rather than in a new formal objective (Yang et al., 5 Oct 2025).
6. Relation to adjacent agricultural multimodal resources
Agri-3M-VL occupies a distinct place within agricultural multimodal research. AgMMU is best understood as an agriculture-specific multimodal QA benchmark plus a companion development or training resource derived from 116,231 real-world conversations between users and authorized agricultural experts. Its emphasis is expert-grounded factual diagnosis, symptom description, management instructions, and benchmark realism rather than broad multimodal pretraining scale. In that sense, AgMMU is complementary to Agri-3M-VL: the former emphasizes evaluative rigor and expert factuality, whereas Agri-3M-VL emphasizes large-scale instruction-ready training data (Gauba et al., 14 Apr 2025).
Agri-LLaVA is presented as the first multimodal instruction-following dataset in agriculture and the first agricultural multimodal assistant, but it is narrowly focused on pests and diseases. Its resources include 391,785 image-text pairs, 6,000 instruction conversations, and dedicated chatbot and VQA benchmarks. Relative to Agri-3M-VL, Agri-LLaVA is narrower in scope and more explicitly knowledge-infused around pest and disease diagnosis, transmission, and control, whereas Agri-3M-VL is broader across pests and diseases, insects, crops, weeds, and fruits and is embedded in a progressive curriculum with expert and GRPO components (Wang et al., 2024).
AgroNVILA explicitly compares against Agri-3M-VL and characterizes Agri-3M-VL as large-scale and reasoning-capable, but not multi-view and lacking spatial grounding. AgroNVILA’s contribution is a 288K multi-view corpus spanning Ground, UAV, and Satellite perspectives, together with a Perception-Reasoning Decoupling architecture. This suggests that Agri-3M-VL represents a terrestrial, instruction-centric training paradigm, while AgroNVILA targets cross-altitude spatial intelligence (Zhang et al., 15 Mar 2026).
SUM-AgriVLN is even further downstream. It does not mention Agri-3M-VL directly and is not a general-purpose agricultural vision-LLM paper. Instead, it extends agricultural vision-and-language navigation with persistent 3D scene memory, evaluated on the A2A benchmark. Its relevance to Agri-3M-VL is therefore conceptual rather than direct: it illustrates how agricultural language grounding can be coupled to action, repeated-scene memory, and 3D spatial reasoning in embodied settings (Zhao et al., 16 Oct 2025).
7. Limitations, ambiguities, and interpretive boundaries
The AgriGPT-VL paper does not present a dedicated limitations section for Agri-3M-VL, but several caveats are explicit. The training corpus requires filtering, manual verification, hallucination removal, and factual alignment, implying that synthetic generation would otherwise introduce noise. The paper does not explicitly state geographic imbalance, language imbalance, demographic or regional bias, class imbalance statistics, image quality statistics, expert coverage limitations, licensing constraints for inherited source datasets, or ethical risks specific to the training data. It also does not provide a public train, validation, and test split for Agri-3M-VL itself (Yang et al., 5 Oct 2025).
Several points should therefore be stated negatively. Agri-3M-VL is not described as multilingual. It is not described through a complete ontology of crops or diseases. It is not accompanied, in the provided text, by exact prompt templates, deduplication algorithms, or human-annotation protocols for the training set. Specialist human expert involvement in the curation of Agri-3M-VL itself is not clearly specified, even though the benchmark side of the AgriGPT-VL suite refers to agriculture-literate annotators. Likewise, the exact overlap among the caption, VQA, expert, and GRPO components is not fully specified (Yang et al., 5 Oct 2025).
These omissions matter for interpretation. Agri-3M-VL is best understood as a large, instruction-ready, quality-controlled agricultural training corpus whose downstream value is demonstrated through the AgriGPT-VL curriculum and AgriBench-VL-4K results, rather than as a fully exhaustively documented data standard. Its significance lies in showing how agricultural image collections can be transformed into caption, VQA, expert, and reinforcement-oriented supervision at multimillion-sample scale, while its main unresolved questions concern provenance granularity, subset overlap, and dataset governance (Yang et al., 5 Oct 2025).