AgriBench-VL-4K: Agricultural Vision-Language Benchmark

• AgriBench-VL-4K is a multimodal benchmark that evaluates agricultural models through 3,876 QA pairs, testing both image recognition and agronomic reasoning.
• The benchmark combines generative open-ended responses and image-grounded single-choice questions, enabling detailed assessment of identification, diagnosis, and management strategies.
• AgriBench-VL-4K integrates multi-metric evaluation and an LLM-as-a-judge framework to measure agronomic soundness and cross-consistency in model outputs.

AgriBench-VL-4K is an evaluation suite for agricultural vision-language understanding introduced within the AgriGPT-VL Suite. It is described as a compact yet challenging benchmark that combines open-ended and image-grounded questioning, multi-metric evaluation, and an LLM-as-a-judge framework for preference-based assessment (Yang et al., 5 Oct 2025). Its design targets multimodal systems that must not only recognize agricultural entities and symptoms in images, but also produce agronomically sound answers about mechanisms, diagnosis, and management.

1. Definition and scope

AgriBench-VL-4K is a multimodal benchmark centered on agricultural question answering over images. It contains both generative and discriminative tasks: open-ended questions over held-out images and image-grounded single-choice questions with candidate options (Yang et al., 5 Oct 2025). The suite is intended to evaluate agricultural vision-LLMs on recognition, symptom or mechanism analysis, management recommendations, and simple multi-step reasoning.

The benchmark is framed as part of a broader effort to address the scarcity of domain-tailored models, curated vision-language corpora, and rigorous evaluation in agriculture. Within that context, AgriBench-VL-4K functions as the evaluation component of the AgriGPT-VL Suite. The paper states that it is a compact yet challenging evaluation suite and emphasizes paired use of multi-metric reporting and preference-based judging.

A central feature of the benchmark is its mixture of answer modalities. Open-ended items require free-form response generation, whereas image-grounded single-choice items require selecting from hard distractors mined from confusable taxa or co-occurring conditions. This combination places the benchmark between conventional visual question answering and domain-specific decision support evaluation. A plausible implication is that the suite is designed to test both surface-level identification and agronomic reasoning rather than image classification alone.

2. Dataset composition and coverage

The benchmark contains 3,876 questions in total, composed of 2,018 open-ended QA pairs and 1,858 single-choice QA pairs (Yang et al., 5 Oct 2025). The single-choice portion is organized as two questions per image for 929 evaluation images, enabling image-level cross-consistency analysis.

Component	Count	Description
Open-ended QA pairs	2,018	Held-out images; generative answers
Single-choice QA pairs	1,858	Two questions per image
Evaluation images for single-choice set	929	Each image paired with two distinct questions
Total questions	3,876	Sum of both formats

The covered subdomains are explicitly listed as recognition, symptom or mechanism analysis, management recommendations, and simple multi-step reasoning. The examples provided clarify the intended semantic range. Recognition includes prompts such as identifying an insect species. Symptom or mechanism analysis includes questions such as determining the cause of yellow spots on leaves. Management recommendation questions ask how a farmer should treat a fungal infection. Simple multi-step reasoning includes prompts that connect observed deficiency signs to a fertilizer choice and a justification.

The benchmark is also described as the first medium-scale, rigorously de-duplicated, human-reviewed multimodal benchmark covering both open-ended reasoning and hard-distractor classification in agriculture (Yang et al., 5 Oct 2025). Because this characterization appears in the benchmark’s significance statement, it defines the benchmark not merely by size but by curation protocol and task structure.

3. Question formats and task design

The open-ended format consists of an image plus a free-form question with no predefined options. The model must generate an answer string, and the answer is normalized for synonyms and domain terminology (Yang et al., 5 Oct 2025). The supplied example shows tomato leaves with concentric necrotic rings and asks for both disease description and a cultural control strategy. The challenge arises from the need to jointly identify the disease and give a valid agronomic recommendation such as crop rotation or removal of infected debris.

The image-grounded single-choice format consists of an image, a stem question, and 3 to 5 candidate options. One option is correct, while the remaining options are hard distractors mined from confusable taxa or co-occurring conditions. The benchmark uses two distinct questions per image, and this design is explicitly linked to cross-consistency checks via $\mathrm{Acc}^+$.

The example given for the single-choice format depicts a rice field with pale yellow leaves in strips and asks which nutrient deficiency is shown. The options are nitrogen deficiency, potassium deficiency, phosphorus deficiency, and zinc deficiency. The challenge is that the distractors are other common deficiencies with similar symptoms, including potassium deficiency, which also causes chlorosis.

This dual-format design distinguishes answer generation from option selection. The open-ended items stress semantic adequacy and domain terminology under free-form generation, while the single-choice items test fine-grained discrimination under confusable alternatives. This suggests that AgriBench-VL-4K is intended to expose failure modes that would be obscured by a benchmark using only one answer format.

4. Evaluation metrics

AgriBench-VL-4K evaluates discriminative and generative outputs with multiple metrics (Yang et al., 5 Oct 2025). For single-choice questions, the benchmark reports per-question accuracy $\mathrm{Acc}$ and image-level cross-consistency $\mathrm{Acc}^+$. In prose, $\mathrm{Acc}$ is the fraction of correct questions over the total number of questions, while $\mathrm{Acc}^+$ is the fraction of images for which both associated questions are answered correctly.

The use of $\mathrm{Acc}^+$ is methodologically notable. Because each image in the single-choice split has two distinct questions, the benchmark can measure whether a model answers both correctly rather than succeeding on one by chance. The paper explicitly states that this two-question cross-consistency design better reflects true model understanding versus chance accuracy.

For open-ended responses, the benchmark reports BLEU-$n$, METEOR, and ROUGE-L-f. BLEU-$n$ is described as n-gram precision with brevity penalty; METEOR as a harmonic mean of unigram precision and recall with penalty; and ROUGE-L-f as an F-measure based on the longest common subsequence. These metrics provide lexical and sequence-level overlap signals for generated agricultural answers.

The details section also notes optional Precision, Recall, and F1 as common in QA evaluation, while stating that they are not explicitly used in the paper. This distinction is important: these quantities are presented as definable alternatives, not as part of the benchmark’s reported protocol. The benchmarking protocol itself specifies that discriminative results report $\mathrm{Acc}$ and $\mathrm{Acc}^+$, while generative results report BLEU, METEOR, and ROUGE-L.

5. LLM-as-a-judge framework

AgriBench-VL-4K incorporates an LLM-as-a-judge framework for open-ended evaluation (Yang et al., 5 Oct 2025). For each evaluation query, two candidate answers are collected, denoted A and B, from two models being compared. A judge LLM, exemplified by GPT-4o, receives the image-question pair and the two answers and is asked which answer is more accurate, agronomically sound, and evidence-based, with possible outputs “A,” “B,” or “Tie,” plus a brief justification.

To reduce order bias, the positions of A and B are swapped in a second pass, and the two judgments are averaged. The framework reports pairwise Win Rate excluding ties and also reports Tie Rate as the fraction of comparisons judged as “Tie.” In effect, this supplements lexical-overlap metrics with comparative preference judgments grounded in agronomic adequacy.

The illustrative example concerns treatment recommendations for powdery mildew on squash leaves. Answer A recommends applying a copper-based fungicide weekly until symptoms disappear, while answer B recommends systemic fungicides containing triazoles, rotating with sulfur sprays. The judge scores B above A because of the correct rotation advice and the domain-appropriate fungicide class.

This framework matters because open-ended agricultural responses can be partially correct, terminologically varied, or more or less actionable even when they do not maximize token overlap. The benchmark’s inclusion of judge-based preference therefore provides a second axis of evaluation beyond string matching. A plausible implication is that the benchmark is trying to distinguish agronomic usefulness from purely textual similarity.

6. Benchmarking protocol and reported significance

The benchmarking setup gives all vision-LLMs the same 2,018 open-ended and 1,858 single-choice items, consisting of images and questions (Yang et al., 5 Oct 2025). Depending on the task, models either generate an answer string or select from the given options. Reporting is standardized: single-choice performance uses $\mathrm{Acc}$0 and $\mathrm{Acc}$1, open-ended performance uses BLEU, METEOR, and ROUGE-L, and preference-based performance uses pairwise win rates from the LLM-judge framework.

The associated paper reports that AgriGPT-VL outperforms leading general-purpose VLMs on AgriBench-VL-4K, achieving higher pairwise win rates in the LLM-as-a-judge evaluation (Yang et al., 5 Oct 2025). It also states that ablation studies confirm consistent gains from alignment and GRPO refinement stages. These findings pertain to the benchmark’s use in comparative model evaluation rather than to its internal construction, but they indicate the intended role of the suite as a discriminative testbed for domain-specialized multimodal models.

The benchmark’s stated significance is broader than model ranking. It is presented as advancing agricultural vision-and-language research by combining open-ended reasoning, hard-distractor classification, cross-consistency analysis, and multi-metric assessment. This holistic positioning differentiates it from narrower evaluation sets that test only recognition or only answer generation.

7. Applications, interpretation, and limitations of scope

The benchmark’s listed use cases include automated pest and disease diagnosis tools that explain both what is wrong and how to manage it, decision support systems for nutrient management based on leaf symptom analysis, educational apps for extension agents and smallholder farmers, and continuous monitoring pipelines that combine camera data with language-based advice to flag emerging problems such as nutrient deficiencies or pathogen outbreaks (Yang et al., 5 Oct 2025).

These applications reflect the benchmark’s combination of visual interpretation and agronomic response generation. Because the benchmark includes management recommendations and simple multi-step reasoning, it is aligned with workflows in which correct identification alone is insufficient. Systems evaluated on AgriBench-VL-4K are expected to connect observation to action.

At the same time, the benchmark’s metrics define the boundaries of what it directly measures. Lexical metrics evaluate generated text against references; $\mathrm{Acc}$2 and $\mathrm{Acc}$3 evaluate correctness under fixed options; and judge-based win rates evaluate relative preference in paired comparisons. None of these, by themselves, establish downstream field efficacy. This suggests that benchmark performance should be interpreted as evidence of multimodal agricultural reasoning quality within the benchmark protocol, rather than as a complete validation of deployment readiness.

Taken together, the benchmark’s design choices—open-ended and single-choice questioning, cross-consistency via paired image questions, multi-metric reporting, and LLM-based comparative judgment—make AgriBench-VL-4K a technically specific benchmark for evaluating whether agricultural vision-LLMs can support real-world agricultural decision-making in a manner that is both diagnostically and agronomically informed (Yang et al., 5 Oct 2025).