AgriEval: AI Benchmarking for Agriculture

• AgriEval is a suite of benchmarks, frameworks, and evaluation protocols designed to rigorously assess AI systems in diverse agricultural applications.
• It covers language models, vision, multimodal systems, and ontology evaluation using detailed metrics such as MCQ accuracy, ROUGE-L, and IoU.
• Empirical findings reveal that region-specific adaptations and tailored evaluation strategies significantly enhance performance and reduce errors.

AgriEval

AgriEval denotes a family of rigorously designed benchmarks, frameworks, and evaluation protocols targeting AI systems for agriculture. The label “AgriEval” is used across independent research programs for: (a) evaluating LLMs on agricultural question answering, reasoning, and decision support; (b) benchmarking vision or multimodal systems on agronomic tasks; (c) assessing ontologies and structured knowledge representations; (d) toolkits and protocols for sustainability assessment, economic modeling, and regionalized advisory. Implementations span diverse languages, modalities, and domains, but share a focus on empirical, task-driven, and metric-rich evaluation tailored to agriculture’s unique requirements.

1. Dataset Composition and Evaluation Design

AgriEval benchmarks are characterized by high-quality, domain-specific data assets meticulously curated for broad coverage and real-world representativeness.

• Chinese Agricultural LLM Benchmark: The AgriEval suite for LLMs in Chinese agriculture includes 16,864 questions across six top-level agricultural domains (Plant Production, Forestry, Grass Science, Aquaculture, Animal Science & Technology, Traditional Chinese Herbology) and 29 subfields, derived from undergraduate/graduate exams and assignments. At least 100 questions per subfield ensure adequate sample depth. Formats comprise 14,697 multiple-choice (MCQ) and 2,167 open-ended generation questions, vetted by PhD-level experts (Cohen’s κ = 0.85), with distractor augmentation by GPT-4 and subsequent expert filtering (Yan et al., 29 Jul 2025).
• Region-Aware Advisory Benchmark: AgriRegion-Eval consists of 160 reference-annotated questions (12 subfields) plus human-vetted answers, derived from US extension documents, peer-reviewed agricultural texts, and field reports. Questions emphasize agronomic localization (80% North Carolina, 20% Southeast US), with precise regional tags and high inter-annotator consistency (κ > 0.85) (Fanuel et al., 10 Dec 2025).
• Visual Grounding Multi-Source Benchmark: In vision, the AgriEval (AgroVG) resource spans 10,071 image–query pairs (eight development, two test splits) drawn from ten expert-curated datasets and six “target families”—crop/weed, fruit, wheat head, pest, plant disease, and tree canopy—explicitly constructed to probe single-target, multi-target, and target-absent scenarios under both bounding-box and instance-mask protocols (Li et al., 21 May 2026).
• Passage Retrieval and Ontology Evaluation: Further instantiations (e.g. AgAsk, ontology evaluation) employ tens of thousands of domain-dense passages, structured via expert-written queries, answer rationales, and relevance judgments to facilitate retrieval-based and knowledge-integration tasks (Koopman et al., 2022, Goldstein et al., 2019).

The explicit inclusion of region, task type, cognitive scenario, and domain stratification is a defining trait throughout, enabling rigorous dissection of model performance and generalizability.

2. Methodologies and Metric Taxonomies

AgriEval protocols employ rigorous, multi-faceted quantitative metrics and evaluation schemas targeting both system-level and task-specific competencies.

• LLM and Multimodal LLMs:
  • MCQ Accuracy: $$\text{Accuracy} = \frac{\text{Number of Correct Predictions}}{\text{Total Number of Questions}} \times 100\%$$, with all-or-nothing marking for multi-answer MCQs.
  • Generation Quality: ROUGE-L computed as $$\text{ROUGE-L} = \frac{(1 + \beta^2) R_{lcs} P_{lcs}}{R_{lcs} + \beta^2 P_{lcs}}$$, where $R_{lcs}$, $P_{lcs}$ are recall/precision from the longest common subsequence.
  • Trust and Hallucination: Faithfulness (RAGAS), hallucination rate ($\text{Halluc.} = 1 - F$), and normalized trust scores.
• Visual Grounding:
  • Set-F1: $$F_{1, \tau}(q) = \frac{2\,\mathrm{TP}_\tau(q)}{2\,\mathrm{TP}_\tau(q) + \mathrm{FP}_\tau(q) + \mathrm{FN}_\tau(q)}$$ for bounding-box set matching; Macro-F1 aggregates across queries.
  • Mask Metrics: mIoU ($\mathrm{mIoU}$), cIoU, Dice score, and IoU@τ success rates; empty-query accuracy as a separate abstention metric.
• Retrieval and Modeling:
  • Precision@k, MAP, MRR, nDCG@k: Standard IR and ranking metrics for passage retrieval.
  • Regional Prioritization: Geo-injected re-ranking using a distance-decay score and final composite retrieval score.
  • Completeness/Accuracy (RAG QA): Completeness $$= \frac{\text{domain points covered}}{\text{total required}} \in [0,1]$$, accuracy as the proportion of correct answers judged against expert references.
  • Consumer Feedback: Perceived reliability $R = \frac{1}{N} \sum_{i=1}^N r_i$ (Likert scale).
• Meta-Frameworks and Economic/Sustainability Modeling:
  • Net Present Value (NPV), IRR, BCR, Payback Period: Standard discounted cash flow equations for investment assessment (Medici et al., 2020).
  • Sustainability Indices: Aggregated indicator score averages; for example, $S = (1/21)\sum_{i=1}^{21} I_i$ and its 7-indicator variant $$\text{ROUGE-L} = \frac{(1 + \beta^2) R_{lcs} P_{lcs}}{R_{lcs} + \beta^2 P_{lcs}}$$0 (Fonseca et al., 2019).

This multi-tiered metric stack is designed to capture factuality, completeness, reasoning, robustness, latency, trust, domain localization, and user-reported utility.

3. Empirical Findings and System Benchmarks

AgriEval resources function both as challenge problems for benchmarking and as testbeds for systematic ablation and scaling law studies.

• LLMs on Chinese Agricultural QA: The 2025 AgriEval study demonstrated that even the best LLMs (Qwen2-72B, Qwen-Plus) achieve only ~63% MCQ accuracy versus a human PhD baseline of 70.62%, with numerically intensive and pedigree reasoning tasks remaining the most challenging (often <40% accuracy). Scaling parameter count above 14B gives diminishing returns; instruction tuning delivers roughly 10 pp gains over base models. Chinese-native LLMs outperform English-centric ones by about 20 pp. Retrieval-augmentation delivers +4 pp on MCQs, especially for smaller architectures (Yan et al., 29 Jul 2025).
• Region-Aware RAG Systems: AgriRegion's geospatially aware retrieval mechanism yields significant improvements relative to vanilla models: F1 rises +12 pp, hallucination rates drop by 18 pp (absolute), trust scores increase from 0.72 to 0.85. Domain-wise, soil science, pathology, and irrigation benefit most (+0.19, +0.17, +0.21 F1), demonstrating the value of explicit regional constraints (Fanuel et al., 10 Dec 2025).
• Visual Grounding and Multimodal Models: AgroVG AgriEval highlights persistent limitations: best multi-target Set-F1 remains ≤0.40, fine-grained instance-mask mIoU <0.32 (even for LISA-13B), and positive-query mask [email protected] ≤0.17 (PSALM). Key deficiencies are set completeness and abstention calibration, with models either hallucinating on empty queries or missing true targets. Future research recommendations emphasize set-level recall modeling, existence-aware loss, mask refinement, and richer referring-expression diversity (Li et al., 21 May 2026).
• Ontology Evaluation: The AgriEval ontology framework emphasizes mapping the ontology’s purpose to an evaluation plan (criteria-based, gold-standard, application-based, data-driven) and recognizes the limited use of closed-form metrics within ontology assessment, favoring explicit counts of modeling errors and user-study metrics (Goldstein et al., 2019).
• Decision Support Agents and Passage Retrieval: In retrieval-based QA, neural rerankers (monoBERT, TILDEv2) surpass term-scoring baselines (BM25) in early-precision (NDCG@5 achieved 0.64 monoBERT vs 0.35 BM25), with substantial efficiency gains when using lightweight rerankers. Latency remains a deployment bottleneck for cross-encoder models in real-time chat, leading to TILDEv2 adoption for practical deployments (Koopman et al., 2022).

4. Analysis of Influences, Limitations, and Best Practices

AgriEval studies consistently reveal context- and architecture-sensitive performance dependencies, limitations, and suggest actionable improvements.

• Influencing Factors:
  • Model scale improves performance, but most gains for LLMs are realized by ~14B parameters, with larger sizes offering only modest improvement.
  • Instruction tuning, regional and domain adaptation, and retrieval augmentation are consistently beneficial.
  • Prompting strategies such as chain-of-thought (CoT) boost inference (+9.8 pp) but can degrade factual recall (−4.3 pp).
  • Position-permutation during MCQ training helps mitigate positional bias (noted 7 pp drop when options are shuffled).
• Limitations and Gaps:
  • No system achieves human-expert parity on comprehensive, multi-domain agricultural QA.
  • Visual grounding models show steep drop-offs for class-uniformity (CV of per-class F1 ranges 26%–58%), mask granularity, and abstention.
  • RAG and generative QA systems are highly sensitive to document embedding quality, region-label provenance, chunking logic, and preprocessing pipelines.
  • Small models trade off completeness and factuality for latency (PORO-2-8B completeness 0.68, L=3.1s vs PORO-2-70B completeness 0.72, L=5.4s) (Hasan et al., 2 Feb 2026); overlong inference times (>5–6s) reduce user trust.
• Best Practices:
  • Prioritize robust data ingestion, preprocessing, and native-language retrieval pipelines before aggressive model scaling.
  • Embed lightweight, in-app feedback and analytics loops from early prototyping; user ratings should directly inform iteration priorities.
  • Modular architecture (ingestion, indexing, generation, UI separation) underpins rapid ablation, tuning, and swap-in of model/ranking modules.
  • Integrate geo-metadata and regional scoring for context-specific advice in advisory systems; suppress hallucinations and increase trust by grounding all recommendations to authoritative, location-tagged sources (Fanuel et al., 10 Dec 2025).

5. System Architectures, Deployment, and Reproducibility

AgriEval initiatives typically publish data, code, and protocols with explicit modular decomposition to enable reproduction, extension, and customization.