AgriDoctor: A Multimodal Agricultural Assistant

• AgriDoctor is a family of intelligent agricultural assistants combining icon interfaces, image analysis, and conversational advisory modules.
• It employs image processing, lesion detection, and neural network-based classification to diagnose diseases with high accuracy.
• Recent multimodal developments integrate bilingual retrieval, intent routing, and LLM fusion to provide localized, actionable agricultural guidance.

AgriDoctor denotes an agricultural intelligent-assistance paradigm whose published instantiations span icon-based information retrieval, image-based crop disease diagnosis, closed-domain advisory chatbots, and modular multimodal assistants. In the arXiv literature, the name appears explicitly in a system that combines a hierarchically organized icon interface with automated disease identification for rice leaves (Mittal et al., 2014), and later in a bilingual multimodal framework integrating intent routing, disease classification, lesion detection, knowledge retrieval, and LLM-based response fusion (Zhang et al., 21 Sep 2025). Closely related work extends the same problem space through expert systems, retrieval agents, and agriculture-specific conversational and vision-LLMs (Darapaneni et al., 2022).

1. Conceptual scope and problem setting

AgriDoctor systems address a recurrent agricultural bottleneck: the need to provide timely, practically actionable expertise when direct access to agronomists, plant pathologists, or extension specialists is limited. The early formulation is explicitly motivated by the fact that many farmers cannot comfortably use text-heavy web interfaces because of language barriers and low digital literacy, while still needing fast access to expert agricultural knowledge and disease diagnosis (Mittal et al., 2014). The later multimodal formulation reframes the same problem in terms of sustainable agriculture and food security, arguing that real-world crop assistance requires not only visual recognition but also localization of symptomatic regions, agricultural knowledge interaction, and bilingual support (Zhang et al., 21 Sep 2025).

The term does not refer to a single fixed architecture. Rather, the literature uses it for systems that differ substantially in modality and reasoning strategy. One line emphasizes language-independent interaction through icons and symptom-driven image analysis (Mittal et al., 2014). Another emphasizes conversational intent detection, entity extraction, and database-backed remedies (Darapaneni et al., 2022). A more recent line uses modular multimodal reasoning, with separate tools for routing, classification, detection, retrieval, and language generation (Zhang et al., 21 Sep 2025). This suggests that AgriDoctor is best understood as a family of agricultural decision-support systems rather than a single software artifact.

2. The early icon-based and image-processing formulation

The 2014 AgriDoctor system is organized as a two-part platform for agriculture users who may not be digitally literate: an icon-based information retrieval interface and an automatic crop disease identification module (Mittal et al., 2014). The information-retrieval side replaces typed search with an iconic query-building approach. Farmers select hierarchically organized icons representing crops and agricultural topics, and the selection is transformed into an intermediate query representation for backend retrieval. The example interface includes crop icons such as cotton, soybean, maize, wheat, groundnut, and millet, and branches into topical categories including high yielding varieties, problem identification, disease management, insect management, weed management, manure and fertilizers, crop protection, navigation, FAQ, contact us, feedback, blog, admin login, and video gallery (Mittal et al., 2014).

The disease-identification side focuses on rice leaf diseases, specifically leaf blast and brown spot. Its workflow begins when a farmer brings a diseased leaf image to a Kisan help center, where an operator uploads the image. Training images were collected from Google Images and normalized to 200 × 200 pixels, with brightness and contrast enhancement. The pipeline then converts RGB to the HSI color space, forms a binary saturated mask by thresholding the saturation image, masks the hue image, generates a histogram, performs threshold-based segmentation, detects spots using component labeling, and extracts morphological descriptors such as major axis length, minor axis length, bounding box area, eccentricity, perimeter, solidity, Euler number, orientation, area, extent, diameter, and convex hull (Mittal et al., 2014).

Classification is performed by a feed-forward neural network with two hidden layers. The training set contains 25 images per disease class and the test set 20 images per class; because each spot is treated as a separate input sample, the number of training samples increases to 1008. Final image classification uses a voting scheme over spot-level predictions. On the reported testing scheme, the system achieves an overall accuracy of 89.23% (Mittal et al., 2014). The diagnostic output is not limited to a label: the system can display structured fields such as crop name, part affected, disease English name, disease pathogen name, disease life cycle, initial symptoms, final symptoms, control measures, and prevention techniques. Even in this early formulation, diagnosis is coupled to an advisory layer rather than treated as a purely visual classification problem (Mittal et al., 2014).

3. Conversational, symbolic, and retrieval-based extensions

A second major trajectory in AgriDoctor-like research is the transition from icon and image pipelines to conversational and symbolic interfaces. The paper "Template-based Chatbot for Agriculture Related FAQs" describes a chatbot in which template-based questions are answered by AIML while LSA is used for other service-based questions, with the stated goal of assisting farmers by dealing with industry problems conveniently and efficiently (Zhang et al., 2021). This places question answering alongside disease diagnosis as a central agricultural assistance function.

The paper "LSTM-RASA Based Agri Farm Assistant for Farmers" develops a closed-domain conversational assistant built on Rasa Open Source, with Rasa NLU for tokenization, featurization, intent classification, and entity extraction, and Rasa Core for dialogue management (Darapaneni et al., 2022). The system uses DIET for joint intent and entity modeling, TED policy for dialogue action selection, and an SQLite knowledge base storing plant protection, nutrient management, and agricultural officers’ contact information. Its data derive from Kisan Call Center records from data.gov.in, using three years of Tamil Nadu data. The analysis reported in the paper includes that about 51% of queries were for RABI crops, Paddy was the most frequent crop, the data came from 32 districts with the top 15 districts contributing about 75% of all queries, and 63 query types were observed, with the top 15 covering about 93% of the data (Darapaneni et al., 2022). In this formulation, the AgriDoctor function is realized through intent detection, entity extraction, and structured remedy retrieval rather than image analysis.

A more explicitly evidence-oriented extension appears in AgAsk, an agent that answers natural language agriculture questions by mining scientific documents (Koopman et al., 2022). AgAsk uses a Telegram client, Macaw for conversational information seeking, BM25 first-stage retrieval of 1000 candidates, and TILDEv2 reranking. The released Ag-valuate collection contains 210 topics, 86,846 documents, and 9,441,693 passages, with 3,948 judged passages. The paper reports that monoBERT and TILDEv2 are far more effective than BM25 on early precision, and highlights monoBERT Success@3 = 0.96, meaning 48 out of 50 test topics had a relevant passage in the top 3 (Koopman et al., 2022). In this line of work, AgriDoctor-like behavior is grounded in passage retrieval from scientific evidence rather than direct generative response.

Symbolic expert-system approaches remain relevant as well. A SWI-Prolog expert system for diagnosing diseases and pests in rice, tobacco, tomato, pepper, corn, cucumber and bean crops uses predicates, production rules, a crop-specific knowledge base, and yes/no symptom questioning to infer the most likely pest or disease and display explanatory information (Carbó et al., 2020). This preserves the classical expert-system pattern of condition-action rules within the broader AgriDoctor problem setting.

4. Multimodal AgriDoctor and domain-adapted reasoning

The 2025 paper "AgriDoctor: A Multimodal Intelligent Assistant for Agriculture" presents the most explicit modern formulation of the name (Zhang et al., 21 Sep 2025). The system has five components: a router, a disease classifier, a detector, a knowledge retriever, and an LLM used for output fusion and reasoning. The router first detects whether the query is Chinese or English, applies a language-specific BERT intent classifier, and dispatches one of three supported intents: disease classification, disease detection, or knowledge-grounded question answering/agricultural knowledge retrieval. Only one tool is invoked for each interaction, after which the LLM fuses the user query with the selected tool’s output at the prompt level (Zhang et al., 21 Sep 2025).

The visual modules are domain-adapted rather than generic. The disease classifier is a CLIP-ViT-based recognition model with a supervised trainable classification head. The detector is a YOLOv12-based lesion localization model trained to output bounding boxes for diseased regions. The retrieval component operates over a bilingual agricultural knowledge base, using language-specific BERT embeddings, Jieba keyword extraction for Chinese-style query summarization, mean-pooled token embeddings for knowledge entries, and FAISS indexing with $L_2$ similarity. The training split is 8:1:1 into train, validation, and test. Router accuracy is reported as 99.00% for English and 98.82% for Chinese; the classifier reaches 96.2% classification accuracy on the test set; and the detector reports Precision 0.876, Recall 0.868, mAP@50 0.893, and mAP@50–95 0.761 (Zhang et al., 21 Sep 2025).

AgriDoctor’s data foundation is AgriMM, which contains 400,000 annotated disease images, 831 expert-curated knowledge entries, and about 300,000 bilingual prompt instances for intent-driven tool selection. The images cover 29 crops and 138 disease types, with a long-tailed class distribution; the knowledge entries describe symptoms, transmission mechanisms, physiological effects, pests, and integrated management strategies (Zhang et al., 21 Sep 2025). The classifier uses a weighted cross-entropy design for class imbalance, with class weight $w_y = \min\left(\frac{N}{N_y}, 10\right)$ and a capped minority upweighting strategy (Zhang et al., 21 Sep 2025).

On a balanced 300-sample evaluation set with 100 instances each for classification, detection, and knowledge QA, AgriDoctor achieves overall score 0.8630, compared with 0.6593 for Qwen2.5-VL and 0.7043 for GPT-4o-mini. By task, it scores 0.8074 on classification, 0.8050 on detection, and 0.9765 on knowledge retrieval (Zhang et al., 21 Sep 2025). The largest reported gain is therefore in knowledge-grounded QA, which is consistent with the system’s explicit modularization and curated bilingual knowledge base.

Related multimodal systems support the same design direction. Agri-LLaVA constructs 391,785 agricultural image-text pairs covering over 221 pest/disease categories and about 6,000 high-quality multimodal conversations, then adapts LLaVA-1.5 through a two-stage knowledge-infused training scheme; its best reported VQA benchmark average is 60.05, about 4.87 points higher than LLaVA (Wang et al., 2024). AgroGPT builds a 70k expert-tuning dataset called AgroInstruct from vision-only agricultural datasets and shows that expert tuning raises disease ID from 10.27% to 51.37% and insect ID from 0.25% to 35.89% on AgroEvals (Awais et al., 2024). These systems are not named AgriDoctor, but they occupy the same multimodal agricultural-assistant space.

5. Data regimes, evaluation practices, and operational design patterns

Across the AgriDoctor literature, performance is tightly coupled to the construction of domain-specific datasets and knowledge bases. The early icon-and-image system uses small, disease-specific datasets and morphological lesion analysis (Mittal et al., 2014). Later systems move toward larger public corpora and curated knowledge repositories. SMARD uses PlantVillage, described as containing 54,309 images of healthy and diseased leaves from 14 crops, and reports using 16,011 images spanning 9 disease classes plus 1 normal class. Its web-application-based platform couples image diagnosis with expert contact through text message, voice call, or video call, as well as guidance on crop care, seed selection, fertilizer and pesticide use, product purchasing, and bank loans; it reports 97.3% classification accuracy and 96% F1-score in crop disease classification (Debnath et al., 2024).

Retrieval-augmented agricultural assistants push this further by treating agricultural advice as a corpus-grounded QA problem. AgroLLM organizes an agricultural database into four domains, embeds overlapping text chunks, stores them in FAISS, and evaluates three models over 108 domain-related questions. ChatGPT-4o Mini with RAG is reported as the best-performing configuration, with MRR 0.93, Recall@10 91.5%, and 93% accuracy, compared with 88.8% accuracy without RAG (Samuel et al., 28 Feb 2025). AgriRegion, a region-aware RAG framework, restricts its knowledge base to verified local agricultural extension services, uses geospatial metadata injection and region-prioritized re-ranking, and evaluates on AgriRegion-Eval with 160 domain-specific questions across 12 agricultural subfields. Its reported results include EM 0.76, F1 0.82, BLEU-4 0.65, ROUGE-L 0.72, BERTScore 0.90, and RAGA-Precision 0.86, together with hallucination reductions of 10–20% relative to state-of-the-art LLM systems (Fanuel et al., 10 Dec 2025).

Image reliability is another recurring operational pattern. AgroPath inserts a no-reference image quality assessment stage before disease classification, accepting an image only if quality > 80%. Its pipeline consists of image capture, leaf bounding-box extraction, resizing to $224 \times 224 \times 3$, image quality assessment, quality thresholding, and disease classification using EfficientNet. On a self-collected dataset of 227 images, the paper reports that AgroPath discarded 41 low-quality images, leaving 186 for classification, and achieved 99.42% accuracy for AgroPath-mobile (2209.12443). Plant Doctor extends the operational scope from still images to video, using YOLOv8-PD for leaf detection, DeepSORT for tracking, Laplacian variance and SSIM for best-frame selection, and DeepLabV3Plus-PlantDoctor for segmentation-based damage quantification (Marques et al., 2024). These designs indicate that AgriDoctor-like systems increasingly treat input quality control, evidence retrieval, and task routing as first-class components rather than peripheral engineering choices.

6. Limitations, misconceptions, and likely directions

A common misconception is that AgriDoctor denotes a single mature product with a stable architecture. The literature shows the opposite. The 2014 system is preliminary, limited to two rice diseases, and depends on operator-assisted upload at a help center; it also warns that icon misinterpretation can lead to incorrect queries and that icon optimization using synonyms and a thesaurus is a future improvement (Mittal et al., 2014). The 2025 multimodal AgriDoctor is larger and more capable, but its authors note that AgriMM may still not sufficiently cover rare crop species, uncommon disease variants, and region-specific conditions, and that multilingual support is currently limited to Chinese and English (Zhang et al., 21 Sep 2025).

Another misconception is that benchmark accuracy alone is sufficient for deployment. Related systems document failure modes that arise from poor image capture, narrow language support, incomplete corpora, or locality mismatch. AgroPath improves robustness by discarding poor images rather than attempting to diagnose them (2209.12443). The Rasa-based assistant supports only English and may require handoff to a human when users ask follow-up or return questions that were not anticipated during design (Darapaneni et al., 2022). AgriRegion shows that generic LLM advice can be scientifically sound in one region but wrong or harmful in another because of differences in soil, climate, crop calendars, pest pressure, storage conditions, and regulations (Fanuel et al., 10 Dec 2025). AgAsk notes that passage retrieval still falls short of full answer generation and does not yet use user profile or contextual factors such as location, weather, market conditions, or user expertise (Koopman et al., 2022).

The literature therefore points toward a composite design rather than a single-model solution. This suggests that future AgriDoctor systems are likely to combine multimodal diagnosis, evidence-backed retrieval, region-aware ranking, bilingual or multilingual interaction, input-quality control, and explicit escalation for uncertain or out-of-distribution cases. The trajectory from icon-based retrieval and two-class lesion analysis (Mittal et al., 2014) to tool-routed multimodal reasoning over curated agricultural knowledge (Zhang et al., 21 Sep 2025) indicates a broad shift from isolated prediction modules toward integrated agricultural decision-support ecosystems.