Visual Diagnostic Assistant (VDA)
- Visual Diagnostic Assistant (VDA) is a family of systems that couples visual inputs with domain-specific reasoning, retrieval, and reporting to support diagnosis and operational decision making.
- VDA architectures employ explicit evidence grounding through multimodal methods—such as exact retrieval, deep learning explanations, and gaze-guided interactions—to ensure transparent and accountable outputs.
- Applications range from pathology and radiology to assistive and industrial settings, demonstrating a shift from autonomous screening towards interactive, auditable diagnostic aids.
Searching arXiv for recent and relevant papers on “Visual Diagnostic Assistant” and related uses of the acronym “VDA.” {"query":"\"Visual Diagnostic Assistant\" OR VDA arXiv", "max_results": 10} Visual Diagnostic Assistant (VDA) is used in recent literature to denote a family of systems that couple visual input with domain-specific reasoning, retrieval, reporting, or control in order to support diagnosis, interpretation, or operational decision making. Depending on the domain, a VDA may function as a pathology chatbot over histology images, a mixed-reality whole-slide viewer, a traceable battery-maintenance assistant grounded in charts and linked images, an integration-free radiology copilot that reads images from displays, an offline-first assistive system for visually impaired users, or a plug-and-play module for diagnosing GUI-agent failures (Lu et al., 2023, Veerla et al., 5 May 2025, Ru et al., 2 Jul 2026, Li et al., 1 Nov 2025, Florea et al., 2 Jul 2026, Ji et al., 30 Apr 2026). A recurring expectation is that the system should expose why a conclusion, localization, or action was produced. A terminological caveat is that the acronym is not unique: in solar energetic particle research, “VDA” denotes “velocity dispersion analysis,” not a visual assistant (Ding et al., 2016).
1. Concept and historical trajectory
Early work in this corpus is closer to automatic image-based screening than to interactive assistance. “Automatic diagnosis of retinal diseases from color retinal images” presents a teleophthalmology-enabled computer-aided diagnosis framework whose stated goal is to diagnose the type of retinal disease, automatically detect and segment diseased regions, and do so without human supervision or interaction. Its pipeline consists of fundus image capture, pre-processing, locating anatomic structures, detecting lesions, feature extraction, screening/classification, and output of diabetic retinopathy or drusen, with class-specific Auto Associative Neural Networks used through minimum squared reconstruction error (Jayanthi et al., 2010).
ViDi marks a different emphasis. Rather than presenting AI as a fully autonomous decision maker, it is explicitly framed as a radiologist assistant that combines a VGG-19 classifier, DeepSHAP explanations, and descriptive clustering so clinicians can inspect groups of chest X-rays with similar explanation patterns. Favorable and unfavorable saliency maps are used as complementary channels for why the model predicts the current class and which regions push it toward competing classes, and the framework is described as a bridge between AI and radiologists rather than a “silver bullet” (Ravi et al., 2020).
Later systems broaden the term toward interactive multimodal assistance. PathChat is a vision-language generalist AI assistant for human pathology, PathVis is a mixed-reality pathology assistant, the BESS assistant is a traceable fault-diagnosis system that grounds answers in operational and visual evidence, and FineState-Bench uses VDA as a diagnostic augmentation for isolating visual grounding failures (Lu et al., 2023, Veerla et al., 5 May 2025, Ru et al., 2 Jul 2026, Ji et al., 30 Apr 2026). This suggests a trajectory from automatic screening pipelines toward assistants that are expected to justify outputs, coordinate tools, and remain embedded in human workflows.
2. Architectural patterns and evidence grounding
A central architectural theme is explicit grounding in heterogeneous evidence rather than free-form generation. In the BESS assistant, user questions are first routed into business routes, then dispatched either to a single-agent fast path or a multi-agent deep-research path. Returned rows, retrieved passages, images, and charts are assembled into an evidence bundle, and the system logs route decisions, raw query plans, sanitized plans, final SQL, execution status, and evidence snapshots. Its hybrid retrieval module combines exact and semantic search through
and the database module uses a plan-then-execute workflow with deterministic sanitization before executable SQL is built and run on MySQL / TDengine (Ru et al., 2 Jul 2026).
PathChat exemplifies a tightly integrated multimodal architecture. It combines a foundational vision encoder initialized from UNI and further vision-language pretrained into CONCH-Large, a multimodal projector built from an attention pooling / Perceiver-style resampler with 128 learned latent queries plus a 2-layer MLP, and Llama 2 13B as the text backbone. The instruction-tuning objective is a causal language modeling loss conditioned on instruction text and image tokens, with the vision encoder frozen after pretraining and the projector and LLM finetuned end-to-end during instruction finetuning on PathChatInstruct (Lu et al., 2023).
GLARIFY shows a different grounding mechanism tailored to gaze-facilitated interaction. It operates on video keyframes, text query, and gaze-derived heatmaps aligned to the same frames, and injects gaze as a patch-level bias into the visual stream:
The design is deliberately lightweight, preserving the pretrained capabilities of Qwen2.5-Omni-3B while adding only about 0.0341% new parameters, and it treats gaze as a spatiotemporal heatmap sequence rather than as a single fixation label (Wang et al., 26 Sep 2025).
VisionCAD and FineState-Bench use VDA in a more modular sense. VisionCAD is organized as a six-stage pipeline—Vision Capturer, Screen Detector, Quality Enhancer, Modality Router, Diagnostic Engine, and Report Assistant—so that a camera-captured screen image can be transformed into a diagnostic-quality image and routed to task-specific models (Li et al., 1 Nov 2025). FineState-Bench, by contrast, defines VDA as a plug-and-play diagnostic aid that first generates a Description of the target UI element and then predicts a tight bounding box Localization Hint , enabling controlled w/ vs. w/o comparisons that isolate visual grounding as a failure source (Ji et al., 30 Apr 2026). A plausible implication is that VDA is better understood as an evidence-grounding pattern than as a single canonical model class.
3. Interaction paradigms and embodied interfaces
Some VDA systems are built around immersive or embodied interaction rather than conventional desktop interfaces. PathVis addresses the mismatch between gigapixel whole-slide images and the small view of a 2D monitor by placing the WSI into a mixed-reality workspace on Apple Vision Pro. It uses tile-based streaming with OpenSlide-generated multi-resolution pyramids, supports navigation from roughly a 5 mm field of view down to about 50 m detail, and relies on eye gaze as the primary pointing mechanism, tap for selection, pinch-and-drag for panning, two-handed pinch for zooming, and voice commands for similar-case search and conversational assistance. Similar patient search is backed by Yottixel and returns the top similar cases with in the prototype, while the conversational assistant uses OpenAI GPT-4o (Veerla et al., 5 May 2025).
GLARIFY addresses interaction ambiguity in smart-glasses and AR settings where a user looks at the world and asks underspecified questions such as “What is that made of?” or “What is he doing?” The paper quantifies the noisy character of gaze with an irrelevant gaze ratio that stays above 20% and rises sharply near the end of queries, reaching roughly 50%, motivating explicit spatiotemporal and noise-aware modeling rather than treating all fixations as equally meaningful (Wang et al., 26 Sep 2025).
VisionAId turns a commodity smartphone into a real-time visual assistant for people with visual impairment. All critical assistive functions run locally through ONNX Runtime, and feedback is intentionally multimodal: Romanian speech synthesis, voice commands, vibration, AR markers, spatial audio / beeping, and distance-proportional haptics. Its most distinctive interaction loop is personalized object retrieval: the user photographs an object from several angles, and the system later locates that specific instance in the environment and guides the user toward it with augmented-reality markers and step-by-step directional guidance (Florea et al., 2 Jul 2026).
A more physically embodied instantiation appears in the tethered aerial visual assistant literature. There, the assistant is an autonomous tethered UAV whose purpose is to reposition itself to provide the “cognitively best external viewpoint” for a teleoperated primary robot. Its planner explicitly balances viewpoint reward against motion risk through the utility objective
and the assistant is realized with tether-based localization, motion primitives, tether planning, and reactive 6-DoF visual servoing (Xiao, 2020). This broadens VDA beyond software assistants to include autonomous viewpoint managers.
4. Domain-specific instantiations
The term spans heterogeneous application domains, and the role of the assistant changes accordingly.
| System | Domain | Stated role or formulation |
|---|---|---|
| PathChat (Lu et al., 2023) | Human pathology | Vision-language generalist AI assistant for diagnosis, morphology, ancillary testing, and clinical interpretation |
| PathVis (Veerla et al., 5 May 2025) | Digital pathology | Mixed-reality visualization platform with similar patient search and multimodal conversational support |
| MAARTA (Awasthi et al., 18 Jun 2025) | Radiology education | Multi-agent framework that analyzes gaze patterns and radiology reports to provide personalized feedback |
| ViDi (Ravi et al., 2020) | COVID-19 chest X-ray support | Descriptive clustering framework using DeepSHAP and cluster-level saliency inspection |
| BESS assistant (Ru et al., 2 Jul 2026) | Battery energy storage O&M | Traceable fault-diagnosis assistant grounded in operational data, tables, documents, and linked images |
| VisionCAD (Li et al., 1 Nov 2025) | Radiology workflow | Integration-free radiology copilot using camera capture, restoration, modality routing, and report generation |
| VisionAId (Florea et al., 2 Jul 2026) | Assistive perception | Offline-first multimodal Android assistant with personalized object retrieval |
| FineState VDA (Ji et al., 30 Apr 2026) | GUI state setting | Diagnostic augmentation tool based on Description and Localization Hint |
In pathology and radiology, the dominant use case is image-grounded interpretation. PathChat can handle direct diagnosis from histology, morphologic description, differential diagnosis, IHC suggestion, molecular / ancillary testing, and multi-turn interactive case workups (Lu et al., 2023). MAARTA targets a narrower but highly specific educational problem: explaining whether a missed radiologic finding arose from missed fixation, brief dwell time, or incomplete knowledge by comparing teacher and student thought graphs and dispatching Perceptual Error Teacher agents (Awasthi et al., 18 Jun 2025). VisionCAD shifts attention from model design to deployment, arguing that camera-based capture can bypass HIS, RIS, PACS, EHR, and LIS integration barriers while still supporting automated analysis and report generation (Li et al., 1 Nov 2025).
Outside medicine, the BESS assistant treats VDA as a traceable O&M copilot that must connect alarms, cell-level and pack-level measurements, device topology, diagnostic tables, historical cases, maintenance manuals and procedures, and visual evidence such as screenshots, plots, diagrams, and photos (Ru et al., 2 Jul 2026). VisionAId treats VDA as an autonomy-support tool for blind or low-vision users, with metric monocular depth estimation, banknote detection, face recognition, color identification, and personalized object retrieval all serving practical mobility and access tasks (Florea et al., 2 Jul 2026).
Benchmarking work extends the term even further. In FineState-Bench, VDA is not the target agent but a controlled diagnostic intervention for studying state-conditioned grounding failures (Ji et al., 30 Apr 2026). In VIABLE, the VDA notion appears as a judge-like component that evaluates whether a candidate assistive response is correct, safe, helpful, and complete in visually impaired assistance settings (Zhao et al., 29 May 2026). This suggests that the literature uses VDA both for end-user assistants and for diagnostic scaffolds that analyze the assistants themselves.
5. Evaluation regimes and empirical findings
Reported results are strongly domain-specific and are not directly comparable, but they indicate what each community treats as success. In pathology, PathChat achieves 87% diagnostic accuracy on public multiple-choice pathology cases when relevant clinical context is provided, and 86.1% accuracy on 115 open-ended questions; in blind ranking by a board-certified pathologist, it wins 57.4% of head-to-head comparisons against GPT-4V (Lu et al., 2023). ViDi reports positive predictive value of 95% for COVID and 97% for pneumonia, and its explanation-space clustering reaches up to 80% homogeneity in COVID clusters (Ravi et al., 2020). VisionCAD reports F1-score degradation typically less than 2% relative to original digital images, with task-specific drops from 0.3 to 1.9 points, and report-generation metrics that remain within about one percentage point of the original-image baseline (Li et al., 1 Nov 2025).
In assistive and industrial settings, VisionAId reports that INT8 quantization reduces depth latency from about 1200 ms to about 491 ms, that the custom YOLO26n banknote detector reaches mAP@50 = 0.986 and mAP@50–95 = 0.961, and that calibrated metric depth yields mean error below 1 cm within the 0.5–3 m range (Florea et al., 2 Jul 2026). The BESS assistant reports a preliminary internal evaluation in which routing improves action accuracy from 20.0% to 70.0% while reducing latency from 87.45 s to 40.01 s, schema validation raises safe query-plan success from 0% to 100%, and multi-agent diagnostic reasoning improves quality from 3.60 to 4.80 despite longer latency (Ru et al., 2 Jul 2026).
GUI-oriented VDA studies show a different performance bottleneck. FineState-Bench reports that exact goal-state success remains low, with ES-SR@Int peaking at 32.8% on Web and 22.8% on average across platforms, while controlled VDA localization hints improve Gemini-2.5-Flash by +14.9 ES-SR@Int points on average (Ji et al., 30 Apr 2026). A related FineState-Bench version reports that the most advanced models achieve only 32.8% fine-grained interaction accuracy and that ideal visual localization reduces localization ambiguity by 85%, visual feature confusion by 72%, fine-grained state perception failure by 45%, and interaction context ignorance by 23% (Ji et al., 12 Aug 2025).
VIABLE evaluates VLM-as-a-Judge reliability rather than task execution. Across 312,365 judgment samples, the strongest judge, GPT-5.4, reaches only 52.6% single-injected diagnostic accuracy, 20.6% dual-injected full accuracy, and 65.9% consistency, while exhibiting a 94.2% self-preference rate; VIA-Judge-Agent improves both single- and dual-failure diagnosis by adding visual evidence extraction and taxonomy-guided verification (Zhao et al., 29 May 2026). The quantitative picture is therefore mixed: VDA systems can be highly effective within a bounded domain, but their reliability degrades when exact grounding, safety diagnosis, or judge impartiality becomes the primary target.
6. Limitations, controversies, and open questions
A repeated limitation is incomplete validation under real use. PathVis is explicitly described as a system/design paper without a formal user study with quantitative outcome measures, and its claimed reductions in cognitive strain or context switching are therefore design-motivated rather than experimentally established (Veerla et al., 5 May 2025). VisionAId reports only single-device evaluation and no structured usability study with blind or low-vision participants (Florea et al., 2 Jul 2026). PathChat is presented as a proof-of-concept assistant rather than a clinical diagnostic system, and MAARTA relies on a simulated student error dataset because no public dataset of student radiologist perceptual errors exists (Lu et al., 2023, Awasthi et al., 18 Jun 2025).
Another recurring issue is that better multimodal access does not by itself guarantee reliability. GLARIFY shows that gaze is informative but noisy, with substantial irrelevant fixation even when the user’s intent is coherent (Wang et al., 26 Sep 2025). FineState-Bench shows that providing localization hints can recover a substantial portion of failure, yet overall exact-state accuracy remains insufficient for reliable fine-grained interaction, and the paper explicitly states that VDA is diagnostic, not deployable (Ji et al., 30 Apr 2026). VIABLE further shows that current VLM judges are largely unreliable across effectiveness, impartiality, and stability axes, with open-source judges especially biased and adversarially fragile (Zhao et al., 29 May 2026).
Deployment constraints are equally domain-specific. VisionCAD depends on camera battery, compute, and imaging quality, and it notes privacy, security, and connectivity concerns when cloud-based models are used (Li et al., 1 Nov 2025). VisionAId notes that mid-range devices can be 2–3× slower and that MobileCLIP quantization remains unsolved because ONNX Runtime Android lacked the needed ConvInteger support (Florea et al., 2 Jul 2026). The BESS assistant addresses a different risk surface, emphasizing schema-constrained database access, route-table mappings, and audit logging so that SQL generation remains safe and traceable (Ru et al., 2 Jul 2026).
A common misconception is that VDA denotes a single mature product category. The literature instead presents a spectrum: automatic screening systems, human-in-the-loop explanatory tools, multimodal copilot frameworks, assistive mobile systems, educational feedback agents, plug-and-play diagnostic augmenters, and evaluation harnesses. Taken together, these works suggest that VDA research is moving toward systems that are evidence-grounded, domain-specialized, and auditable, but that robust deployment still depends on solving validation, grounding precision, and reliability under ambiguity.