DIVE: Descriptive Interpretation of Visual Expression

• DIVE is a framework that converts visual elements in scientific literature into structured, descriptive representations for enhanced data extraction and semantic reasoning.
• It employs multi-agent workflows and intermediary textual embeddings to replace raw visuals with explicit descriptions, enabling precise schema mapping and quantitative evaluation.
• Practical implementations of DIVE show improved accuracy in tasks like figure classification, visual commonsense generation, and biomedical interpretation while enhancing model transparency.

Descriptive Interpretation of Visual Expression (DIVE) denotes a family of research efforts that convert visual material into explicit descriptive forms for extraction, reasoning, generation, or explanation. The phrase is used most explicitly for a multi-agent workflow that reads figures and tables in scientific literature for hydrogen storage materials discovery (Zhang et al., 18 Aug 2025), but closely related work across vision-language modeling, information visualization, interpretable machine learning, and visual commonsense generation relies on the same operational principle: replacing or supplementing raw visual input with structured descriptions, sense-anchored abstractions, or explanation-bearing intermediates (Wang et al., 2024, Menon et al., 2022, Park et al., 2024).

1. Scope and terminology

The literature does not define DIVE as a single standardized framework. Instead, the label appears in multiple, partially overlapping senses. In materials informatics, DIVE refers to a workflow that “systematically reads and organizes experimental data from graphical elements in scientific literatures” and uses a multi-agent design to improve extraction accuracy and coverage (Zhang et al., 18 Aug 2025). In visual commonsense generation, DIVE denotes “Descriptive and dIverse Visual commonsense gEneration,” a framework intended to improve the descriptiveness and diversity of generated inferences for images (Park et al., 2024). In biomedical model interpretation, LCS-DIVE denotes the “Learning Classifier System Discovery and Visualization Environment,” an automated interpretation pipeline built around rule-based machine learning and visualization (Zhang et al., 2021).

This multiplicity of usage is consistent with two broader conceptual backdrops. The “Visual Expression Process” organizes visual comprehension into the stages Pre-attentive Stimuli, Analytical Perceptions, Abstract Patterns, and Decision Support (Rodrigues-Jr et al., 2015). Separately, interpretation analysis has been formalized through a black-box interpretation function $f_b : I_b \rightarrow M_b$, where a perspective maps inputs such as images to meanings or outputs (Blandfort et al., 2018). Taken together, these frameworks place DIVE-style systems within a larger agenda: making the transition from visual signal to semantic or decision-relevant content explicit.

A recurrent misconception is to treat DIVE as identical with conventional captioning. The surveyed work is broader. Some systems target figures and tables rather than natural images (Zhang et al., 18 Aug 2025); some evaluate idiomatic versus literal grounding under visual abstraction (He, 19 Apr 2026); some reason over vector graphics through symbolic descriptions (Wang et al., 2024); and some expose model-internal or expert-facing rationales through saliency maps, clustering, or descriptor scores (Ravi et al., 2020, Menon et al., 2022).

2. Descriptive intermediates as the central method

A defining methodological pattern is the insertion of an intermediate descriptive representation between perception and downstream reasoning. In the materials-science DIVE workflow, PDFs are parsed into text and images using MinerU; a lightweight model performs figure caption classification; a multimodal LLM produces a “descriptive embedding of figures”; and a specialized LLM performs final textual data extraction into dictionaries or tables in JSON format (Zhang et al., 18 Aug 2025). The key step is explicit replacement: “The figure is effectively ‘replaced’ in the document with this textual description.” This makes graphical evidence amenable to batch extraction, schema mapping, and benchmarking.

A closely related pattern appears in the Visually Descriptive LLM (VDLM), which introduces Primal Visual Description (PVD) as an intermediate textual representation for vector graphics reasoning. Raster images are converted to SVG, SVG is translated to PVD, and a general-purpose LLM performs reasoning over the resulting text without direct visual input (Wang et al., 2024). PVD encodes primitive attributes such as shape, position, and measurement, thereby disentangling perception from reasoning.

Descriptor-mediated classification provides another variant of the same design. In “classification by description,” category names are replaced by LLM-generated descriptor sets, and class prediction is computed by aggregating descriptor-level similarity scores:

$$s(c, x) = \frac{1}{|D(c)|} \sum_{d \in D(c)} \phi(d, x).$$

The predicted class is $\arg\max_{c \in C} s(c,x)$ (Menon et al., 2022). This creates an explicit linguistic bottleneck: the system must check for descriptive features rather than broad categories.

For non-literal language, ViPE functions as a “meta-prompt” generator that rewrites figurative or implicit text into visualisable descriptions suitable for text-to-image systems (Shahmohammadi et al., 2023). In visual commonsense generation, DIVE alters both the training data and objective through generic inference filtering and contrastive retrieval learning so that generated inferences become more descriptive, specific, and diverse (Park et al., 2024). Across these settings, description is not merely an explanation appended after inference; it is a computational object that reconfigures the task itself.

3. Measurement and evaluation

DIVE-style research is notable for treating descriptive interpretation as measurable rather than purely qualitative. The most explicit metricization appears in DIVA, a benchmark for ambiguous noun compounds that provides paired literal and idiomatic visualizations and evaluates semantic anchorage under photo and icon conditions (He, 19 Apr 2026). The signed literal bias is defined as

$$b(t) = \mathcal{S}(v_{lit}, t) - \mathcal{S}(v_{id}, t),$$

and the Semantic Alignment Gap is

$$\Delta(t) = \left| \mathcal{S}(v_{lit}, t) - \mathcal{S}(v_{id}, t) \right| = |b(t)|.$$

Positive $b(t)$ indicates literal preference, and smaller $\Delta$ indicates that literal and idiomatic interpretations are treated as similarly compatible with the ambiguous anchor (He, 19 Apr 2026).

In the materials DIVE workflow, evaluation is centered on extraction accuracy and completeness. AI-extracted and human-extracted JSON outputs are aligned by a shared embedding model with unit-standardized relative errors. The scoring algorithm assigns 10, 8, 5, or 0 points per item for relative errors $\leq 10\%$, $10$–$20\%$, $$s(c, x) = \frac{1}{|D(c)|} \sum_{d \in D(c)} \phi(d, x).$$0–$$s(c, x) = \frac{1}{|D(c)|} \sum_{d \in D(c)} \phi(d, x).$$1, and $$s(c, x) = \frac{1}{|D(c)|} \sum_{d \in D(c)} \phi(d, x).$$2, respectively; completeness is

$$s(c, x) = \frac{1}{|D(c)|} \sum_{d \in D(c)} \phi(d, x).$$3

and the total score is 100 points, split into 50 for accuracy and 50 for completeness (Zhang et al., 18 Aug 2025).

In commonsense generation, descriptiveness and diversity are evaluated through Length, Yngve Depth, Dist-2, Dist-3, Image Retrieval Recall, Entropy, Unique, and Novel, alongside human judgments of plausibility, descriptiveness, and diversity (Park et al., 2024). Human-centered applications use still different metrics. In the diary study of MLLM-enabled visual interpretation for Blind and Low Vision people, trust and satisfaction were measured on 5-point scales, correctness was coded as correct, incorrect, partial, or unanswered, and follow-up dialogue behavior was quantified by conversation rate and average conversation length (Penuela et al., 7 Mar 2025).

These evaluation regimes are heterogeneous, but they converge on a shared requirement: descriptive quality must be assessed jointly with semantic correctness, retrieval specificity, or task success. A plausible implication is that DIVE is best understood as a measured interface between visual evidence and explicit semantic commitments, rather than as a single benchmark family.

4. Representative implementations across domains

The breadth of the literature is most visible in the domains to which DIVE-like methods have been applied.

System	Domain	Descriptive mechanism
DIVE (Zhang et al., 18 Aug 2025)	Hydrogen storage materials	Caption classification, descriptive embedding, textual extraction
DIVA (He, 19 Apr 2026)	Idiomatic noun compounds in VLMs	Paired sense-anchored images, $$s(c, x) = \frac{1}{|D(c)|} \sum_{d \in D(c)} \phi(d, x).$$4, $$s(c, x) = \frac{1}{|D(c)|} \sum_{d \in D(c)} \phi(d, x).$$5
VDLM (Wang et al., 2024)	Vector graphics reasoning	SVG $$s(c, x) = \frac{1}{|D(c)|} \sum_{d \in D(c)} \phi(d, x).$$6 PVD $$s(c, x) = \frac{1}{|D(c)|} \sum_{d \in D(c)} \phi(d, x).$$7 LLM reasoning
DIVE (Park et al., 2024)	Visual commonsense generation	Generic inference filtering, contrastive retrieval learning
LCS-DIVE (Zhang et al., 2021)	Biomedical classification	FT clustermaps, rule clustermaps, rule co-occurrence networks
ViDi (Ravi et al., 2020)	COVID-19 chest X-rays	DeepSHAP saliency maps, explanation-space clustering
ViPE (Shahmohammadi et al., 2023)	Figurative text visualization	Visual elaboration generation from text

In materials discovery, DIVE processed over 4,000 primary publications from 1972–2025 and extracted 30,435 unique entries into the DigHyd database (Zhang et al., 18 Aug 2025). It was reported to improve extraction accuracy by 10–15% over best commercial models and by more than 30% over open-source models, and the downstream DigHyd AI agent used an XGBoost regressor with $$s(c, x) = \frac{1}{|D(c)|} \sum_{d \in D(c)} \phi(d, x).$$8 and RMSE $$s(c, x) = \frac{1}{|D(c)|} \sum_{d \in D(c)} \phi(d, x).$$9 for inverse design (Zhang et al., 18 Aug 2025). The pipeline was demonstrated on solid-state hydrogen storage materials and described as broadly transferable across diverse materials.

In symbolic reasoning over graphics, VDLM showed that a structured textual abstraction can outperform monolithic LMMs on tasks such as angle classification, length comparison, and maze solving, while making the split between perception errors and reasoning errors explicit (Wang et al., 2024). In figurative and non-literal language, ViPE scaled visual elaboration generation through a large synthetic dataset of lyric-elaboration pairs and used symbolic knowledge distillation from GPT-3.5 to lightweight GPT-2 models (Shahmohammadi et al., 2023). In visual commonsense generation, DIVE was reported to achieve human-level descriptiveness and diversity on Visual Commonsense Graphs, with higher image-retrieval specificity and greater proportions of unique and novel inferences than baseline models (Park et al., 2024).

The commonality across these systems is methodological rather than domain-specific. Each replaces an opaque visual-to-output mapping with an intermediate descriptive layer that can be inspected, scored, edited, or optimized.

5. Interpretability, expert use, and human-facing systems

A major strand of DIVE research is explicitly interpretability-oriented. LCS-DIVE uses ExSTraCS feature-tracking scores and IF:THEN rules to characterize feature importance, additive patterns, epistatic patterns, heterogeneous associations, and model-driven instance subgroups (Zhang et al., 2021). FT scores are normalized to $\arg\max_{c \in C} s(c,x)$0, hierarchical clustering uses Pearson correlation and Ward linkage, cluster cuts are supported by Monte Carlo significance testing, and the elbow method is used to suggest the number of clusters (Zhang et al., 2021). The output is not only predictive but descriptive: heatmaps and networks reveal which features co-occur, specialize, or separate into heterogeneous submodels.

ViDi pursues a related goal in radiology. DeepSHAP produces favorable and glum saliency maps, K-Means++ clusters images in the space of explanation vectors, and radiologists inspect central images and cluster-level overlays rather than isolated predictions (Ravi et al., 2020). The system was proposed as a radiologist assistant for COVID-19 streamline diagnostic, emphasizing human-machine interaction and expert decision making rather than replacement.

Human-facing implications also appear in accessibility research. In the diary study of VisionPal, participants considered the application’s interpretations trustworthy with mean $\arg\max_{c \in C} s(c,x)$1 and satisfying with mean $\arg\max_{c \in C} s(c,x)$2; 75% of diary entries led to follow-up conversations; average conversation length was $\arg\max_{c \in C} s(c,x)$3 messages; and of 80 questions, 58 were correct, 15 incorrect, 5 partial, and 2 unanswered (Penuela et al., 7 Mar 2025). The study also reported trust in high-stakes scenarios, including medical dosage advice. This result is technically significant because it separates descriptive fluency from factual reliability.

Mechanistic analyses of VLMs reinforce the same point from the model side. Attention analysis showed that query tokens can store global image information, middle layers dominate cross-modal information flow, and fine-grained attributes are extracted from image tokens in a spatially localized manner (Kaduri et al., 2024). This helps explain why DIVE-like descriptive bottlenecks can work: global summaries and localized evidence are separable and can be routed differently through the model.

6. Limitations, controversies, and prospective directions

Several tensions recur across the literature. One is the tension between descriptiveness and correctness. In BLV visual interpretation, high trust and satisfaction did not perfectly track accuracy, and plausible but incorrect answers remained a concern (Penuela et al., 7 Mar 2025). In descriptor-based classification, LLM-generated descriptors can be wrong or irrelevant, although the advantage is that they remain visible and editable (Menon et al., 2022).

A second tension concerns realism versus abstraction. DIVA found a consistent Literal Superiority Bias across 8 recent VLMs, with all models showing positive $\arg\max_{c \in C} s(c,x)$4 in the ambiguous-anchor condition and with high-fidelity Photo inputs yielding significantly higher $\arg\max_{c \in C} s(c,x)$5 than Icon inputs (He, 19 Apr 2026). For all models, the reduction in $\arg\max_{c \in C} s(c,x)$6 under iconographic abstraction had Wilcoxon signed-rank $\arg\max_{c \in C} s(c,x)$7, and sense-specified 5-way selection accuracy improved from $\arg\max_{c \in C} s(c,x)$8 for discriminative models, $\arg\max_{c \in C} s(c,x)$9 for open-generative models, and $$b(t) = \mathcal{S}(v_{lit}, t) - \mathcal{S}(v_{id}, t),$$0 for proprietary models (He, 19 Apr 2026). This suggests that more visual detail does not uniformly help semantic interpretation; under some conditions it amplifies literal bias and weakens symbolic alignment.

A third issue is computational and pipeline complexity. Multi-agent extraction, SVG conversion, intermediate schema construction, contrastive objectives, and two-stage reasoning pipelines improve auditability, but they also introduce orchestration overhead. The DiVE-k framework for fine-grained recognition makes this explicit: it uses the model’s own top-$$b(t) = \mathcal{S}(v_{lit}, t) - \mathcal{S}(v_{id}, t),$$1 outputs to construct multiple-choice questions and trains with GRPO to force differential reasoning among plausible alternatives, outperforming QWEN2.5-VL-7B and ViRFT on the Harmonic Mean metric in base-to-novel, mixed-domain, and few-shot settings (Kumar et al., 23 Nov 2025). The gain comes from structured reasoning over alternatives, but the procedure is more elaborate than a single forward pass.

Across the surveyed work, a plausible synthesis is that DIVE succeeds when three conditions are jointly met: the descriptive intermediate preserves the semantically relevant content of the visual input, the downstream module is optimized to use that intermediate rather than bypass it, and evaluation penalizes plausible but unfaithful outputs. The current literature supports several concrete directions: iconographic abstraction and sense anchoring for compositional understanding (He, 19 Apr 2026), modular multi-agent decomposition for figure and table extraction (Zhang et al., 18 Aug 2025), explicit separation of perception from reasoning through symbolic descriptions (Wang et al., 2024), editable descriptor sets for steerable classification (Menon et al., 2022), and stronger uncertainty communication in human-facing visual interpretation systems (Penuela et al., 7 Mar 2025).