PGVL: Graph-based Visual-Language Interaction

Updated 10 July 2026

PGVL is a multimodal paradigm that employs graph structures to mediate language and vision interactions, enabling explicit cross-modal reasoning.
It encompasses diverse formulations from parse-guided routing in VQA to scene graph editing and neuro-symbolic graph reasoning.
This approach improves task performance by leveraging structured message passing, graph-conditioned fusion, and hierarchical token processing.

Parse Graph-based Visual-Language Interaction (PGVL) denotes a family of multimodal formulations in which graph-structured representations mediate the coupling between language and vision. In the cited literature, the mediating structure may be a dependency or syntax tree extracted from a question, a scene graph, a visual graph rendered as an image, a visual knowledge graph under edit, or modality-specific hierarchical parse graphs; the common objective is to make cross-modal reasoning explicit through routing, message passing, symbolic execution, graph-conditioned reconstruction, or graph-conditioned editing rather than relying only on undifferentiated feature fusion (Cao et al., 2020, Su et al., 2023, Liang et al., 2021, Wei et al., 2024, Bauer et al., 13 Feb 2025, Wang et al., 3 Aug 2025, Liu et al., 9 Sep 2025, Han et al., 19 Oct 2025, Shahriari et al., 12 Dec 2025).

1. Scope and conceptual variants

Taken together, the cited works suggest that PGVL is better understood as a design pattern than as a single model class. In 2020, one line used a linguistic parse tree to carve a compositional reasoning process inside a CNN for visual question answering, while another formalized scene-graph modification from natural-language commands. Subsequent work extended the pattern to syntax-constrained graph networks, scene-graph question answering, multimodal reasoning over rendered graph images, neuro-symbolic parsing of graph images, language-driven editing of visual knowledge graphs, graph-conditioned remote sensing change detection, human pose estimation, and agent-driven large-graph understanding (Cao et al., 2020, He et al., 2020, Su et al., 2023, Liang et al., 2021, Wei et al., 2024, Bauer et al., 13 Feb 2025, Wang et al., 3 Aug 2025, Liu et al., 9 Sep 2025, Han et al., 19 Oct 2025, Shahriari et al., 12 Dec 2025).

PGVL regime	Mediating graph structure	Representative papers
Parse-guided VQA	Dependency tree or syntax tree constraining routing or message passing	(Cao et al., 2020, Su et al., 2023)
Scene-graph or graph-image QA	Scene graph, rendered visual graph, or parsed graph image	(Liang et al., 2021, Wei et al., 2024, Bauer et al., 13 Feb 2025)
Language-driven graph transformation	Scene graph or visual knowledge graph updated by commands	(He et al., 2020, Shahriari et al., 12 Dec 2025)
Domain-specific graph-conditioned fusion	Semantic token graph, modality-specific parse graphs, or hierarchical GraphRAG base	(Wang et al., 3 Aug 2025, Liu et al., 9 Sep 2025, Han et al., 19 Oct 2025)

A recurring misconception is that PGVL necessarily requires either hand-authored program layouts or explicit scene graphs. The literature is more heterogeneous. Some systems deliberately avoid layout annotations and learn with answer supervision alone, while others explicitly parse graph images into symbolic facts or operate over pre-existing graph structures. This suggests that the defining feature of PGVL is not a particular supervision protocol, but the use of graph structure to regulate visual-language interaction.

2. Formal abstractions and shared computational primitives

Despite their diversity, PGVL systems repeatedly adopt explicit graph formalisms. Visual graph question answering over graph images writes the visual parser, language parser, and reasoner as

$\widehat{G} = \phi_v(I), \qquad \Pi = \phi_l(q), \qquad a = R(\widehat{G}, \Pi),$

where $\widehat{G}$ is a symbolic graph extracted from an image, $\Pi$ is a set of ASP facts encoding the question semantics, and $a$ is the computed answer (Bauer et al., 13 Feb 2025). Interactive graph editing uses a richer graph model,

$G = (V, E, A_V, A_E),$

with node and edge attributes, and formalizes utterance grounding and execution through $(I, A) := \phi(u, S)$ followed by $G' = f(G, I, A)$ , where the current visualization state $S$ constrains reference resolution (Shahriari et al., 12 Dec 2025).

Parse-guided routing formulations make the graph prior operational inside differentiable modules. In the linguistically driven graph capsule model, parent-capsule assignments $Y$ at a layer are inferred by a CRF with energy

$E(Y \mid X) = \sum_i \phi_i(Y_i; X) + \sum_{(i,j)\in \mathcal{E}} \phi_{ij}(Y_i, Y_j; X),$

where unary potentials come from multimodal capsule features and pairwise potentials are induced from parse-derived compatibility; mean-field marginals $\widehat{G}$ 0 then become routing coefficients $\widehat{G}$ 1 (Cao et al., 2020). Scalable graph reasoning introduces a different control layer: a planner chooses $\widehat{G}$ 2 by

$\widehat{G}$ 3

while retrieval from a hierarchical GraphRAG base solves

$\widehat{G}$ 4

thereby binding modality selection to task type and context budget (Han et al., 19 Oct 2025).

Graph-conditioned fusion also appears in dense token spaces. In remote sensing change detection, visual and textual tokens are concatenated, projected, and converted into a learned dense adjacency by

$\widehat{G}$ 5

after which graph-conditioned vision-language tokens reconstruct both streams through attention before transformer fusion (Wang et al., 3 Aug 2025). This suggests that PGVL spans both explicitly discrete graphs and learned dense token graphs; what persists is the use of graph structure as an intermediate control object for multimodal computation.

3. Parse-tree-constrained visual question answering

A canonical PGVL instance appears in the “Linguistically Driven Graph Capsule Network” (Cao et al., 2020). Questions are tokenized, embedded, encoded by a GRU or BiGRU, and parsed into a dependency tree with the Stanford Parser. The dependency tree is converted into a leveled linguistic layout $\widehat{G}$ 6, after which each lowest-layer capsule binds the embedding of a single word to visual evidence via low-rank bilinear pooling. Routing is not performed by dynamic-by-agreement; it is performed by a CRF whose pairwise terms are derived from sibling and parent-child relations in the parse tree. Higher-layer capsules progressively represent phrases and clauses, and sentence perception fields are propagated by $\widehat{G}$ 7. Training uses only answer cross-entropy, with no specialized capsule margin loss and no layout annotations. Empirically, the model reaches 97.9% overall accuracy on CLEVR, with per-type results Count 95.6, Exist 98.7, Compare Integer 97.2, Compare Attribute 98.8, and Query 98.8; on CLEVR-CoGenT it attains A 98.34% and B 84.06%; on FigureQA it attains 90.58% on validation and 90.70% on test, and ablations show that adding unary and binary parser-guided CRF terms is critical for compositional generalization (Cao et al., 2020).

The “Syntax Tree Constrained Graph Network” uses a different but closely related strategy (Su et al., 2023). It parses each question with the Stanford syntax parsing tool, builds a syntax tree with typed dependencies and direction labels, extracts word-level and phrase-level representations through a hierarchical tree convolutional network, and then uses phrase-conditioned instruction vectors to drive multi-step message passing over visual entities. Phrase-aware visual states are updated for $\widehat{G}$ 8 steps, and top-down attention combines refined entity features with the global question representation for answer prediction. On VQA2.0, the model reports Overall = 70.99 on Test-std and 70.21 on Test-dev, surpassing LCGN, MuRel, ReGAT, and BAN as reported, while ablations show that removing tree convolution or message passing reduces performance and that accuracy peaks at $\widehat{G}$ 9 before declining due to redundant messages and noise (Su et al., 2023).

These models share a characteristic PGVL thesis: syntax is not merely an auxiliary text encoder but a structural prior that constrains where multimodal composition should occur. In both cases, the parse structure determines which word or phrase representations should be merged, propagated, or used as instructions for visual refinement.

4. Scene graphs, visual graphs, and neuro-symbolic graph reasoning

Graph-based VQA over scene graphs provides a second major lineage. “GraphVQA” represents an image as a scene graph $\Pi$ 0 whose nodes are object names and attributes and whose edges are typed relations. A sequence-to-sequence transformer maps a question into $\Pi$ 1 instruction vectors, and each reasoning step concatenates the current instruction to node and edge features before applying a GCN, GINE, or GAT layer. Reverse edges are added by negating relation embeddings for non-paired directed edges. The best variant, GraphVQA-GAT, achieves 94.78% accuracy on GQA, compared with 88.43% for LCGN and 85.70% for an unconditioned GCN baseline; on relationship questions it reaches 95.53%, and the reported analyses indicate that language conditioning and attention are especially beneficial for long questions (Liang et al., 2021).

Rendered visual graphs constitute a different PGVL substrate. GITA renders graphs with Graphviz, encodes the resulting image with the visual encoder from LLaVA, feeds textual graph descriptions and questions to Vicuna, and concatenates projected visual embeddings with text embeddings before autoregressive answer generation. The benchmark described in the detailed exposition contains 423K instances across 8 tasks, including Connectivity, Cycle, Topological Sort, Shortest Path, Maximum Flow, Bipartite Graph Matching, Hamilton Path, and a one-step GNN update task. GITA-13B with joint visual and textual input reports 55.48% average accuracy, compared with 41.68% for GPT-4V (V+T), 52.48% for Vicuna-13B (T-only), and 38.54% for LLaVA-13B (V-only). Layout augmentation is especially consequential: for LLaVA-7B, average accuracy rises from 38.13% to 49.19% when using 6 layouts, with Shortest Path increasing from 11.75% to 76.55% and Hamilton Path from 1.11% to 70.74% (Wei et al., 2024).

A neuro-symbolic alternative parses graph images back into symbolic facts. “Visual Graph Question Answering with ASP and LLMs for Language Parsing” uses optical graph recognition for nodes and adjacency, EasyOCR for labels, an LLM or rule-based parser to emit ASP facts, and clingo for reasoning. The resulting baseline on metro-like graph images attains 73.0% overall accuracy on a 3000-pair dataset, with 80.9% on small graphs, 71.0% on medium graphs, and 67.2% on large graphs; with full ground-truth symbolic inputs, the ASP component solves all queries at 100%, while OGR errors are identified as the main bottleneck. For language parsing, GPT-4 reaches 85% full-match accuracy on an extended question set and 94% on a human question set (Bauer et al., 13 Feb 2025).

These variants suggest that PGVL does not commit to a single visual substrate. The “graph” may be an annotated scene graph, a visual graph image given directly to a multimodal model, or a graph image parsed into a symbolic representation before logical execution.

5. Language-driven graph transformation and editing

PGVL also encompasses stateful graph transformation. “Scene Graph Modification Based on Natural Language Commands” defines the task as predicting a target graph $\Pi$ 2 from a source graph $\Pi$ 3 and a command $\Pi$ 4 through

$\Pi$ 5

Its encoder builds edge-aware node embeddings, applies a graph-based sparse transformer with topology-respecting masks, and fuses graph and language through cross-attention. Node and edge decoders then generate the modified graph. On the MSCOCO synthetic test set, the best model, Sparse Trans + Flat-Edge + Cross-Attn, reports Edge F1 86.52, Node F1 95.40, and Graph Accuracy 82.97, compared with 75.68, 91.21, and 71.38 for the best graph-transformer baseline. On the user-generated test set it reaches Edge F1 62.10, Node F1 83.69, and Graph Accuracy 60.90, and on multi-operation datasets it remains superior as edit length increases, although all models degrade sharply for 5+ operations (He et al., 2020).

Natural-language editing of visual knowledge graphs makes the interaction explicitly user-facing. The graph is formalized as $\Pi$ 6, the implemented atomic operations include add, delete, rename, and attribute updates for nodes and edges, and the interface compares GUI interaction, structured textual commands, and free-form natural language. The parser is implemented with OpenAI’s “gpt-3.5-turbo” via the ChatGPT API using few-shot prompting. In a $\Pi$ 7 mixed design with 76 analyzed participants, Natural Language outperforms GUI Interaction on Graph Changes Per Time with $\Pi$ 8 and outperforms Textual Command with $\Pi$ 9; for Graph Changes Per Action, Natural Language exceeds GUI Interaction with $a$ 0 and Textual Command with $a$ 1. Average backend processing time for Natural Language and Textual Command is 2.43 seconds, 51% of Natural Language inputs are command-like, and undo is used in only 2.8% of Natural Language trials that used it (Shahriari et al., 12 Dec 2025).

This editing-oriented branch broadens PGVL beyond question answering. Here the graph is not simply inferred or queried; it is a persistent state that must be grounded to the current visualization, updated transactionally, validated, and, when necessary, reversed.

6. Domain extensions, scalability, and recurring limitations

Remote sensing change detection adopts PGVL as graph-conditioned multimodal reconstruction. MGCR-Net first uses a LLaVA-based optimization strategy to generate change-relevant text, encodes bi-temporal images with a Siamese PVT and text with a CLIP text encoder, constructs a dense semantic graph over concatenated visual and textual tokens inside the Semantic Graph-Conditioned Module, reconstructs both streams via cross-attention keyed by graph-conditioned vision-language tokens, and then fuses them with the Language Vision Transformer. Training uses

$a$ 2

with $a$ 3 reported as more stable and superior than $a$ 4. The model reports LEVIR-CD F1 92.07 and IoU 85.30, WHU-CD F1 94.91 and IoU 90.32, GZ-CD F1 89.55 and IoU 81.07, and SYSU-CD F1 82.91 and IoU 70.80, while the paper notes the cost of dense adjacency and the sensitivity of the graph to text quality (Wang et al., 3 Aug 2025).

Human pose estimation uses PGVL as a hierarchy-to-hierarchy fusion mechanism. The visual and language modalities are each decomposed into parse graphs, low-level nodes preserve local responses in occluded areas, high-level nodes encode more global semantics, recursive bidirectional cross-attention exchanges information between aligned nodes, and a Guided Module lets high-semantic nodes guide the update of low-semantic nodes. The full objective combines pose heatmap loss and a Visual-Language Matching Loss, with ungated GM used in the main experiments. Reported results include Mean PCKh 91.7 on MPII val, mAP 68.2 on CrowdPose test, mAP 62.8 on OCHuman val, mAP 82.1 on AP-10K val, and mAP 79.3 on Animal-Pose val; ablations show that removing GM causes the largest drop among the tested module removals (Liu et al., 9 Sep 2025).

Scalable graph understanding introduces a planner-centered PGVL formulation. GraphVista organizes graph information into a Hierarchical GraphRAG Base $a$ 5, retrieves only task-relevant text or high-resolution visual subgraphs under a token budget, and uses a planning agent to route tasks to “Say” for simple property reasoning, “See” for local structural reasoning, or fusion when both are helpful. On the Grena benchmark with 13,860 tasks and graphs up to 2050 nodes, simple-task accuracy exceeds 0.93 and remains stable with scale; on complex local tasks, Visual Graph Thoughts with DPO reaches about 0.391 accuracy, compared with 0.143 for the best fusion baseline and 0.078 for graph-token encoders, and the reported overall gain reaches up to 4.4× over state-of-the-art baselines. The paper also identifies misrouting, retrieval errors, and clutter in rendered subgraphs as persistent failure modes (Han et al., 19 Oct 2025).

Across the literature, recurring limitations include parser mismatch between syntax and task semantics, errors in graph extraction from images, quadratic or dense interaction costs, rendering clutter, and disambiguation failures in larger or noisier graphs. This suggests that the central open problem in PGVL is not merely how to add a graph, but how to choose, construct, and maintain the right graph at the right granularity for the multimodal task at hand.