Graph-Fused Vision-Language-Action (GF-VLA)

• The paper introduces GF-VLA, a framework that integrates graph-based representations to mediate between vision, language, and control, leading to improved task robustness.
• It leverages diverse graph substrates—such as 3D entity graphs, semantic scene graphs, and dense geometry tokens—to enhance semantic grounding and guide action selection.
• Empirical evaluations demonstrate that GF-VLA systems boost success rates in complex tasks by improving disambiguation, temporal persistence, and non-Markovian reasoning.

Graph-Fused Vision-Language-Action (GF-VLA) denotes a family of vision-language-action architectures in which a graph-structured representation mediates the coupling between perception, linguistic intent, and control. In concrete formulations, vision is no longer treated only as raw image tokens: it is recast as a 3D entity graph, a semantic scene graph, a continuous scene graph, or a dense geometry token field that functions as an implicit graph. Language is correspondingly elevated from a flat conditioning string to symbolic sub-goals, behavior-tree nodes, executable code plans, or trajectory constraints that query or transform that structure. Actions are then generated from graph-grounded intermediate states rather than from unverified multimodal fusion alone. ProGAL-VLA gives a fully specified graph-fused formulation centered on a 3D entity graph and a verified goal embedding, while related systems extend the pattern to non-Markovian long-horizon manipulation, dual-arm assembly from human demonstrations, continuous scene-graph orchestration of VLA skill banks, and geometry-supervised driving policies (Darabi et al., 10 Apr 2026, Vo et al., 24 Apr 2026, Li et al., 9 Sep 2025, Neau et al., 6 Nov 2025, Yao et al., 10 Jun 2026).

1. Representational scope and graph substrates

GF-VLA is best understood through the kinds of graph-structured state it admits. In ProGAL-VLA, the Grounded State Module builds an entity-centric 3D representation at time $t$,

$$G_t = \{e_1, e_2, \dots, e_n\}, \quad M_t = \text{Update}(M_{t-1}, G_t), \quad E_t = \text{Concat}(G_t, \text{Retrieve}(M_{t-1}, O_t)),$$

where current-frame entities, short-term temporal memory, and retrieved entities together form the grounding substrate used downstream. Nodes encode appearance, 3D position, and semantics; the graph is explicitly described as a “3D object graph,” although its structure is exploited through attention rather than explicit graph convolutions (Darabi et al., 10 Apr 2026).

A second line of work uses persistent semantic or symbolic graphs. CodeGraphVLP maintains a semantic-graph state $\mathcal{G}_t=(\mathcal{V}_t,\mathcal{E}_t)$ over task-relevant entities and relations such as in, on, and near, updated online under partial observability and queried by an executable planner (Vo et al., 24 Apr 2026). GraSP-VLA similarly constructs a Continuous Scene Graph $G_k^+$ with persistent object identities, multi-layer relations, and temporal refinement, then induces symbolic action models and planning domains from graph transitions (Neau et al., 6 Nov 2025). In dual-arm manipulation, GF-VLA constructs temporally ordered scene graphs from hand-object and object-object interactions extracted from RGB-D demonstrations, using nodes for hands and objects and directed edges typed by interaction semantics (Li et al., 9 Sep 2025).

A third family uses geometry as an implicit graph. VLGA introduces geometry as a fourth modality through dense geometry tokens supervised by pointmap reconstruction against LiDAR, with 12,240 geometry tokens per scene derived from six camera views. Although not expressed as an explicit scene graph, the per-patch geometry field is a structured spatial substrate with grid adjacency and ego-frame 3D meaning (Yao et al., 10 Jun 2026). The geometric-foundation-model study around GR00T-N1.5 and VGGT makes the same point from another angle: multi-view geometry tokens, point maps, and depth/normals are graph-like carriers of spatial structure even when implemented with dense attention rather than object nodes (Yang et al., 23 May 2026).

GraphCoT-VLA adds another explicit variant through a “real-time updatable 3D Pose-Object graph,” designed to capture robot-joint and object topology for ambiguous instruction following, together with a structured Chain-of-Thought module (Huang et al., 11 Aug 2025).

System	Graph substrate	Control interface
ProGAL-VLA	3D entity graph with temporal memory	verified goal embedding $g_t$
CodeGraphVLP	persistent semantic-graph state	code planner + VLA executor
GraSP-VLA	Continuous Scene Graph	symbolic domain + VLA skill bank
GF-VLA (multi-arm)	temporally ordered HO/OO scene graphs	behavior trees + Cartesian primitives
VLGA	dense geometry token field / pointmaps	action expert attends to geometry

2. Fusion mechanisms from graph to action

The defining GF-VLA operation is not merely building a graph, but forcing action selection to pass through graph-grounded intermediates. ProGAL-VLA makes this explicit. A slow planner $\pi_{\text{slow}}$ maps instruction, observation, and memory to a symbolic sub-goal template $s_t$, and State Alignment Cross Attention then binds that sub-goal to graph entities: $$Q = \text{Embed}_{\text{sym}}(s_t), \qquad K,V = \text{Embed}_{\text{gnd}}(E_t),$$

$$g_t = \text{Softmax}\!\left(\frac{QK^\top}{\sqrt d}\right)V.$$

The resulting verified goal embedding $g_t$ is the only semantic channel seen by the fast policy,

$$G_t = \{e_1, e_2, \dots, e_n\}, \quad M_t = \text{Update}(M_{t-1}, G_t), \quad E_t = \text{Concat}(G_t, \text{Retrieve}(M_{t-1}, O_t)),$$0

so language affects control only through grounded entities (Darabi et al., 10 Apr 2026).

CodeGraphVLP realizes fusion differently but with the same architectural intent. Its synthesized Python planner executes over the semantic graph and returns both a subtask instruction $$G_t = \{e_1, e_2, \dots, e_n\}, \quad M_t = \text{Update}(M_{t-1}, G_t), \quad E_t = \text{Concat}(G_t, \text{Retrieve}(M_{t-1}, O_t)),$$1 and a subtask-relevant object set $$G_t = \{e_1, e_2, \dots, e_n\}, \quad M_t = \text{Update}(M_{t-1}, G_t), \quad E_t = \text{Concat}(G_t, \text{Retrieve}(M_{t-1}, O_t)),$$2. These outputs are then used to construct clutter-suppressed observations, and the VLA executor acts only on the masked visual evidence and subtask string corresponding to the current graph state (Vo et al., 24 Apr 2026). This creates a graph-guided perceptual bottleneck rather than a graph-derived latent vector.

GraSP-VLA inserts a symbolic layer between graph perception and control. Functional and topological relations in the Continuous Scene Graph are converted into PDDL-style action schemas with preconditions and effects; the resulting symbolic sequence orchestrates a bank of low-level VLA policies rather than a single monolithic policy (Neau et al., 6 Nov 2025). In the dual-arm GF-VLA formulation, temporally ordered scene graphs are fused with a language-conditioned transformer that produces hierarchical behavior trees and interpretable Cartesian motion commands, while a cross-arm allocation strategy selects gripper assignment (Li et al., 9 Sep 2025).

GeneralVLA, while not naming itself GF-VLA, exhibits a closely related hierarchical pattern: affordance points define object-centric interaction anchors, a mid-level 3DAgent generates a discrete 3D path with gripper actions, and a low-level 3D-aware controller realizes the path. A plausible implication is that GF-VLA can be seen as the explicit graph-structuring of such hierarchical interfaces, with objects, affordances, skills, and waypoints as node types (Ma et al., 4 Feb 2026).

3. Information-theoretic and supervision principles

A central theoretical theme in GF-VLA is that graph mediation should increase the information actually used for control. ProGAL-VLA makes this claim formally in two ways. First, it uses a Grounding Alignment Contrastive loss

$$G_t = \{e_1, e_2, \dots, e_n\}, \quad M_t = \text{Update}(M_{t-1}, G_t), \quad E_t = \text{Concat}(G_t, \text{Retrieve}(M_{t-1}, O_t)),$$3

an InfoNCE objective aligning symbolic sub-goals with the correct grounded entity embeddings. Under the usual contrastive assumption, the paper states

$$G_t = \{e_1, e_2, \dots, e_n\}, \quad M_t = \text{Update}(M_{t-1}, G_t), \quad E_t = \text{Concat}(G_t, \text{Retrieve}(M_{t-1}, O_t)),$$4

so minimizing $$G_t = \{e_1, e_2, \dots, e_n\}, \quad M_t = \text{Update}(M_{t-1}, G_t), \quad E_t = \text{Concat}(G_t, \text{Retrieve}(M_{t-1}, O_t)),$$5 increases a lower bound on mutual information between symbolic goals and entity representations. Second, the verification bottleneck imposes

$$G_t = \{e_1, e_2, \dots, e_n\}, \quad M_t = \text{Update}(M_{t-1}, G_t), \quad E_t = \text{Concat}(G_t, \text{Retrieve}(M_{t-1}, O_t)),$$6

making the verified goal embedding the sole mediator of language influence on action (Darabi et al., 10 Apr 2026).

The same paper also turns graph attention into an intrinsic uncertainty estimator. With attention weights $$G_t = \{e_1, e_2, \dots, e_n\}, \quad M_t = \text{Update}(M_{t-1}, G_t), \quad E_t = \text{Concat}(G_t, \text{Retrieve}(M_{t-1}, O_t)),$$7 over entity nodes, ambiguity is quantified by

$$G_t = \{e_1, e_2, \dots, e_n\}, \quad M_t = \text{Update}(M_{t-1}, G_t), \quad E_t = \text{Concat}(G_t, \text{Retrieve}(M_{t-1}, O_t)),$$8

Low entropy corresponds to concentrated grounding, high entropy to ambiguous grounding, and an entropy-gated clarify policy triggers a [CLARIFY] action when $$G_t = \{e_1, e_2, \dots, e_n\}, \quad M_t = \text{Update}(M_{t-1}, G_t), \quad E_t = \text{Concat}(G_t, \text{Retrieve}(M_{t-1}, O_t)),$$9 exceeds a threshold (Darabi et al., 10 Apr 2026).

In dual-arm GF-VLA for manipulation, information theory is used earlier in the pipeline. Shannon entropy over sliding windows identifies active entities, while mutual information between hand and object trajectories detects true coupling: $\mathcal{G}_t=(\mathcal{V}_t,\mathcal{E}_t)$0 This supports scene-graph edge typing into Coupled-Motion, Docked, Efficient OO, and Transitory OO interactions, and provides the basis for subtask segmentation (Li et al., 9 Sep 2025).

In geometry-centric systems, dense supervision plays the role that InfoNCE plays in entity-centric systems. VLGA supervises a dedicated geometry stream with a confidence-weighted per-pixel pointmap regression loss against LiDAR,

$\mathcal{G}_t=(\mathcal{V}_t,\mathcal{E}_t)$1

forcing the policy’s geometry modality to reconstruct dense 3D structure rather than merely receive frozen geometric features (Yao et al., 10 Jun 2026). The study of geometric foundation models and VLA fusion reinforces why such direct supervision matters: linear probing reveals a “geometric gap” between GR00T-N1.5 and VGGT, and early or late geometry fusion nearly closes that gap for depth and normal reconstruction (Yang et al., 23 May 2026).

4. Empirical regimes and representative results

Across manipulation, long-horizon planning, and driving, GF-VLA systems are associated with gains in robustness, instruction sensitivity, and non-Markovian reasoning.

On LIBERO-Plus, ProGAL-VLA improves robustness under robot perturbations from 30.3 to 71.5 percent, reduces language ignorance by 3x-4x, and improves entity retrieval from 0.41 to 0.71 Recall@1. On the Custom Ambiguity Benchmark, it reaches AUROC 0.81 versus 0.52, AUPR 0.79, and raises clarification on ambiguous inputs from 0.09 to 0.81 without harming unambiguous success (Darabi et al., 10 Apr 2026).

On real-world non-Markovian tabletop tasks, CodeGraphVLP reports 81.7 percent average success, compared with 56.7 percent for a history-enabled Gr00T N1.5 variant and 31.7 percent for Gr00T N1.5 without multi-frame history. In Swap Cups, a VLM-in-the-loop planner with RGB only achieves 25 percent success and 2.967 s/step latency, RGB plus semantic graph reaches 55 percent and 3.142 s/step, while code-as-planner plus graph reaches 85 percent at 0.328 s/step (Vo et al., 24 Apr 2026).

In long-horizon real-world table setting, GraSP-VLA shows that symbolic orchestration over a Continuous Scene Graph extends the effective horizon of low-level VLA skills. Reported overall task success is 0.6 for 2 skills, 0.4 for 4 skills, and 0.4 for 6 skills, whereas full SmolVLA fine-tuning on whole tasks yields 0.2, 0.1, and 0.0 respectively (Neau et al., 6 Nov 2025).

In dual-arm robotic manipulation, GF-VLA reports over 95 percent graph accuracy and 93 percent subtask segmentation, enabling a language-action planner that achieves 94 percent grasp reliability, 89 percent placement accuracy, and 90 percent overall task success across stacking, letter-formation, and geometric reconfiguration tasks (Li et al., 9 Sep 2025).

In autonomous driving, VLGA sets a new state of the art among VLA methods without ego status on nuScenes, with 0.50 m average L2 and 0.18 percent 3-second collision rate, and attains a Bench2Drive closed-loop driving score of 79.08, +0.71 over the strongest prior VLA, at comparable efficiency and comfort (Yao et al., 10 Jun 2026).

These results point to a recurring empirical profile: graph mediation improves precisely those behaviors that flat token fusion tends to under-serve—robust grounding under perturbation, disambiguation, temporal persistence under occlusion, and safety-sensitive spatial control.

5. Relation to adjacent structured VLA designs

A recurrent misconception is that GF-VLA requires explicit graph convolutions or discrete scene-graph APIs. The recent literature is broader. ProGAL-VLA is explicitly described as a “3D object graph,” yet its graph structure is exploited through attention over nodes rather than explicit graph convolutions (Darabi et al., 10 Apr 2026). DeepVision-VLA goes further in a token-centric direction: it studies the decay of visual grounding in deeper VLA layers and introduces a Vision-Language Mixture-of-Transformers framework with shared attention between a vision expert and the VLA backbone, plus Action-Guided Visual Pruning. Its own synthesis explicitly interprets tokens as nodes, attention as edges, and shared QKV fusion as joint message passing over a combined graph, even though the implementation remains transformer-based rather than graph-neural (Luo et al., 16 Mar 2026).

Efficiency work supplies another graph-compatible ingredient. VLA-Pruner proposes temporal-aware dual-level visual token pruning using semantic prefill attention and temporally smoothed action-decode attention; its detailed interpretation treats pruning as node selection over an implicit patch graph and diversity filtering as graph coarsening (Liu et al., 20 Nov 2025). A plausible implication is that graph-fused systems can use such selection mechanisms to construct compact, task-relevant subgraphs before reasoning.

GeneralVLA contributes a related hierarchy: affordance segmentation produces object-centric 2D and 3D points, a 3DAgent plans a waypoint sequence with gripper actions, and a low-level controller executes the plan. Although described without explicit graph formalism, objects, affordance points, and waypoints already form a graph-like decomposition of the control problem (Ma et al., 4 Feb 2026). Likewise, LinkVLA and Unified Diffusion VLA show that structured multimodal decoding can be organized through shared codebooks, hybrid attention, or synchronous denoising trajectories; these are not GF-VLA systems in the narrow sense, but they clarify how typed interactions among language, scene structure, and action can be enforced architecturally (Wang et al., 2 Mar 2026, Chen et al., 3 Nov 2025).

The broader lesson is that GF-VLA is not a single implementation template. It spans explicit entity graphs, semantic relation graphs, continuous scene graphs, geometry-token lattices, and token-graph approximations, provided that the structured substrate is not merely auxiliary but actually mediates action generation.

6. Limitations, misconceptions, and research directions

Several limitations recur across GF-VLA systems. First, graph quality remains a bottleneck. GraSP-VLA identifies low recall for functional relations such as holding as a major failure source, since downstream symbolic action induction depends on those relations (Neau et al., 6 Nov 2025). CodeGraphVLP similarly relies on a manually engineered semantic-graph construction pipeline, and explicitly points to open-world semantic graph generation and planner verification as future work (Vo et al., 24 Apr 2026). In the dual-arm GF-VLA system, graph extraction assumes reliable RGB-D sensing, segmentation, and 6D pose estimation; temporal segmentation degrades under occlusion and rapid interaction overlap (Li et al., 9 Sep 2025).

Second, graph fusion should not be confused with universal geometric competence. The geometric-foundation-model study shows that current VLAs can have a substantial geometric gap relative to VGGT, and that architecture alone is insufficient without careful fusion design and sufficient data (Yang et al., 23 May 2026). VLGA reaches strong driving results precisely by introducing a dedicated geometry expert with its own supervision, rather than by assuming that frozen 3D features will automatically be used (Yao et al., 10 Jun 2026).

Third, graph reasoning and real-time control remain in tension. CodeGraphVLP demonstrates that graph-based code planning is dramatically faster than repeated VLM-in-the-loop planning, but its solution depends on synthesized task-specific code rather than a universally adaptive planner (Vo et al., 24 Apr 2026). VLA-Pruner suggests one route to scalability: prune or coarsen the visual node set using both semantic and action relevance so that structured reasoning can remain within real-time budgets (Liu et al., 20 Nov 2025).

A final misconception is that GF-VLA is synonymous with one particular application domain. Current instances range from ambiguity-aware single-arm manipulation (Darabi et al., 10 Apr 2026), to long-horizon non-Markovian tasks (Vo et al., 24 Apr 2026), to automatic symbolic-domain induction for skill orchestration (Neau et al., 6 Nov 2025), to dual-arm assembly from human videos (Li et al., 9 Sep 2025), to dense-geometry driving policies (Yao et al., 10 Jun 2026). This suggests that GF-VLA is better understood as an architectural principle: make the representation that links vision, language, and action explicitly structured, and make action depend on that structure in a verifiable way.

A plausible implication is that the next phase of GF-VLA research will converge on three themes already visible in the literature: richer open-world graph construction, tighter joint training between structured world models and action heads, and selective graph sparsification that preserves the entities and relations most critical for both semantic understanding and low-level execution.