---
title: 'VL-KnG: Vision-Language Knowledge Graph'
url: https://www.emergentmind.com/topics/vl-kng
type: topic
---

# VL-KnG: Vision-Language Knowledge Graph

VL-KnG, short for **Vision-Language Knowledge Graph**, is a visual scene understanding system for **navigation goal identification** that converts a long demonstration tour video into a **persistent, queryable spatiotemporal knowledge graph**. It was introduced to address four limitations attributed to standard vision-language models in long-horizon robotic navigation: lack of **persistent scene memory**, limited **explicit spatial reasoning**, poor scaling with video duration, and high query-time cost for **real-time deployment** when the full visual history must be reprocessed for every query. In VL-KnG, modern VLMs are used for chunk-level perception and semantic extraction, but the primary representation is the graph itself rather than the VLM’s transient context [2510.01483].

## 1. Problem formulation and motivation

VL-KnG studies **visual scene understanding for robot navigation goal identification**. The input is a demonstration tour video represented as a sequence of frames,
$$
\mathcal{I} = \{I_t\}_{t=1}^T,\quad I_t \in \mathbb{R}^{H \times W \times 3},
$$
together with a set of natural-language navigation questions,
$$
\mathcal{Q} = \{q_n\}_{n=1}^N.
$$
Given a query $q_n$ and the video $\mathcal{I}$, the system must identify the most relevant frame indices
$$
\mathcal{F} \subseteq \{1,\ldots,T\}
$$
that contain the goal object or goal location described in the query [2510.01483].

The motivating claim is that direct VLM inference over long videos is not an adequate memory and reasoning substrate for navigation. The paper identifies three operational issues beyond raw answer quality: temporal coherence degrades when observations are processed sequentially, structured relational reasoning is weak when the representation remains implicit, and repeated querying becomes expensive if each query requires re-encoding the entire trajectory. VL-KnG addresses these issues by extracting object- and relation-level knowledge incrementally from video and storing it as a persistent graph.

A central conceptual point is that the VLM is **not the memory itself**. Instead, VLMs act as perceptual and semantic extraction modules inside a larger architecture whose persistent state is a spatiotemporal graph. This distinguishes VL-KnG from a generic long-video VLM pipeline and explains why the method is framed simultaneously as a memory system, a retrieval system, and a navigation-oriented reasoning system.

## 2. Graph construction and representation

VL-KnG processes the video in chunks rather than frame-by-frame isolation or full-video prompting. For chunk size $b$, the video is partitioned as
$$
\mathcal{C}_k = \{i_{kb+1}, \ldots, i_{(k+1)b}\}, \quad k=0,\ldots,B,
$$
where
$$
B = \lfloor T/b \rfloor - 1.
$$
The paper reports that **$b=8$** gave the best tradeoff. For each chunk, a modern multi-image VLM jointly inspects the frames and outputs object descriptors and relationships, producing a local chunk graph $\mathcal{G}_k^{chunk}$; the paper mentions **Gemini 2.5** and **Qwen2.5-VL** as examples of the VLMs used in the system [2510.01483].

The global graph is built incrementally. After the first chunk,
$$
\mathcal{G}^{(0)} \leftarrow \mathcal{G}_0^{chunk},
$$
and subsequent chunks are merged by **spatiotemporal object association**:
$$
\mathcal{G}^{(k)} \leftarrow \text{STOA}(\mathcal{G}^{(k-1)}, \mathcal{G}_k^{chunk}).
$$
The final graph $\mathcal{G}^{(B)}$ becomes the environment knowledge graph $\mathcal{G}$ and is stored in a graph database.

The graph is formalized as
$$
\mathcal{G} = (V,E),
$$
with nodes representing **unique objects with rich descriptors** and edges representing **spatial relationships**. The explicit object descriptor given in the paper is
$$
\text{o}_i = \{bbox_i, id_i, color_i, material_i, size_i, t_i, affordances_i, relationships_i\}.
$$
Accordingly, the core stored fields are bounding box, persistent identity, color, material, size, timestamp, affordances, and relationships. Temporal information is not presented as a separate dynamic-graph formalism; rather, temporality is operationalized through chunk-wise updates, timestamps, and persistent identity across chunk merges.

The graph schema is intentionally lightweight. Object-level attributes and inter-object spatial relations are the formal primitives. The paper does **not explicitly define** a room-containment schema, a formal edge taxonomy for semantic relations such as `isA`, or a logical query language such as Cypher or Datalog. This suggests that VL-KnG prioritizes a practical object-centric scene memory over a more heavily axiomatized knowledge representation.

## 3. Spatiotemporal object association and graph-based reasoning

A key technical claim of VL-KnG is that identity persistence should be based on **semantic** comparison rather than only low-level visual similarity. For objects $o_i^k$ and $o_j^{k+1}$ from adjacent chunks, semantic similarity is defined as
$$
\text{Sim}(o_i^k, o_j^{k+1}) = \text{LLM}\big(\text{desc}(o_i^k), \text{desc}(o_j^{k+1})\big) \in [0, 1],
$$
and association is thresholded as
$$
\text{Assoc}(o_i^k, o_j^{k+1}) =
\begin{cases}
1 & \text{if } \text{Sim}(o_i^k, o_j^{k+1}) > \tau \\
0 & \text{otherwise.}
\end{cases}
$$
If two chunk-level instances are associated, they are treated as the same persistent graph entity. The paper presents this as more robust than purely visual matching under lighting changes, occlusion, and viewpoint shifts [2510.01483].

At query time, VL-KnG uses a **GraphRAG-style** pipeline. A natural-language query is decomposed into target entities, attributes, spatial relations, and temporal constraints using LLM reasoning. The system then retrieves a relevant subgraph
$$
\mathcal{G}_{sub} \subseteq \mathcal{G}
$$
through graph traversal and semantic filtering, and reasons over that subgraph to infer the answer and the relevant frame or frames. If the demonstration tour frames were paired with poses, the pose associated with the selected goal frame is sent to the downstream navigation system.

The paper emphasizes that VL-KnG is **not** a graph neural network and **not** a trained end-to-end neural architecture with a learned graph encoder or loss. It is a modular system built around prompting VLMs and LLMs, storing the extracted structure in a graph database, and querying that structure later. Its explicit query-time complexity claim is
$$
O(|V_{sub}| + |E_{sub}| + |Q|),
$$
where $|V_{sub}|$ and $|E_{sub}|$ are the size of the retrieved subgraph and $|Q|$ denotes query complexity. The reported average query latency for the retrieval-based graph method is **$\sim 1$ s**, compared with **$\sim 120$ s** for **Gemini 2.5 Pro**.

This reasoning design supports questions such as object search, scene description, spatial relation queries, and action-place association. The paper illustrates these with examples such as identifying “the chair next to the desk,” answering “Where can I sit?” through stored affordances, and traversing left-of relations for “What is left of the kiosk?” The graph therefore acts not only as a semantic memory but as a localization substrate.

## 4. WalkieKnowledge benchmark and empirical evaluation

To evaluate the system, the authors introduced **WalkieKnowledge**, built on top of **EgoWalk** and designed for long-horizon navigation-oriented visual reasoning. The benchmark contains **8 recorded trajectories** spanning **indoor and outdoor environments**, totaling approximately **100 minutes of video**, with **193 natural-language questions**; the abstract rounds this to “about 200 manually annotated questions” [2510.01483]. The environments include shopping malls, supermarkets, exhibitions, bazaars, and streets.

Questions are grouped into four categories: **object search**, **scene description**, **spatial relation**, and **action-place association**. Each question is linked to ground-truth frame intervals, and scene description and spatial relation questions also provide multiple-choice options. Evaluation uses **Retrieval Accuracy@k**, **Recall@k**, **Precision@k**, **MRR@k**, and **Answer Accuracy**.

On the benchmark, three VL-KnG variants are compared: **Full KG**, **Retrieval-based**, and **Chunk-Wise Retrieval (CWR)**. The main overall results reported for **VL-KnG Full KG** are: Retrieval Acc.@1 **57.51**, Retrieval Acc.@3 **69.43**, Retrieval Acc.@5 **69.95**, Recall@1 **27.9**, Recall@3 **53.7**, Recall@5 **57.3**, Precision@1 **57.51**, Precision@3 **39.55**, Precision@5 **25.80**, MRR@1 **0.58**, MRR@3 **0.63**, MRR@5 **0.63**, and Answer Acc. **58.14**. For **VL-KnG Retrieval-based**, the corresponding Answer Acc. is **50.00**. For **VL-KnG Chunk-Wise Retrieval**, Answer Acc. drops to **37.21** [2510.01483].

These comparisons are significant because they isolate the effect of persistent global association. The jump from **37.21%** answer accuracy in chunk-wise retrieval to **50.00%** in retrieval-based global reasoning and **58.14%** in full-graph reasoning is the strongest direct evidence that cross-chunk object identity improves coherence. A plausible implication is that the principal benefit of VL-KnG is not only retrieval efficiency, but the formation of a global object-centric memory unavailable to chunk-wise prompting.

Against general-purpose VLM baselines, **Gemini 2.5 Pro** achieves the strongest overall benchmark retrieval, with Retrieval Acc.@1 **68.91** and Answer Acc. **61.63**. However, VL-KnG remains competitive in category-specific settings. In **scene description**, VL-KnG Full KG achieves **62%** answer accuracy, matching **Gemini 2.5 Flash’s 62%** and approaching **Gemini 2.5 Pro’s 68%**. In **spatial relations**, VL-KnG Full KG attains **59% Recall@3**, compared with **52% for Gemini 2.5 Pro**. The paper uses this result to argue that the graph representation improves relational reasoning even when raw benchmark dominance remains with Gemini in many global metrics.

## 5. Real-world deployment and practical role in robotics

VL-KnG was also evaluated on a real robot platform, reflecting the paper’s emphasis on deployment rather than benchmark-only performance. The hardware stack is a **differential drive robot** equipped with an **Intel NUC11PHKI7C000 PC** and an **NVIDIA RTX 2060 GPU**. The software stack uses **SLAM Toolbox** for localization and map handling and the **ROS Navigation Stack** for navigation [2510.01483].

In this setting, the source tour video is paired with poses. Once VL-KnG identifies the best goal frame for a language query, the associated pose is passed to the navigation system. The reported real-world results are: **VL-KnG** Success Rate **77.27%**, Answer Accuracy **76.92%**; **Gemini 2.5 Pro** Success Rate **77.27%**, Answer Accuracy **76.92%**; **RoboHop** Success Rate **27.27%**, Answer Accuracy **23.08%**.

These deployment numbers define the practical identity of VL-KnG more clearly than the offline benchmark alone. The system is intended to support **localization**, **navigation**, and **planning** from a persistent visual memory. Its practical value, as stated in the paper, lies in shifting computation offline or amortizing it through graph construction, answering queries from retrieved subgraphs rather than raw video, maintaining an explainable representation through explicit nodes and edges, and interfacing with standard robotics infrastructure rather than requiring a bespoke end-to-end policy.

A common misconception is that VL-KnG is simply a QA wrapper around a large VLM. The real-world setup suggests a narrower but more operational interpretation: it is a structured memory-and-retrieval layer that turns prior visual experience into actionable goal proposals for downstream navigation.

## 6. Subsequent extensions, critiques, and limitations

A direct extension, **VL-MemKnG**, explicitly “builds directly on the VL-KnG framework” and inherits four components: a **caption-derived spatio-temporal knowledge graph**, **cross-chunk identity consistency**, a **GraphRAG-style graph retrieval pipeline**, and **spatio-temporal object association (STOA)**. In this later work, VL-KnG is described as organizing video observations into a spatio-temporal knowledge graph containing **objects, attributes, and relations**, with repeated objects linked through spatiotemporal association to enable persistent relational reasoning across the trajectory [2606.17183].

The central critique introduced by VL-MemKnG is specific rather than general: **graph-centric retrieval alone may underrepresent broader temporal continuity and contextual cues distributed across long video segments**. According to that paper, VL-KnG remains strong for **object search**, **action-place association**, object-centric localization, and spatial reasoning, but degrades on **temporal-global** and **temporally scattered aggregation** questions. VL-MemKnG addresses this by adding a second persistent memory, **segment-level contextual memory**, alongside the graph.

The new benchmark **WalkieKnowledgeT+** extends the original setup to **8 long egocentric navigation trajectories** and **262 natural-language questions**, adding **temporal-global** and **temporally scattered aggregation** categories. Relative to the strongest VL-KnG variant, VL-MemKnG improves **Retrieval Acc.@1** from **58.33%** to **66.83%**, **Recall@1** from **34.50%** to **40.55%**, and **MRR@1** from **0.583** to **0.668**. At the same time, full-context VLMs still lead on pure answer accuracy in that benchmark, with **Gemini 2.5 Pro** at **67.97** and **VL-MemKnG** at **65.81**. The paper also reports average latency per question of **1.28 s** for VL-MemKnG and **1.24 s** for VL-KnG, preserving the persistent-memory efficiency advantage over long-context VLMs.

The limitations of the original VL-KnG paper remain consequential. The authors identify future work in **dynamic environment handling** and **multi-modal reasoning capabilities**. The graph schema is relatively lightweight; the semantic association method is threshold-based and not fully specified at the level of matching order or optimization details; no formal graph query language is defined; and the benchmark itself is modest in size. Later work adds a further systems limitation: the graph is constructed **offline** and, in VL-MemKnG, **does not support incremental updates**, which constrains use in streaming or continuously changing environments.

Taken together, these developments position VL-KnG as a foundational graph-based memory architecture for navigation-oriented visual reasoning. Its defining contribution is the substitution of a **persistent spatiotemporal graph** for ephemeral long-context inference, while later work clarifies that this object-centric memory is most effective when supplemented by broader temporal context for long-horizon evidence aggregation.

Source: https://www.emergentmind.com/topics/vl-kng