DyGEnc: Encoding a Sequence of Textual Scene Graphs to Reason and Answer Questions in Dynamic Scenes

Abstract: The analysis of events in dynamic environments poses a fundamental challenge in the development of intelligent agents and robots capable of interacting with humans. Current approaches predominantly utilize visual models. However, these methods often capture information implicitly from images, lacking interpretable spatial-temporal object representations. To address this issue we introduce DyGEnc - a novel method for Encoding a Dynamic Graph. This method integrates compressed spatial-temporal structural observation representation with the cognitive capabilities of LLMs. The purpose of this integration is to enable advanced question answering based on a sequence of textual scene graphs. Extended evaluations on the STAR and AGQA datasets indicate that DyGEnc outperforms existing visual methods by a large margin of 15-25% in addressing queries regarding the history of human-to-object interactions. Furthermore, the proposed method can be seamlessly extended to process raw input images utilizing foundational models for extracting explicit textual scene graphs, as substantiated by the results of a robotic experiment conducted with a wheeled manipulator platform. We hope that these findings will contribute to the implementation of robust and compressed graph-based robotic memory for long-horizon reasoning. Code is available at github.com/linukc/DyGEnc.

DyGEnc: Encoding a Sequence of Textual Scene Graphs to Reason and Answer Questions in Dynamic Scenes

The paper presents DyGEnc, a novel approach to encode dynamic scene graphs for question answering in environments that are continuously changing. The challenge, as outlined by the authors, lies in providing interpretable spatial-temporal object representations from visual models, which are often implicit and opaque. DyGEnc integrates LLM capabilities with compressed spatial-temporal structural observation representation to address this issue.

Key Contributions and Methodology

DyGEnc incorporates several components in its architecture:

1. Graph Encoding: The method employs text attributes from nodes and edges, which are processed by a pre-trained text encoder, ModernBert. These attributes are aggregated into graph tokens using a GraphTransformer model.
2. Sequence Encoding: Temporal continuity is preserved using a Rotary Positional Encoding mechanism. The sequence of graph tokens is then compressed into a compact representation using a Q-Former module to handle temporal relations efficiently.
3. LLM Fine-Tuning: The approach uses a parameter-efficient tuning strategy of the LLM, further aligning the compressed graph token space into the LLM's embedding space to facilitate reasoning grounded in sensory input.

Through extensive benchmarking with STAR and AGQA datasets, DyGEnc demonstrates superior performance in addressing questions related to human-object interaction history, outperforming existing visual methods by 15–25%. The tangible margin suggests significant improvements in handling queries about dynamic scenes, which are integral to real-world applications in robotics.

Implications and Future Directions

The implication of these findings is multifold. Firstly, the architecture proposes a refined method to augment robotic memory systems for long-horizon reasoning, potentially paving the way for more adaptive and reliable intelligent agents in dynamic environments. The ability to encode scene graph sequences compactly ensures that large volumes of data do not lead to fallacious reasoning, thus enhancing the robustness of autonomous systems.

In practical terms, the method can be employed for real-world robotic applications, as illustrated by experiments involving a wheeled manipulator platform. The seamless extension of DyGEnc to process raw input images through foundational models broadens its applicability and could serve as a cornerstone for further advancements in robotic perception.

Looking ahead, an exploration involving 3D scene graphs and multimodal learning could provide additional complexity and depth to the current framework. Such advancements might include integrating object tracking processes, which would further improve the temporal precision in understanding scene dynamics.

In conclusion, DyGEnc presents a significant advancement in the encoding of dynamic scene graphs by integrating LLMs with structured graph sequence encoding. The results from the STAR and AGQA datasets affirm its effectiveness and highlight its promise in enhancing robotic cognitive capabilities, thereby driving the field towards more intuitive and autonomous systems.

The authors cite foundational works and datasets such as Visual Genome, GQA, PSG, and others which have provided the framework for their development. Enhancements in text encoding and graph neural network models have been pivotal in achieving the context-aware reasoning illustrated through the DyGEnc methodology. The use of Open-source models and adaptive training strategies strengthens the model's applicability in diverse scenarios.