Symbolic graph integration with vision-language models

Unify discrete graph-based representations of egocentric interactions with the continuous embeddings and open-ended language interface of vision-language models.

Background

The survey presents graphs as a way to represent hands, objects, contact, and manipulation relations explicitly, thereby addressing the noun-shortcut bias of dense vision-LLMs. However, graph representations are discrete and symbolic, whereas VLMs operate over continuous embeddings.

Existing attempts inject hypergraphs into LLMs or use scene graphs to generate training supervision, but the survey states that these approaches do not yet provide a unified solution combining robust egocentric graph construction, long-horizon reasoning, and open-ended language interaction.