GaLa: Hypergraph-Guided Visual Language Models for Procedural Planning

Abstract: Implicit spatial relations and deep semantic structures encoded in object attributes are crucial for procedural planning in embodied AI systems. However, existing approaches often over rely on the reasoning capabilities of vision LLMs (VLMs) themselves, while overlooking the rich structured semantic information that can be mined from multimodal inputs. As a result, models struggle to effectively understand functional spatial relationships in complex scenes. To fully exploit implicit spatial relations and deep semantic structures in multimodal data, we propose GaLa, a vision language framework for multimodal procedural planning. GaLa introduces a hypergraph-based representation, where object instances in the image are modeled as nodes, and region-level hyperedges are constructed by aggregating objects according to their attributes and functional semantics. This design explicitly captures implicit semantic relations among objects as well as the hierarchical organization of functional regions. Furthermore, we design a TriView HyperGraph Encoder that enforces semantic consistency across the node view, area view, and node area association view via contrastive learning, enabling hypergraph semantics to be more effectively injected into downstream VLM reasoning. Extensive experiments on the ActPlan1K and ALFRED benchmarks demonstrate that GaLa significantly outperforms existing methods in terms of execution success rate, LCS, and planning correctness.

• The paper introduces a hypergraph-based semantic abstraction that models both object and regional cues to reduce planning errors in complex scenes.
• The methodology employs a Tri-View HyperGraph Encoder with contrastive objectives to robustly integrate high-order spatial and semantic relations.
• Experimental results on ActPlan-1K and ALFRED benchmarks demonstrate improved executability and correctness over prior VLM-based approaches.

GaLa: Hypergraph-Guided Visual LLMs for Procedural Planning

Introduction and Motivation

Procedural planning for embodied AI fundamentally depends on models’ ability to exploit the latent semantic and spatial relations within complex visual scenes. Existing Vision-LLMs (VLMs) often restrict reasoning to geometric and explicit representations, such as bounding boxes and object coordinates, thus neglecting implicit relations encoded by object attributes and functional regions. This oversimplification frequently induces degenerate planning trajectories, especially in ambiguous or counterfactual scenarios. "GaLa: Hypergraph-Guided Visual LLMs for Procedural Planning" (2604.17241) introduces a hypergraph-based semantic abstraction, explicitly modeling functional and spatial relations as an intermediate representation. This framework is designed to inject structured semantic knowledge into VLMs, improving both the interpretability and coherency of multimodal procedural planning.

Figure 1: Visualization of the effect of introducing hypergraph-based semantic modeling versus standard VLM reasoning, highlighting logical deadlocks without global structural context.

Hypergraph Semantic Representation

GaLa leverages a hypergraph where object instances in an image correspond to nodes, and hyperedges aggregate objects via shared attributes and inferred functional semantics (region-level). This approach explicitly models:

• Object-level semantics: Nodes encode localized semantic and spatial descriptors derived from open-vocabulary object detectors (YOLO-World).
• Area-level structure: DBSCAN is applied for spatial clustering, and region semantics are reasoned via LLM prompting over grouped object categories.
• Counterfactual attribution: Attributes are scored using LLMs to distinguish standard from counterfactual states, ensuring robustness under non-standard environmental configurations.

The hypergraph captures hierarchical and high-order relations, mitigating the relational blindness present in token-level VLM pipelines.

Tri-View HyperGraph Encoder and Contrastive Learning

To disseminate structured hypergraph semantics into the VLM components, GaLa introduces a Tri-View HyperGraph Encoder. This module enforces semantic consistency across three views via contrastive objectives:

1. Node view: Focused on object-level invariances across augmentations of the scene.
2. Area view: Structured group-level (region/cluster) contrast for aligning semantic region representations.
3. All view: Enforces node–hyperedge association, explicitly discriminating real and spurious memberships.

Projection heads disentangle semantic encoding from contrastive optimization, and InfoNCE loss is adopted across all views. This mechanism yields robust embeddings that encapsulate both fine-grained object semantics and higher-order spatial hierarchies.

Figure 2: Overview of the GaLa procedural planning pipeline, which integrates hypergraph construction, tri-view semantic optimization, and downstream VLM-guided action prediction.

Figure 3: Detailed schematic of the Tri-View HyperGraph Encoder architecture, illustrating the interplay between node, area, and all-level reasoning within a Chain-of-Thought pipeline.

Procedural Planning: Training and Inference in GaLa

GaLa incorporates the generated hypergraph knowledge into InternVL3-based VLMs. During training, visual input, textual instructions, and the refined hypergraph representation are fused to directly supervise action-sequence prediction. At inference, the model utilizes distilled hypergraph representations for next-action prediction, with improved awareness of implicit spatial constraints and object-region co-dependencies.

Experimental Results

On two challenging benchmarks—ActPlan-1K and ALFRED—GaLa demonstrates significant advances over prior work. Notable empirical results include:

• On ActPlan-1K (counterfactual activities): 55.1% Executability, 0.59 LCS, 37.2% Correctness.
• On ActPlan-1K (normal): 65.8% Executability, 0.66 LCS, 47.5% Correctness.
• On ALFRED: 89.9% Executability, 0.67 LCS, 50.1% Correctness.
• Consistently superior to baselines such as LLaPA, Embodied-GPT, and InternVL3-based models.

Ablation studies confirm that both the hypergraph module and the tri-view contrastive learning individually contribute to these performance gains.

Figure 4: Illustration of the instruction decomposition process demonstrating how GaLa utilizes hierarchical scene semantics for action prediction.

Figure 5: Ablation study results indicating the contribution of hypergraph construction and tri-view contrastive objectives on ALFRED metrics.

Figure 6: Comparative examples of predicted action sequences, highlighting GaLa's improved logical consistency and spatial awareness relative to conventional approaches.

Practical and Theoretical Implications

The introduction of hypergraph-guided procedural planning in GaLa provides several implications:

• Explicit structural modeling: Delivers interpretable intermediate representations, facilitating debugging and error analysis.
• Improved robustness: The model’s ability to reason under counterfactual and ambiguous scenarios suggests higher reliability for deployment in open-world embodied agents.
• Chain-of-Thought alignment: The chain-of-thought organization of hypergraph reasoning enables progressive improvement and refinement of scene understanding, directly benefiting complex planning tasks where stepwise consistency is essential.

From a theoretical perspective, this approach situates high-order graph-theoretic modeling as a critical component for next-generation multimodal reasoning frameworks, bridging gaps left by token- or patch-based fusion methods.

Limitations and Future Directions

Some limitations remain: the constructed hypergraphs are only as comprehensive as the available visual evidence allows, and ActPlan-1K provides limited imagery per instance, which may occlude certain semantic relations. Additionally, while GaLa leverages basic hypergraph-theoretic constructs, future work may investigate more advanced structural priors (e.g., dynamic hypergraphs, richer relational predicates) to further elevate planning fidelity.

Conclusion

GaLa represents a shift in multimodal procedural planning, foregrounding explicit high-order semantic abstraction via hypergraphs and tri-view contrastive learning. The resulting models show substantially improved action-sequence prediction in both conventional and counterfactual settings, underscoring the value of structured scene graph integration for robust VLM-based embodied planning. This work motivates deeper investigations into high-order semantic fusion for generalized multimodal reasoning and long-horizon task planning.