---
title: 'MomaGraph-R1: Zero-Shot Scene Graph & Task Planning'
url: https://www.emergentmind.com/topics/momagraph-r1
type: topic
---

# MomaGraph-R1: Zero-Shot Scene Graph & Task Planning

MomaGraph-R1 is a 7-billion parameter vision–language model designed to predict unified, task-relevant scene graphs and support zero-shot planning for embodied agents operating in household environments. Developed on the Qwen2.5-VL-7B-Instruct backbone and trained through a combination of supervised learning and reinforcement learning, MomaGraph-R1 introduces a compact yet semantically rich representation that integrates spatial, functional, and part-level affordance relationships in household scenes. The model achieves 71.6% accuracy on the MomaGraph benchmark, outperforming prior open-source models by over 11 percentage points, and demonstrates strong generalization to both public benchmarks and physical robot deployments [2512.16909].

## 1. Model Architecture

MomaGraph-R1 employs a multi-component architecture consisting of:

- **Vision–Language Backbone:**  
  - An image encoder, composed of a ResNet-like CNN and a transformer projector, converts each RGB image into 1,024-dimensional visual tokens.
  - A 32-layer transformer text encoder (hidden size 4,096) processes the task instruction.
  - Cross-modal fusion is achieved by inserting visual tokens into the lower layers of the text transformer via cross-attention, yielding a unified 4,096-dimensional feature space.

- **Graph-Prediction Head:**  
  - Multimodal features are passed to a causal language model head (vocabulary ≈32,000), producing a structured JSON string that encodes the scene graph.
  - The JSON schema contains a list of nodes (with unique IDs, labels, and part-level flags) and edges (each with functional and spatial relationship types, and source/target IDs).
  - Node representation $f_n \in \mathbb{R}^{864}$, comprising a 768-dimensional semantic embedding, 32-dimensional part indicator, and 64-dimensional bounding-box encoding.
  - Edge embedding consists of one-hot vectors for functional (6 types: OPEN_OR_CLOSE, ADJUST, CONTROL, ACTIVATE, POWER_BY, PAIR_WITH) and spatial (9 types: LEFT_OF, RIGHT_OF, IN_FRONT_OF, BEHIND, HIGHER_THAN, LOWER_THAN, CLOSE, FAR, TOUCHING) relations, linearly mapped to a 128-dimensional feature.

- **Plan-Generation Head:**  
  - Once the scene graph is generated, it is re-tokenized and combined with the instruction. The same language model then decodes a sequence of high-level actions (e.g., "turn knob X clockwise," "press microwave door handle").

This architecture enables MomaGraph-R1 to predict compact, action-relevant scene representations and generate executable plans in a fully neural, end-to-end fashion.

## 2. Training Procedures

Training proceeds in two stages:

- **Supervised Fine-Tuning (SFT):**  
  The model is trained on approximately 1,050 annotated graphs from the MomaGraph-Scenes dataset. The loss function is a standard token-level cross-entropy over the ground-truth graph JSON sequence:
  $$
  L_{SFT} = -\sum_{t=1}^T \log P_{\theta}(y_t^{gt} \mid y_{<t}^{gt}, I, T)
  $$

- **Reinforcement Learning (DAPO-style Policy Gradient):**  
  Post-SFT, the model is further optimized using a reward-driven RL objective. For each sampled candidate graph $G$ and ground truth $G^{gt}$, the reward combines action agreement, node and edge set overlaps, format correctness, and brevity:
  $$
  R(G, G^{gt}) = w_a R_{action} + R_{edges} + R_{nodes} + w_f R_{format} + w_l R_{length}
  $$
  The RL loss:
  $$
  L_{RL} = \mathbb{E}_{G \sim \pi_{\theta}} \left[ - (R(G, G^{gt}) - b) \log \pi_{\theta}(G \mid I, T) + \beta \, \mathrm{KL}[\pi_{\theta} \,\|\, \pi_{ref}] \right]
  $$
  where $b$ is a learned value-function baseline and $\beta \approx 0.01$ is the KL-penalty coefficient.

This two-phase procedure first instills precise output structure and semantics, then further aligns generation to task-relevant, compact graphs via reward shaping.

## 3. Graph-then-Plan Inference Framework

Inference in MomaGraph-R1 follows a dual-stage process:

1. **Scene-Graph Generation:**  
   Given multi-view images $\{I_i\}_{i=1}^{n}$ and instruction $T$, the model predicts the most likely task-relevant scene graph $G^*$ via beam or greedy search:
   $$
   G^* = \operatorname*{arg\,max}_G P_{\theta}(G \mid \{I_i\}, T)
   $$
   The resulting graph represents nodes (objects/parts), edges (spatial/functional), and part-level affordances in structured JSON format.

2. **Zero-Shot Task Planning:**  
   The predicted graph $G^*$ is encoded as text and prepended to the original instruction. The language model then generates a sequence of high-level actions:
   $$
   A^* = \operatorname*{arg\,max}_A P_{\theta}(A \mid G^*, T)
   $$
   This process requires no additional training or domain-specific engineering; both stages utilize the same underlying LM head.

Graph updates are performed post-action via:
$$
G^{(t+1)} = U(G^{(t)}, a_t, s_{t+1}),
$$
where $U$ prunes or modifies edges according to the observed state change, maintaining a dynamically consistent scene representation after each step.

## 4. Internal Representation and Dynamic Updates

Node representations ($d_n = 864$) combine object semantics, part indicators, and geometric bounding boxes. Edge features ($d_e = 128$) are composed from spatial and functional codes. Explicit part-level nodes enable fine-grained affordance detection (e.g., “knob” vs. “stove”).

Dynamic scene updates ensure that after each action and state transition, the representation remains accurate:
- Edges corresponding to realized functional effects (e.g., knob controlling burner) are retained.
- Edges contradicted by new observations are removed.
- The update operator $U(\cdot)$ is responsible for this pruning and augmentation at each step.

## 5. Empirical Performance and Benchmarking

### MomaGraph-Bench Results

The model is evaluated on zero-shot multi-choice VQA (294 scenes):

| Model                 | Params | Tier-1 | Tier-2 | Tier-3 | Tier-4 | Overall |
|-----------------------|-------:|-------:|-------:|-------:|-------:|--------:|
| Qwen2.5-VL-7B         |   7B   |  62.1% | 58.5%  | 51.9%  | 56.5%  |  60.2%  |
| LLaVA-Onevision       |   7B   |  60.0% | 52.4%  | 58.4%  | 43.4%  |  55.6%  |
| MomaGraph-R1          |   7B   |  76.4% | 71.9%  | 70.1%  | 68.1%  |  71.6%  |

MomaGraph-R1 achieves an 11.4% absolute improvement over its base (Qwen2.5-VL-7B), approaching performance levels of closed-source models.

### Visual Correspondence

| Model                 | BLINK  | MomaGraph-Corr |
|-----------------------|-------:|---------------:|
| DeepSeek-VL2          |  57.4% |        68.4%   |
| Qwen2.5-VL-7B         |  58.7% |        72.7%   |
| LLaVA-Onevision       |  59.7% |        70.7%   |
| MomaGraph-R1          |  63.5% |        77.5%   |

### Real-Robot Transfer (RobotEra Q5; 10 trials)

- Graph-generation success: 80%
- Planning success (given correct graph): 87.5%
- End-to-end task completion: 70%

These results indicate robust generalization from simulation to real-world robotic manipulation [2512.16909].

## 6. Strengths, Limitations, and Prospects

**Strengths:**
- Unified modeling of spatial, functional, and part-level relationships within a single, updatable scene graph.
- Reinforcement learning with graph-alignment rewards yields concise and task-relevant graphs.
- The Graph-then-Plan paradigm supports zero-shot planning and strong generalization across benchmarks and hardware platforms.

**Limitations:**
- Dependence on high-quality graph annotations for RL training rewards poses challenges for scaling to unlabeled datasets.
- JSON-based decoding may generate malformed graphs under severe occlusion.
- Low-level control and fine-grained closed-loop corrections remain outside the model’s planning scope.

**Potential Extensions:**
- Self-supervised graph refinement through learning from failed execution feedback.
- Integration of 3D point-cloud data for enhanced geometric awareness.
- End-to-end training of both high-level graph prediction and low-level motor control for fully closed-loop visuomotor grounding.

MomaGraph-R1 demonstrates that explicit prediction and updating of unified, state-aware scene graphs is an effective approach for vision-language-based embodied task planning, delivering strong zero-shot performance in both virtual and real-world task executions [2512.16909].

Source: https://www.emergentmind.com/topics/momagraph-r1