Spatial-ORMLLM: 3D Reasoning in Operating Rooms

• Spatial-ORMLLM is a large vision-language model that infers 3D spatial and semantic context from single-view RGB images in operating rooms.
• It integrates inferred pseudo-modalities like depth, segmentation, and point cloud using a spatial-enhanced feature fusion block to provide detailed scene understanding.
• It enables accurate intraoperative awareness and decision-making with state-of-the-art performance metrics, eliminating the need for additional sensors or annotations.

Searching arXiv for the specified paper and closely related spatial-reasoning MLLM work. Spatial-ORMLLM is a large vision-LLM for 3D spatial reasoning in operating rooms that uses only RGB modality to infer volumetric and semantic cues, with the stated goal of enabling downstream medical tasks with detailed and holistic spatial context (He et al., 11 Aug 2025). It is presented as the first large vision-LLM for 3D spatial reasoning in operating rooms using only RGB modality, and it addresses a setting in which operating rooms typically lack multiple video and audio sensors, making multimodal 3D data difficult to obtain, while training solely on readily available 2D data fails to capture fine-grained details in complex scenes (He et al., 11 Aug 2025).

1. Clinical problem formulation and scope

Precise spatial modeling in the operating room is described as foundational to many clinical tasks, supporting intraoperative awareness, hazard avoidance, and surgical decision-making (He et al., 11 Aug 2025). Spatial-ORMLLM is motivated by a gap between the spatial requirements of operating-room perception and the practical limitations of data acquisition in clinical environments. The paper identifies two concrete constraints: operating rooms typically lack multiple video and audio sensors, and training solely on 2D data does not capture fine-grained details in complex scenes (He et al., 11 Aug 2025).

The model therefore targets a specific regime: single-view clinical RGB input with implicit recovery of 3D spatial and semantic context. This positioning distinguishes it from systems that depend on RGB-D, LiDAR, multiview arrays, or manual expert annotations at runtime. The paper states that Spatial-ORMLLM requires no additional sensors such as depth, LiDAR, or multiview arrays, and no manual expert annotations at runtime (He et al., 11 Aug 2025).

A recurring misconception in spatial MLLM research is that clinically useful 3D reasoning requires privileged sensing. Spatial-ORMLLM explicitly contests that assumption by internalizing 3D cues from RGB at runtime. Another possible misconception is that it directly observes 3D structure; the paper instead frames depth, segmentation, and point cloud as pseudo-modalities inferred from the image rather than measured sensor streams (He et al., 11 Aug 2025).

2. Core architecture

The architecture enriches single-view RGB images with three complementary pseudo-modalities: a depth map, a panoptic segmentation mask, and a point cloud (He et al., 11 Aug 2025). The depth map $D \in \mathbb{R}^{H \times W}$ encodes per-pixel geometric distance and is monocularly estimated using a transformer-based encoder-decoder. The panoptic segmentation mask $S \in \{1,\ldots,K\}^{H \times W}$ provides semantic and instance delineation of the scene. The point cloud $\mathcal{P}$ is reconstructed as a partial set from inferred depth and camera intrinsics, encoding explicit 3D geometry (He et al., 11 Aug 2025).

The paper formalizes these components as follows. Depth estimation is written as

$$D = \mathcal{F}_d(I) = \mathrm{Dec}(\mathrm{Enc}(I)),$$

with a training loss

$$\mathcal{L}_{\text{depth}} = \lambda_{\ell_1}\|D - D^*\|_1 + \lambda_{\mathrm{grad}}\|\nabla D - \nabla D^*\|_1.$$

Panoptic segmentation is written as

$S = \mathrm{SegDec}(\mathrm{SegEnc}(I)).$

Point cloud generation from depth and camera parameters is written as

$$P_{u,v} = T \cdot \left(K^{-1} \begin{bmatrix} u\ v\ 1 \end{bmatrix} \cdot D(u,v)\right).$$

The set of all $(u,v)$ forms the 3D point cloud $\mathcal{P}$ (He et al., 11 Aug 2025).

These pseudo-modalities are then integrated by the Spatial-Enhanced Feature Fusion Block. In the paper’s description, each pseudo-modality is projected via its own lightweight MLP head into a shared token space, and the resulting tokens are concatenated into a unified sequence:

$$T_{\text{input}} = [T_{\text{img}}, T_{\text{seg}}, T_{\text{pc}}, T_{\text{prompt}}].$$

This sequence is then processed by the LLM backbone, enabling joint reasoning over visual, semantic, and geometric cues (He et al., 11 Aug 2025).

3. Spatial-Enhanced Feature Fusion Block and multimodal reasoning

The Spatial-Enhanced Feature Fusion Block, abbreviated SEFFB in the paper’s ablation description, is the central architectural mechanism for combining RGB evidence with inferred spatial structure (He et al., 11 Aug 2025). The paper characterizes it as integrating 2D modality inputs with rich 3D spatial knowledge extracted by the estimation algorithm and then feeding the combined features into the visual tower. The details further specify token-level fusion rather than early image fusion or simple stacking (He et al., 11 Aug 2025).

Encoding and projection are modality-specific. The paper gives examples such as

$S \in \{1,\ldots,K\}^{H \times W}$0

after which the concatenated sequence is sent to an LLM such as Vicuna-7B (He et al., 11 Aug 2025). The intended effect is that the LLM’s self-attention layers can contextualize across all tokens, enabling implicit spatial alignment and spatial reasoning.

This design places Spatial-ORMLLM within a broader class of systems that augment language reasoning with explicit spatial structure. Earlier prompt-based work improved MLLM spatial awareness by injecting geometric positions and scene graphs into prompts, using Faster R-CNN and PSG without modifying the core model architecture (Zhao et al., 2023). Spatial-ORMLLM instead moves that structured spatial augmentation into an end-to-end token-fusion pipeline. This suggests a shift from prompt-level supplementation to representational integration.

4. Training setup and supervision regime

Spatial-ORMLLM uses a two-stage setup (He et al., 11 Aug 2025). In Stage 1, the LLM is domain-adapted with text-only surgical data. In Stage 2, visual encoders and modality heads are trained to align their features into LLM space while the LLM weights are frozen. The model is described as a unified end-to-end MLLM framework that combines powerful spatial features with textual features to deliver robust 3D scene reasoning without any additional expert annotations or sensor inputs (He et al., 11 Aug 2025).

The paper also states that supervision comes from weakly-labeled or automatically amplified data, leveraging foundation models fine-tuned on public medical imagery (He et al., 11 Aug 2025). It separately emphasizes that no human annotation is required at inference, since only the camera feed is used (He et al., 11 Aug 2025). That distinction matters: training still relies on estimated or amplified supervisory signals, but deployment is framed as RGB-only.

A technical implication is that the model does not reduce operating-room reasoning to monocular captioning. Instead, it inserts geometry-bearing pseudo-modalities into the multimodal token stream before language generation. In general-image spatial reasoning, SpatialLLM similarly argued for systematic integration of 3D-informed data across pre-pretraining, alignment, and instruction tuning, and reported that 3D-informed instruction tuning and alignment were especially important for orientation and composite spatial reasoning (Ma et al., 1 May 2025). Spatial-ORMLLM adapts the same broad intuition—explicitly spatial supervision and fusion matter—but in a clinical RGB-only regime.

5. Empirical performance and ablation results

The paper reports state-of-the-art performance on multiple benchmark clinical datasets and states that the model generalizes robustly to previously unseen surgical scenarios and downstream tasks (He et al., 11 Aug 2025). On the main spatial reasoning benchmark, Spatial-ORMLLM records ROUGE-L 61.2, METEOR 58.6, CIDEr 96.4, EM@1 67.8, and AVG 71.0 (He et al., 11 Aug 2025).

Evaluation	Spatial-ORMLLM	Comparator from paper
Main benchmark AVG	71.0	LLaVA-3D: 65.7
Scene graph F1	84.7	LLaVA-3D: 81.9
Next-Action Prediction F1	36.8	stated as highest on MM-OR
Robot Phase Recognition F1	58.2	stated as best

The same table in the paper lists LLaVA-3D at ROUGE-L 58.9, METEOR 50.5, CIDEr 91.4, EM@1 61.8, AVG 65.7; MM2SG at 58.2, 51.0, 91.7, 61.5, 65.6; ORacle at 53.1, 44.0, 87.1, 55.8, 60.0; and LLaVA-OV at 46.6, 38.8, 60.1, 28.7, 43.6 (He et al., 11 Aug 2025). The paper explicitly notes that Spatial-ORMLLM leads by 5+ points on average, even when other models have access to “real” depth or sensor data (He et al., 11 Aug 2025).

For operating-room scene graph generation, the model achieves Precision 86.1, Recall 83.3, and F1 84.7. The reported comparators are GPT-4V at 83.3/74.0/78.4, MM2SG at 74.9/71.2/73.0, and LLaVA-3D at 83.5/80.4/81.9 (He et al., 11 Aug 2025).

The ablation study is central to the paper’s argument. Removing point cloud, segmentation, or depth estimation each causes significant drops, and without depth estimation the average spatial reasoning score drops from 71.0 to 48.2, while scene graph F1 drops from 84.7 to 75.1 (He et al., 11 Aug 2025). Removing SEFFB also lowers average performance and F1, which the paper interprets as evidence for the superiority of explicit token-level multimodal fusion over early image fusion or stacking (He et al., 11 Aug 2025).

Beyond static reasoning, the model is reported to support downstream tasks. It achieves the highest F1, 36.8%, on MM-OR for Next-Action Prediction, and the best F1, 58.2%, for Robot Phase Recognition (He et al., 11 Aug 2025).

6. Relation to the broader spatial MLLM literature

Spatial-ORMLLM sits within a rapidly expanding literature on spatially intelligent language-vision systems. In general-domain real-image reasoning, SpatialLLM introduced a compound 3D-informed design and reported 62.7% overall accuracy on SpatialVQA, surpassing GPT-4o’s 54.0% by 8.7% (Ma et al., 1 May 2025). In purely 2D observations, Spatial-MLLM used a dual-encoder architecture with a spatial encoder initialized from a visual geometry model and reported 48.4 average accuracy on VSI-Bench, exceeding GPT-4o’s 34.0 and Gemini-1.5 Pro’s 45.4 under the reported input conditions (Wu et al., 29 May 2025). OmniView-Space then emphasized query-aligned egocentric evidence, tool-guided reasoning, and cognitive-map distillation for maintaining spatial consistency in multi-step settings (Li et al., 1 Jul 2026).

Against that backdrop, Spatial-ORMLLM is notable for specializing the spatial-MLLM paradigm to the operating room. Its distinctive claim is not merely better spatial QA, but operating-room 3D reasoning from standard RGB cameras without extra sensors or manual expert annotations at runtime (He et al., 11 Aug 2025). This suggests a clinically pragmatic research direction: internal pseudo-modality construction may substitute for sensor-rich setups when acquisition constraints dominate deployment.

At the same time, the paper should not be read as eliminating the broader challenges documented elsewhere. SpaRC and SpaRP found that state-of-the-art LLMs still perform poorly on controlled spatial reasoning datasets, with strong dependence on model scale, reasoning hops, and targeted finetuning (Rizvi et al., 2024). ORIGAMISPACE similarly showed that MLLMs remain weak in multi-step spatial reasoning under mathematical constraints (Xu et al., 23 Nov 2025). Spatial-ORMLLM therefore appears best understood as a domain-specialized advance in operating-room spatial relation understanding rather than a general solution to spatial reasoning in MLLMs.

Overall, the model’s contribution is the combination of RGB-only pseudo-3D inference, token-level multimodal fusion, and clinical evaluation showing strong results on spatial reasoning, scene graph generation, and downstream surgical tasks (He et al., 11 Aug 2025). A plausible implication is that future operating-room MLLMs may increasingly rely on inferred geometry and semantics as an intermediate representational layer, especially where workflow disruption, sensor cost, and annotation scarcity are dominant constraints.