Interactive Metric World Representation

Updated 5 July 2026

Interactive Metric World Representation is a framework for building world states with explicit metric constraints that preserve geometry and action outcomes.
It employs techniques such as TSDF integration, 3D Gaussian Splatting, and Neural Implicit Scenes to create both explicit and latent scene representations.
The approach is evaluated using benchmarks on action accuracy, physics consistency, and memory retention, highlighting challenges in achieving holistic metric fidelity.

Interactive Metric World Representation denotes a family of world-state constructions in which interaction is constrained by an explicitly or implicitly metric structure rather than by unconstrained frame autoregression alone. In recent work, the term refers most directly to explicit navigable world states such as a hybrid TSDF-plus-topology map for Vision-and-Language Navigation, a 3D Gaussian Splatting scene anchored to a metric panoramic point cloud, a locally anchored Neural Implicit Scene queried by calibrated camera poses, and a pose-indexed global camera trajectory used simultaneously for control and memory (Li et al., 17 Feb 2026, HY-World et al., 15 Apr 2026, Li et al., 29 Jun 2026, Nam et al., 17 Mar 2026). A broader reading, supported by adjacent papers, extends the idea to multi-modal manipulation models with metric depth, pose-free long-horizon memories that approximate metric behavior without explicit geometry, and benchmark suites that operationalize “metric world” competence through action, consistency, physics, and memory metrics (Zhang et al., 10 Oct 2025, Wu et al., 2 Feb 2026, Xu et al., 30 Jun 2026).

1. Definitions and representational scope

The literature does not use a single canonical data structure for Interactive Metric World Representation. Instead, it uses the term for world states that preserve consistent geometry, pose, scale, or action consequences under interaction. In the most explicit formulation, IMWR is “a structured, metric-scale, interactive model of the navigable world built online from RGB-D observations and robot poses,” with all geometry represented in a real-world Euclidean frame with consistent units, rigid-body transformations in $SE(3)$ , and scale-preserved reconstructions (Li et al., 17 Feb 2026). In generative world modeling, closely related constructs include a 3DGS scene in a panorama-centered world frame (HY-World et al., 15 Apr 2026), a fixed-length latent token set $z \in \mathbb{R}^{L\times D}$ that is locally anchored in $SE(3)$ (Li et al., 29 Jun 2026), and a global camera pose $T_t$ obtained by composing action-induced motions on the $SE(3)$ manifold (Nam et al., 17 Mar 2026).

System	World state	Metric anchor
One Agent to Guide Them All	$W_t=\langle M_{vol}, G_{topo}\rangle$	meters, $SE(3)$ , TSDF, ray-casting
HY-World 2.0	3DGS + mesh + NavMesh	panorama-origin frame, MoGe-2 metric depth
NeuWorld	NIS $z \in \mathbb{R}^{L\times D}$	local $SE(3)$ frame, calibrated $K$
WorldCam	global camera poses $z \in \mathbb{R}^{L\times D}$ 0	$z \in \mathbb{R}^{L\times D}$ 1 composition from actions

This suggests that “metric” has at least three distinct meanings in current usage. First, it can mean explicit Euclidean reconstruction with physical units and rigid transforms, as in TSDF or 3DGS-based systems. Second, it can mean pose-consistent local state spaces whose geometry is inherited from calibrated cameras and local frame normalization, even if the state itself is latent rather than explicit. Third, it can mean benchmark-defined measurable competence: correct action response, stable geometry, plausible physics, and faithful memory over long horizons (Xu et al., 30 Jun 2026).

Adjacent work uses the phrase more broadly. iMoWM models RGB, metric depth, and robot-arm masks as a multi-modal world state for robotic manipulation (Zhang et al., 10 Oct 2025). “Convergent World Representations and Divergent Tasks” studies metric latent organization over 5,075 city coordinates and seven geometric tasks (Park, 31 Jan 2026). MM-RealSR uses a 2D metric-space coordinate $z \in \mathbb{R}^{L\times D}$ 2 for interactive restoration strength control (Mou et al., 2022). MetricGrids uses multiple elementary metric grids to approximate complex nonlinear signals in implicit neural representation (Wang et al., 13 Mar 2025). These uses are not identical, but they preserve the core idea that interaction is mediated by a structured metric state rather than by unstructured outputs.

2. Explicit geometric world states

The explicit-navigation formulation defines IMWR as a hybrid state $z \in \mathbb{R}^{L\times D}$ 3, where $z \in \mathbb{R}^{L\times D}$ 4 is a dense 3D Truncated Signed Distance Field volume and $z \in \mathbb{R}^{L\times D}$ 5 is a topological memory graph over metric poses (Li et al., 17 Feb 2026). Coordinate frames are world/map $z \in \mathbb{R}^{L\times D}$ 6, robot body/base $z \in \mathbb{R}^{L\times D}$ 7, and camera $z \in \mathbb{R}^{L\times D}$ 8, with

$z \in \mathbb{R}^{L\times D}$ 9

Back-projection uses

$SE(3)$ 0

TSDF integration is performed by weighted averaging,

$SE(3)$ 1

with free-space carving along camera rays. The topological graph stores nodes $SE(3)$ 2, merges nodes with $SE(3)$ 3, tracks a bounded history $SE(3)$ 4 with window size $SE(3)$ 5, and raises “Upstairs/Downstairs” alerts when $SE(3)$ 6. The MLLM interacts with this state through orthographic BEV, four egocentric RGB-D views, a normalized grid $SE(3)$ 7, and structured JSON outputs whose coordinates are grounded by

$SE(3)$ 8

HY-World 2.0 realizes an explicit metric world state through 3D Gaussian Splatting (HY-World et al., 15 Apr 2026). A scene is represented by $SE(3)$ 9 unstructured 3D Gaussians with center $T_t$ 0, covariance $T_t$ 1, color $T_t$ 2, opacity $T_t$ 3, and an optional MaskGaussian $T_t$ 4. Screen-space covariance is

$T_t$ 5

and compositing is front-to-back: $T_t$ 6 The world coordinate frame is centered at the panorama origin. A panoramic point cloud $T_t$ 7 is reconstructed with MoGe-2, and generated keyframe depths $T_t$ 8 are aligned to $T_t$ 9 through a per-view RANSAC linear model in disparity space,

$SE(3)$ 0

All generated views remain in the same world frame as $SE(3)$ 1, while Global NavMesh and collision-based trajectory curation constrain camera height, proximity to obstacles, and traversable surfaces.

These explicit constructions differ in surface representation—TSDF volume versus 3DGS scene—but they share a common technical commitment: world interaction is mediated by persistent geometry rather than by latent video state alone. In the navigation setting, this yields physically valid waypoints and collision checks. In HY-World 2.0, it yields a renderable 3D world with mesh extraction for physics and collision, automatic image-based lighting, and real-time navigation after MaskGaussian pruning.

3. Latent scene states, camera poses, and pose-free memory

NeuWorld changes the rollout variable from frame latents to a fixed-length renderable state, the Neural Implicit Scene (NIS) (Li et al., 29 Jun 2026). At interaction step $SE(3)$ 2, the scene state is $SE(3)$ 3 with $SE(3)$ 4 tokens and $SE(3)$ 5 channels. The tokens are locally anchored to a chosen reference camera and re-anchored as the agent moves. The factorization is

$SE(3)$ 6

where $SE(3)$ 7 is the camera pose, $SE(3)$ 8 is a short-horizon future pose segment and optional reference image, and $SE(3)$ 9 is retrieved history. The encoder and decoder operate on ray tokens built from pixel-wise Plücker embeddings of calibrated camera rays. All camera extrinsics are transformed into the local frame defined by the current reference pose $W_t=\langle M_{vol}, G_{topo}\rangle$ 0, while translation magnitudes are normalized using farthest-camera-distance statistics and random rescaling. Conditioning is unified by reusing the frozen VAE encoder for pose-only partial NIS, pose-plus-reference partial NIS, and retrieved history memory $W_t=\langle M_{vol}, G_{topo}\rangle$ 1. Long-horizon stability is supported by geometry-aware retrieval and anti-drift condition augmentation.

WorldCam uses camera pose as the unifying geometric representation for both action control and memory (Nam et al., 17 Mar 2026). User actions are mapped to a spatial twist in the Lie algebra $W_t=\langle M_{vol}, G_{topo}\rangle$ 2,

$W_t=\langle M_{vol}, G_{topo}\rangle$ 3

and converted to $W_t=\langle M_{vol}, G_{topo}\rangle$ 4 by the matrix exponential. Global pose is accumulated as

$W_t=\langle M_{vol}, G_{topo}\rangle$ 5

The model conditions a video DiT on Plücker embeddings derived from accumulated global poses and retrieves long-term memory by pose proximity: first Top- $W_t=\langle M_{vol}, G_{topo}\rangle$ 6 by translation $W_t=\langle M_{vol}, G_{topo}\rangle$ 7, then Top- $W_t=\langle M_{vol}, G_{topo}\rangle$ 8 by orientation alignment $W_t=\langle M_{vol}, G_{topo}\rangle$ 9. This gives camera pose a dual role: it is both the control variable and the memory index.

Infinite-World takes the opposite position: it is explicitly pose-free (Wu et al., 2 Feb 2026). Its Hierarchical Pose-free Memory Compressor (HPMC) compresses long histories into a fixed-budget representation with $SE(3)$ 0 tokens, $SE(3)$ 1 sliding windows, and window size $SE(3)$ 2. Its action interface is based on Uncertainty-aware Action Labeling, which maps estimated motion $SE(3)$ 3 into tri-state logic: $SE(3)$ 4 The paper states that there are “no explicit metric distances or scales”; instead, long-range spatial consistency emerges from data-driven distillation, revisit-dense finetuning, and memory compression. A plausible implication is that Interactive Metric World Representation now spans a continuum from explicit metric reconstruction to representations that only approximate metric behavior through loop closure and long-horizon coherence.

4. Construction pipelines and interactive interfaces

In the most explicit embodied interface, the MLLM does not operate directly on raw RGB streams but on a serialized world state (Li et al., 17 Feb 2026). The Interactive Reasoning Interface includes BEV image $SE(3)$ 5, four egocentric RGB-D views aligned with cardinal angles $SE(3)$ 6, topological state, alerts, task plan, recent history, and immutable instruction. The model writes a structured JSON with a source view and normalized coordinates $SE(3)$ 7, then the system recovers a metric waypoint by ray-casting in $SE(3)$ 8. Candidate actions are assessed through counterfactual reasoning over free-space corridors, obstacles, the TODO list, and loop avoidance. Physical validity is enforced by TSDF free-space thresholds, footprint clearance, and deterministic local planning through Nav2.

HY-World 2.0 exposes a generative construction pipeline rather than an online semantic interface (HY-World et al., 15 Apr 2026). The text- or image-conditioned path is a four-stage method: panorama generation with HY-Pano 2.0, trajectory planning with WorldNav, world expansion with WorldStereo 2.0, and world composition with WorldMirror 2.0. WorldNav constructs $SE(3)$ 9, a low-resolution panoramic mesh, semantic masks, and a NavMesh with vertex snapping to ground via dense ray casting, KD-tree boundary erosion, and bridging of isolated areas. It supports five heuristic collision-free trajectory modes: regular, surrounding, reconstruction-aware, wandering, and aerial. WorldStereo 2.0 uses a Keyframe-VAE, a camera control branch, Global-Geometric Memory from point clouds, and Spatial-Stereo Memory retrieval. WorldMirror 2.0 predicts depth and normals, aligns depth to $z \in \mathbb{R}^{L\times D}$ 0, fuses $z \in \mathbb{R}^{L\times D}$ 1, initializes 3DGS, and optimizes it with

$z \in \mathbb{R}^{L\times D}$ 2

Deployment in WorldLens adds automatic IBL lighting, mesh-based collision, character motion, and real-time tile-based GPU splatting. End-to-end world generation averages $z \in \mathbb{R}^{L\times D}$ 3 on NVIDIA H20, with Panorama $z \in \mathbb{R}^{L\times D}$ 4, Trajectory $z \in \mathbb{R}^{L\times D}$ 5, WorldStereo $z \in \mathbb{R}^{L\times D}$ 6, Reconstruction+Align $z \in \mathbb{R}^{L\times D}$ 7, and 3DGS $z \in \mathbb{R}^{L\times D}$ 8.

These systems show two complementary interface philosophies. One exposes a physically grounded map to a reasoning model that outputs grounded actions. The other constructs a high-fidelity metric scene first and then exposes it to interactive rendering, navigation, and collision. In both cases, interaction is validated against persistent geometry rather than against unconstrained image synthesis.

5. Evaluation as an operational definition

WorldRoamBench treats interactive metric world representation as a measurable property along four long-horizon dimensions: action following, vision, physics, and memory (Xu et al., 30 Jun 2026). Action is scored by strict accuracy, partial accuracy, and trajectory geometry: $z \in \mathbb{R}^{L\times D}$ 9 Vision combines absolute quality and segment-based drift: $SE(3)$ 0 Physics is controllability-gated and aggregates mechanics, optics, and 3D consistency,

$SE(3)$ 1

Memory uses transition-localized 3D reconstruction, retention, hallucination, and F1,

$SE(3)$ 2

and the overall score is

$SE(3)$ 3

The benchmark contains 600+ test cases across Nature, Urban, and Indoor scenes, in first- and third-person views, with 10–60 s continuous interaction. Its principal finding is that no evaluated model reliably satisfies all four dimensions.

WorldMark standardizes cross-model comparison by enforcing identical scenes, identical action sequences, and a unified control interface (Xu et al., 23 Apr 2026). It provides 500 evaluation cases across first- and third-person viewpoints, photorealistic and stylized scenes, and three difficulty tiers spanning 20–60 s. The unified action vocabulary is $SE(3)$ 4, and model-specific adapters translate these primitives into caption prompts, structured 6-DoF pose parameters, gamepad controls, custom action functions, or 25-dimensional continuous vectors. Visual Quality uses LAION aesthetic predictor and MUSIQ. Control Alignment reconstructs trajectories with DROID-SLAM and evaluates scale-invariant translation error

$SE(3)$ 5

and rotation error

$SE(3)$ 6

World Consistency uses DROID-SLAM DBA reprojection error and VLM-based state, content, and style consistency. The benchmark reports that visual quality and world consistency are largely uncorrelated, precise control alignment does not guarantee overall quality, and third-person control is a pronounced failure mode.

Taken together, these benchmarks recast IMWR from a purely representational claim into an operational one. A model qualifies only insofar as it preserves action semantics, geometric coherence, plausible physics, and memory under long-horizon interaction.

6. Misconceptions, limitations, and adjacent extensions

A common misconception is that “metric” always implies a persistent global Euclidean map. The literature is more heterogeneous. One Agent to Guide Them All uses meters, $SE(3)$ 7, TSDF integration, and ray-casting in a global frame (Li et al., 17 Feb 2026). HY-World 2.0 anchors scale through MoGe-2 $SE(3)$ 8, NavMesh-constrained trajectories, and per-view RANSAC scale-shift alignment in disparity space (HY-World et al., 15 Apr 2026). NeuWorld is metrically anchored only in a local $SE(3)$ 9 frame, and the paper states that absolute world scale follows posed data normalization (Li et al., 29 Jun 2026). Infinite-World explicitly states that there are “no explicit metric distances or scales,” and that exact metric reconstruction is traded away for robustness to noisy real-world videos and bounded long-horizon compute (Wu et al., 2 Feb 2026).

A second misconception is that interactivity requires explicit 3D mesh or volumetric geometry. This is only one design point. WorldCam achieves precise action controllability and long-horizon revisiting by centralizing camera pose in $K$ 0 and using pose-indexed retrieval, without explicit depth maps in the core generation loop (Nam et al., 17 Mar 2026). NeuWorld separates stochastic scene transition from deterministic rendering through a fixed-size NIS state rather than an explicit surface representation (Li et al., 29 Jun 2026). Infinite-World uses a pose-free compressed memory and revisit-dense finetuning to approximate loop closure (Wu et al., 2 Feb 2026).

The main limitations are also consistent across the literature. HY-World 2.0 is an offline high-fidelity pipeline; “dynamics/long-horizon” are not its primary target, and very long explorations may accumulate inconsistency (HY-World et al., 15 Apr 2026). NeuWorld has a local state that is re-anchored over time and “there is no persistent global map,” which can limit global-scale consistency in very long rollouts (Li et al., 29 Jun 2026). Infinite-World reports residual “cumulative drift and visual degradation,” especially without revisits (Wu et al., 2 Feb 2026). One Agent to Guide Them All depends on depth quality and pose estimates, and severe noise can degrade TSDF fidelity (Li et al., 17 Feb 2026). WorldCam notes drift accumulation, runtime cost from progressive diffusion, noisy pseudo ground-truth poses from ViPE, and scale ambiguity across clips (Nam et al., 17 Mar 2026). WorldRoamBench and WorldMark both show that even strong models separate into different failure modes across action, visual quality, physics, and memory (Xu et al., 30 Jun 2026, Xu et al., 23 Apr 2026).

Adjacent extensions show that the concept is expanding beyond navigation and exploratory video. iMoWM uses RGB, metric depth, and robot-arm masks as an action-conditioned multi-modal world state for robotic manipulation (Zhang et al., 10 Oct 2025). MetricGrids generalizes implicit neural representation by defining multiple feature grids over different metric spaces (Wang et al., 13 Mar 2025). MM-RealSR constructs a 2D metric space for interactive restoration strength (Mou et al., 2022). “Convergent World Representations and Divergent Tasks” studies how geometric tasks induce aligned or divergent latent world structures over entity coordinates (Park, 31 Jan 2026). This suggests that Interactive Metric World Representation is becoming a cross-domain organizing idea: interaction is reliable when the underlying state preserves measurable geometry, calibrated action consequences, and stable update rules across time and scale.