Language as a Sensor: Calibrated Spatial Belief Estimation in 3D Scenes from Natural Language

Abstract: Robots deployed in human-centric environments routinely receive natural-language descriptions of spatial information ("I left my backpack on the table") that reference parts of the world beyond their perceptual field of view. Traditional metric-semantic mapping ignores this signal, while off-the-shelf multimodal models remain limited in 3D spatial reasoning and are not directly amenable to fusion with other sensor modalities. To convert language observations into a calibrated spatial distribution, we train a Language Sensor Model (LSM) that maps each utterance and its scene-graph context to a multimodal distribution, with mixture weights encoding referential ambiguity (e.g., "which table") and component covariances encoding spatial uncertainty (e.g., where "on the table" the target lies). We then introduce VL-Map (Vision-Language Metric-Semantic Mapping), a probabilistic framework that treats these language predictions as stochastic observations and fuses them with onboard perception within a unified belief map. On the VLA-3D benchmark as well as on a real-world mobile robot, LSM is the only language predictor whose covariance estimates remain within the calibrated regime; fused into VL-Map, it leads to more accurate predictions of the target object location (~70% more probability mass on the true target compared to the strongest foundation-model baseline).

• The paper introduces a Language Sensor Model (LSM) that generates calibrated Gaussian mixtures from language-scene pairs to reflect spatial uncertainty.
• The VL-Map framework enables Bayesian fusion of language-derived and visual cues, significantly enhancing closed-loop spatial belief updates.
• Experimental results show LSM reduces RMSE and mitigates overconfidence, demonstrating robust integration of language as a probabilistic sensor in robotics.

Calibrated Spatial Inference from Natural Language: Treating Language as a Probabilistic Sensor

Introduction

"Language as a Sensor: Calibrated Spatial Belief Estimation in 3D Scenes from Natural Language" (2606.08666) addresses the critical challenge of exploiting natural language as a quantitative probabilistic input for spatial reasoning in robotics. While mobile robots receive high-level spatial cues through language (e.g., "I left my backpack on the table"), standard metric-semantic mapping frameworks either ignore such information or fail to properly model uncertainty, resulting in brittle and overconfident inferences. Existing 3D vision-LLMs, though capable of producing deterministic spatial groundings, do not output calibrated probabilistic distributions essential for sensor fusion and recursive state estimation.

The paper introduces two key innovations: a Language Sensor Model (LSM) that outputs calibrated Gaussian mixture models (GMMs) from utterance-scene pairs, and VL-Map, a Bayesian fusion architecture allowing integration of language-derived uncertainty with visual evidence for closed-loop spatial belief updates.

Language Sensor Model (LSM): Probabilistic Language Grounding

The LSM formalizes the notion of language as an ambiguous spatial sensor. Given a natural language utterance and a scene graph, LSM performs a two-stage probabilistic mapping:

• Referential Ambiguity: Utilizes a prompt-based LLM to generate and soft-rank anchor hypotheses (e.g., which "table" is referred to) by parsing the utterance and scene contents.
• Spatial Ambiguity: For each anchor hypothesis, a transformer-based module conditioned on spatial embedding and contextual object features predicts a Gaussian over plausible object locations.

The probabilistic model is decomposed as:

$$P(m_t \mid L_j, m^-) = \sum_{h_i \in H(L_j, m^-)} P(m_t \mid h_i, m^-) \cdot P(h_i \mid L_j, m^-)$$

where $H$ indexes anchor hypotheses, and the per-hypothesis factor is realized via a geometry- and language-aware transformer.

This approach enforces explicit calibration, where mixture weights reflect referential potential and component covariances capture spatial underspecification. Training employs a combined negative log-likelihood and $L_1$ regression objective to promote both probabilistic and direct spatial accuracy.

VL-Map: Bayesian Fusion of Language and Perception

The VL-Map framework fuses language-based spatial likelihoods and visual observations within a voxelized workspace to recursively update posterior beliefs over object locations. Under a conditional independence assumption,

$$P(L_{1:M}, Z_{1:T} \mid m_t) = \prod_{t=1}^T P(Z_t \mid m_t) \prod_{j=1}^M P(L_j \mid m_t)$$

language and vision act as parallel, asynchronous sensing channels. The LSM output is incorporated into the belief update using additive log-odds per voxel, ensuring compatibility with real-time mappers such as Hydra.

This formulation admits direct Bayesian fusion, allowing language cues (with calibrated uncertainty) to propagate and be refined as streaming vision accrues, supporting both anticipatory localization and correction of overconfident/erroneous beliefs.

Experimental Evaluation

Extensive validation is conducted on the VLA-3D benchmark and a Boston Dynamics Spot robot. Notably:

• Calibration and Accuracy: LSM is the only model yielding ANEES (average normalized estimation error squared) within the calibrated interval (e.g., 1.72 on val_seen, 3.14 on val_unseen), reflecting uncertainty estimates commensurate with realized spatial error. Foundation model baselines (Scaffolded-LLM, Scaffolded-VLM) demonstrate severe overconfidence (ANEES 43–80), leading to frequent catastrophic Bayesian fusion failures.
• Spatial Accuracy: LSM halves the RMSE compared to Scaffolded-LLM and reduces negative log-likelihood (NLL) by an order of magnitude. Calibration persists across scene splits, supporting transferability.
• Closed-Loop Fusion: When integrated with VL-Map, LSM achieves significantly higher information gain and maintains approximately 70% more probability mass on the true target location compared to the strongest baseline. In visually unobserved regimes, LSM remains the only method with consistently positive information gain, demonstrating robust belief maintenance in the face of partial observability.
• Failure Modes: Overconfident and uncalibrated language predictors actively degrade the posterior, frequently yielding worse performance than ignoring language. LSM's calibrated outputs mitigate this issue, substantially lowering the frequency of confidence-induced belief collapse.

Practical and Theoretical Implications

From a systems perspective, LSM and VL-Map enable the treatment of language as a first-class probabilistic sensor, not merely as a brittle symbolic cue. The calibrated uncertainty allows for principled integration with other sensory modalities, supporting robust reasoning under ambiguity and partial observability. This has major implications for autonomous systems operating in human environments, facilitating anticipation and planning based on off-board knowledge sources prior to direct perception.

Theoretically, the explicit modeling of referential and spatial ambiguity highlights the necessity of uncertainty-aware intermediate representations for multi-modal grounding. The results challenge the sufficiency of deterministic or self-confident outputs from large vision-LLMs for downstream Bayesian inference tasks.

Limitations and Future Directions

The current LSM implementation assumes all anchor objects referenced in language exist in the prior scene graph, restricting its applicability in cases of incomplete environment mapping or out-of-map references. Additionally, no probability mass is assigned outside the mapped workspace extent. Addressing these limitations by integrating mechanisms for reinterpretation or clarification in the presence of persistent evidence-language disagreement, or for extrapolating to unmapped regions, is a promising direction.

Further, incorporating interactive dialogue for resolving referential ambiguity, continual calibration updates, and generalizing to other forms of linguistic observation (temporal, relational affordances) would extend the utility of language as a multimodal sensor.

Conclusion

The presented approach delivers a rigorous, uncertainty-aware probabilistic interface for translating natural language descriptions into actionable spatial beliefs in robots. By introducing calibrated language-derived likelihoods and enabling their seamless Bayesian fusion with visual data, the framework achieves robust spatial reasoning under ambiguity, outperforming uncalibrated multimodal model baselines across both simulated and real-world tasks. The methodological foundation laid by LSM and VL-Map will inform future research on interactive, trustworthy, and adaptive multi-modal state estimation in embodied AI systems.