What Images Cannot Say: Language-Guided Olfactory Representation Learning

Abstract: Images tell us what a scene looks like, but rarely what it would feel like to be there. While recent datasets pair visual scenes with electronic-nose measurements, aligning smell signals with images remains challenging because many olfactory cues arise from contextual environmental factors that are not directly visible in pixels. We introduce SCENT, a multimodal framework that uses language guidance as a semantic bridge between vision and olfaction. Our approach leverages Vision-LLMs (VLMs) to generate scene descriptors capturing objects, environmental context, and plausible ambient smell cues suggested by the visual scene. These descriptors provide semantic guidance for learning olfactory representations. We train a smell encoder that maps electronic-nose signals into a shared embedding space aligned with both visual and textual representations, and introduce a languageguided latent decomposition that separates object-specific odors from contextual environmental contributions. Experiments on the New York Smells dataset demonstrate that SCENT significantly improves crossmodal retrieval compared to vision-only baselines, achieving state-of-theart performance on smell-to-image and smell-to-text retrieval tasks. In addition, our framework produces interpretable olfactory representations that enable the disentanglement of complex smell mixtures. Our results reveal the importance of contextual semantic information for grounding olfactory perception in multimodal learning and pave the way for future research in this area.

• The paper presents SCENT, a framework that leverages VLMs for semantic scene augmentation to guide olfactory representation learning.
• It employs dual contrastive heads aligning a shared 512-dimensional latent space with vision and language cues, achieving over 7% improvement in S2IT retrieval over baselines.
• The approach demonstrates robust, context-aware cross-modal retrieval and zero-shot generalization, paving the way for applications in sensory robotics and environmental monitoring.

Language-Guided Olfactory Representation Learning Beyond Visual Observability

Motivation and Background

Olfactory perception is intrinsically multimodal, shaped not only by the molecular composition of airborne chemicals but also by contextual cues in the environment. While visual perception has seen massive advancement through vision-language pretraining, cross-modal alignment between visual imagery and olfactory signals faces fundamental challenges. Images capture only fragments of environmental information, obscuring sources of ambient smells that may not be in the camera’s field of view. Moreover, the complexity of natural olfactory scenes—characterized by mixtures of odorants from objects and environmental backgrounds—further complicates the learning of robust semantic representations.

The paper introduces SCENT (Semantic Context-aware e-Nose Transformer), a framework that leverages vision-LLMs (VLMs) to bridge the semantic gap between what is seen and what is smelled. SCENT’s central hypotheses are that (i) language can provide crucial semantic “bridges” that compensate for the partial observability in vision, and (ii) VLMs encode world knowledge capable of inferring plausible olfactory contexts from images, supplementing direct pixel-level cues.

SCENT Framework and Core Methods

SCENT comprises three tightly coupled stages:

1. VLM-Based Semantic Scene Augmentation: For each input image, a pretrained VLM is prompted to generate structured semantic descriptors that span object identification, environmental context, and inferred likely ambient smells seen in the scene.

Figure 2: Textual scene descriptors via VLM inference; the VLM outputs structured object, context, and inferred smell cues from each image.

2. Multimodal Olfactory Representation Learning: SCENT trains a Transformer-based smell encoder to project e-nose sensor data into a shared 512-dimensional latent space, aligned through symmetric InfoNCE contrastive losses with both CLIP-derived image and text embeddings. Dual modality-specific projection heads enable explicit alignment with the visual and linguistic semantic spaces.

Figure 1: Smell representation learning; a unified olfactory embedding is aligned with both visual and textual features through two contrastive heads.

3. Latent Smell Disentanglement: To address the compositional structure of natural olfactory scenes, SCENT introduces a decomposition stage where the learned embedding is factorized into object-specific and context-specific latent vectors. Each component is supervised by alignment with the corresponding part of the textual descriptor, and their concatenation must reconstruct the original sensor signal.

This approach exploits the strong priors and contextual extrapolation capabilities of VLMs to inject abstract, non-observable cues (e.g., “traffic exhaust,” “concrete dust,” “air conditioning”) into the olfactory representation. The use of both object-level and environmental context descriptors encourages the model to disentangle overlapping odor sources.

Experimental Results and Analysis

SCENT is evaluated on the New York Smells (NYS) dataset, containing 7,000 image-olfactory signal pairs covering 3,500 object categories and diverse environments. The main findings can be summarized as follows:

• Supervision Granularity: Alignment with image features alone provides modest retrieval accuracy (S2I R@5 ≈ 12.1%). Incremental introduction of context and inferred smell descriptors substantially enhances alignment (S2T R@5: 8.8% → 11.9%; S2IT R@5: 14.3%). The full semantic template that includes object, context, and VLM-inferred ambient smells yields the highest gains, demonstrating that language guidance encodes information beyond what is visually observable.
• Retrieval Performance: SCENT surpasses the NYS vision-only baseline across all retrieval tasks, including smell-to-image, smell-to-text, and joint cross-modal (S2IT). In S2IT, SCENT improves R@5 by over 7% compared to the “Image Bridge” baseline that chains vision and text via CLIP.
• Qualitative and Discriminative Behavior: Detailed visualizations show that SCENT's cross-modal retrieval is robust to fine-grained visual variability and can correctly retrieve target objects despite large occlusions or drastic pose differences, reflecting true context-aware semantic alignment.

Figure 3: Qualitative results for S2IT retrieval; SCENT retrieves semantically correct images even when visual variations are significant.

• VLM Selection: Retrieval quality scales with the VLM’s general reasoning capability (as assessed by MMLU), not merely model size. Qwen3-VL-30B outperforms both larger and smaller alternatives, indicating that multimodal olfactory inference benefits from strong visual-linguistic reasoning.
• Annotation Verification: Held-out secondary views (unseen during training) were used to validate whether VLM-inferred smells reflect genuine, contextually plausible cues versus generic hallucinations.

Figure 6: Held-out view validation confirms that inferred descriptors often correspond to contextually real but visually occluded olfactory sources.

• Latent Disentanglement and Generalization: Decoded synthesis via latent recombination supports zero-shot generalization to new object-context mixtures, outperforming simple raw signal recombination, indicating true compositionality in the learned embedding.
• Data Distribution and Scene Types: Category distributions reveal successful modeling of both dominant and long-tail environmental classes.

Figure 4: Category statistics across seven environment classes, showcasing SCENT’s coverage of diverse real-world olfactory scenes.

Implications and Outlook

The findings demonstrate that cross-modal olfactory representation learning is fundamentally enriched by linguistic semantic scaffolding. VLM-based descriptors capture surreptitious environmental factors, enabling alignment of sensor readings with latent semantics that are impossible or unreliable to resolve from vision alone. SCENT’s language-guided approach facilitates:

• Enhanced retrieval and discrimination in joint vision-olfaction tasks,
• More interpretable and compositional olfactory embeddings supporting generalization and zero-shot synthesis,
• A pathway for grounding “invisible” environmental phenomena in machine perception via data-driven world knowledge.

Practical applications include improved sensory robotics, environmental monitoring in contexts with incomplete visibility, and machine perception of spaces that require more than pixel-level analysis (e.g., hazardous gas detection, food safety, context-aware AR/VR immersion). Theoretically, the work suggests fruitful research directions in exploiting language as an intermediary for grounding, learning disentangled multimodal representations, and leveraging VLMs for other challenging, non-visual modalities.

Outstanding challenges include the robustness limitations imposed by e-nose hardware drift, data scarcity compared to web-scale visual corpora, and further reducing noisy signal contributions from speculative language annotations. Future advances could explore synthetic augmentation, more sophisticated sensor modeling, and closed-loop evaluation where machines autonomously annotate and validate olfactory contexts in-the-wild.

Conclusion

SCENT establishes that language-guided, context-aware olfactory representation learning dramatically surpasses traditional vision-only or text-naive alignment, especially in environments where crucial sensory information is decoupled from direct visual perception. By leveraging VLMs as semantic annotators and integrators, SCENT forges a new paradigm for machine olfaction—one where contextual and latent environmental knowledge, spoken in the language of VLMs, is harnessed to bridge the gap between what images can show and what real-world scenes actually evoke.