Semantic-to-Olfactory Inference Overview

Updated 1 February 2026

Semantic-to-olfactory inference is a framework that converts abstract semantic cues into predicted olfactory experiences using molecular and multimodal models.
Models like DeepNose and Mol-PECO employ 3D CNNs and GCNs to map molecular features to human odor descriptors, achieving AUROCs up to 0.911.
Integrated approaches leverage vision-language models and LLMs to design temporal scent delivery schemes, synchronizing olfactory responses with multimedia events.

Semantic-to-olfactory inference refers to computational frameworks and models that translate high-level semantic descriptors—such as language-derived concepts, structured labels, or extracted event metadata—into predicted olfactory experiences, molecular features, or actionable scent plans. Such inference spans domains from molecular structure prediction of odor perception to mapping multimodal semantic contexts (e.g., video events) into temporal scent delivery schemes. The principal aim is to bridge the gap between semantic information and the sensory domain of olfaction, facilitating prediction, understanding, and control of human olfactory percepts.

1. Conceptual Foundations

Semantic-to-olfactory inference emerges from the need to operationalize the relationship between abstract, often linguistically-expressed, semantic variables and the complex sensory field of human smell. Olfactory perception is mediated by an ensemble response of odorant receptors (ORs) to molecular stimuli, producing high-dimensional feature representations that are decoded into semantic descriptors such as "garlic," "roasted," or "floral" (Shuvaev et al., 2024, Zhang et al., 2023). The challenge is compounded by the discontinuous perceptual space of odorants, the context-dependence of odor labels, and the mixture effects typical of naturalistic stimuli.

Approaches in this field utilize either a bottom-up mapping (from molecular structures and their physicochemical or spatial properties into semantic labels) or a top-down schema (from semantic events or object descriptors into actionable olfactory syntheses). Techniques involve deep learning architectures (CNNs, GCNs), spectral graph representations, and progressive hybrid pipelines integrating vision-LLMs and LLMs (Wang et al., 27 Jan 2026).

2. Molecular Structure-to-Semantic Odor Prediction

Models such as DeepNose and Mol-PECO represent the state of the art in translating molecular geometry and atomic composition into dense, multi-label semantic predictions of human olfactory experience.

DeepNose Architecture

DeepNose utilizes an SO(3)-equivariant 3D CNN that ingests molecules rasterized in an 18 Å cube, discretized as a 1 Å voxel grid over six atom types, with 640 sampled orientations approximating rotational invariance. The input tensor $X \in \mathbb{R}^{O \times E \times 18 \times 18 \times 18}$ supports orientation-aware feature extraction. Filter responses are weight-shared over orientation channels via a discrete group convolution, ensuring equivariance under sampled rotations. After spatial and orientation pooling, the pipeline culminates in an MLP mapping to 654 semantic descriptors through masked binary cross-entropy loss (Shuvaev et al., 2024).

Mol-PECO Framework

Mol-PECO encodes molecular structure in an $N \times N$ Coulomb matrix $C$ , with entries defined as

$C_{ij} = \begin{cases} 0.5 Z_i^{2.4} & i = j \ \frac{Z_i Z_j}{\|R_i - R_j\|} & i \neq j \end{cases}$

Spectral decomposition of the normalized Laplacian $L^2$ yields eigenpairs that are transformed via a learned positional encoding and fed into a directional GCN. The ultimate molecular embedding $m$ serves as the substrate for a sigmoid-output MLP predicting 118 odor descriptors (Zhang et al., 2023).

Both models demonstrate high-fidelity prediction accuracy, with DeepNose reaching AUROC = 0.835–0.911 for multi-label outputs and Mol-PECO achieving AUROC = 0.813 on 8,503 molecules (Shuvaev et al., 2024, Zhang et al., 2023).

3. Semantic-to-Olfactory Planning in Multimodal Contexts

Semantic-to-olfactory inference extends to multimodal scenarios, notably in the synchronization of scent delivery with temporally structured media content. "Before Smelling the Video" introduces a two-stage pipeline where:

A vision-LLM (Gemini 3 Pro) extracts a time-aligned semantic timeline of events, objects, and attributes from sampled video frames.
A LLM (GPT-5.2), prompted with event metadata, maps this timeline to a fixed odor schema and assigns temporal intervals and intensities for scent delivery (Wang et al., 27 Jan 2026).

Formally, let $V = \{v_t\}_{t=1}^T$ be a video, $S = f_{vis}(V)$ the semantic event timeline, and $O = f_{sem}(S)$ the final scent plan. The mapping $f_{sem}$ employs a text embedding similarity-based heuristic to score candidate odors $N \times N$ 0 for each event $N \times N$ 1:

$N \times N$ 2

Top-scoring descriptors above a threshold $N \times N$ 3 are selected as scent plan elements.

4. Evaluation and Benchmarking

Performance metrics in semantic-to-olfactory inference emphasize multi-label classification accuracy, cross-modal temporal alignment, and perceptual coherence:

DeepNose: AUROC = 0.835–0.911 (654 labels), ensemble averaging over five folds, Pearson $N \times N$ 4 for mixture distance prediction (Shuvaev et al., 2024).
Mol-PECO: AUROC = 0.813 (118 labels), outperforming molecular fingerprint and adjacency-GCN baselines (Zhang et al., 2023).
Video-to-Scent Planning: Mean rank and first-place rates from human subtasks, Friedman’s $N \times N$ 5 and Wilcoxon tests, with system-generated plans preferred in 54.3% of trials for immersion and temporal coherence (Wang et al., 27 Jan 2026).

Model/Method	Main Metric	Result
DeepNose eqCNN (Shuvaev et al., 2024)	AUROC (multi-label)	0.835–0.911
Mol-PECO (Zhang et al., 2023)	AUROC (multi-label)	0.813
Video Scent Planner (Wang et al., 27 Jan 2026)	Mean rank (user study)	1.586 (system)

The models exhibit robust discrimination of stereoisomers and mixture generalization.

5. Interpretability and Attribution

Semantic-to-olfactory models integrate various strategies for interpretability and feature attribution:

DeepNose applies leave-one-atom-out occlusion to quantify atom-level contributions: $N \times N$ 6, where positive values indicate atomic saliency for semantic descriptor $N \times N$ 7 (Shuvaev et al., 2024).
Mol-PECO demonstrates clustering in embedding space for synonyms and attribute classes, with nearest-neighbor retrieval capturing semantic grouping even for structurally divergent molecules (Zhang et al., 2023).
Video Scent Planning provides editable, temporally resolved scent plans with clear linkage between event semantics and odor descriptors (Wang et al., 27 Jan 2026).

6. Generalization: Mixtures and Stereoisomers

Advanced architectures natively accommodate mixture effects and stereoisomer discrimination:

Mixture Inference: DeepNose processes compound mixtures by averaging pure-component features: $N \times N$ 8, yielding feature-space distances correlating with psychophysical measures ( $N \times N$ 9) (Shuvaev et al., 2024).
Stereoisomer Sensitivity: Full 3D input ensures enantiomers map to distinct semantic profiles consistent with human perception, validated across multiple datasets (Shuvaev et al., 2024).

A plausible implication is that the combination of orientation-equivariant geometric processing and atom-level embeddability is critical for capturing human-like olfactory distinctions.

7. Future Directions and Limitations

Semantic-to-olfactory inference continues to evolve, with proposed extensions including:

Development of bidirectional generative frameworks (e.g., variational autoencoders or diffusion models) for generating new molecular structures from semantic constraints (Zhang et al., 2023).
Enrichment of semantic encodings via free-text odor descriptors, multimodal inputs, and pre-training on expansive chemical datasets (Zhang et al., 2023).
Introduction of user-adaptive and closed-loop feedback for scent delivery plans, real-time semantic event streaming, and integration with olfactometer hardware for physical realization (Wang et al., 27 Jan 2026).

Current limitations include sampling granularity in event extraction, absence of online adaptation or hardware latency models, and incomplete bidirectionality in generative mapping. The field is nevertheless advancing toward more interpretable, generalizable, and application-ready semantic-to-olfactory pipelines.