Video-to-Scent Pipeline

• The video-to-scent planning pipeline is a computational framework that decouples visual semantic extraction from olfactory inference, ensuring modularity and interpretability.
• It uses a two-stage architecture with Gemini 3 Pro for detailed visual analysis and GPT-5.2 for mapping visual cues to structured scent plans.
• Experimental evaluations show improved immersive experiences and coherence compared to traditional over-inclusive and naïve mapping approaches.

A video-to-scent planning pipeline is a computational framework designed to generate structured, temporally aligned olfactory plans from unconstrained video content. Its principal purpose is to facilitate scalable, interpretable, and perceptually relevant olfactory augmentation in multimedia environments, overcoming the challenges associated with manual authoring, synchronization, and event-to-odor mapping. The canonical pipeline decouples vision-based semantic extraction from semantic-to-olfactory inference, thereby enabling modular reasoning, prioritization of salient cues, and alignment of scent transitions with visually significant actions (Wang et al., 27 Jan 2026).

1. Motivation and Challenges

Synchronized olfactory cues in interactive and multimedia experiences offer potential for enhanced immersion. However, two primary obstacles have historically limited their deployment. First, scent hardware exhibits complex onset, offset, residuum, and wash-out dynamics, complicating the alignment of odors with rapidly changing video content. Second, traditional design paradigms rely on manually scripted event-to-odor associations, resulting in poor scalability as most video scenes present many objects and actions, but only a few contribute materially to the perceivable "smellscape." Curated hand-scripting approaches fail to generalize to user-generated or unconstrained video sources (Wang et al., 27 Jan 2026).

2. Two-Stage Pipeline Architecture

The pipeline consists of two sequential, modular stages:

1. Visual Semantic Extraction: Utilizes a vision-LLM (VLM) to generate a temporally indexed "visual timeline" of objects, state changes, proximity, and intensity solely from video frames, without olfactory bias.
2. Semantic-to-Olfactory Inference: Employs a LLM to map the visual timeline to a concise, structured scent plan according to a fixed odor schema. This plan specifies which odor sources to present, intensity ramps, and temporal alignment with actions.

A central premise is strict separation between visual information extraction and olfactory reasoning, resulting in improved interpretability, modularity, and opportunities for downstream hardware integration.

3. Stage 1: Visual Semantic Extraction

Model Architecture and Preprocessing

The system employs Gemini 3 Pro—a dual-encoder vision-LLM. The visual encoder follows a ViT-based architecture pretrained on large-scale image–text pairs, while the text encoder leverages a transformer network trained via CLIP-style contrastive objectives for paired alignment. Video adaptation is achieved by uniformly sampling frames (e.g., one every 0.5 seconds), resizing each to $224 \times 224$, and normalizing by ImageNet statistics.

For each frame $f_t$, the system computes a feature vector $x_v(t) \in \mathbb{R}^d$. Candidate object tokens $\{o_i\}$ are detected through a grounded captioning head, with temporal consolidation performed to merge semantically similar tokens across adjacent frames.

Semantic Token Representation

Semantic mapping is defined as $s(t) = f_v(x_v(t))$, producing structured records with the following fields:

• object_name
• visual_state (e.g., "husk is actively pried open")
• state_change (bool)
• proximity (near/far, material_part, exposure_mode)
• intensity (categorical_level, numeric_level)

Temporal events $s_1,\ldots,s_N$ are recorded as an ordered "visual_timeline," each annotated with $[\mathrm{start}_s, \mathrm{end}_s]$ intervals and output as a JSON-style list.

4. Stage 2: Semantic-to-Olfactory Inference

Odor Schema and LLM Prompting

Semantic-to-olfactory inference leverages GPT-5.2 as the backbone LLM. The prompt template $P$ includes (i) a fixed odor schema $\Theta$, a closed set (e.g., "fruity," "sulfuric," "earthy," etc.), (ii) the visual_timeline from stage 1, and (iii) instructions to select at most $K$ dominant sources and specify intensity ramps.

Prompt example:

Given the following visual timeline events and the odor schema, generate a scent plan as a list of olfactory_events. For each event, assign an odor_label from the schema, a start and end time, and a relative intensity in [0,1]. Prioritize olfactorily salient sources and align scent changes with visual state_change=true intervals.

Mathematical Formulation

The LLM computes $P(y | s_1\ldots s_N)$ for each candidate $y \in \Theta$, outputting

$$y_j^* = \arg\max_{y \in \Theta} P_\text{token}(y \mid \text{context})$$

across visual timeline segments, and assembles a structured output

$$Y^* = \{(e_k, \theta_k, [t^\text{k}_\text{start}, t^\text{k}_\text{end}], \alpha_k)\}$$

to maximize $$P(Y | S) \approx \text{greedy\_decode}(\text{Prompt}[S])$$, subject to:

• $\sum_k \alpha_k \leq 1.0$ (soft energy constraint),
• Events align with $s_i$ where $\text{state_change} = \text{true}$.

5. Runtime Operation and Decision Logic

End-to-End Workflow

At application time, the system processes a video $V$ as follows:

1. Ingest video, sample frames, extract semantic events $S = \{s_i\}$.
2. Construct prompt $P$ with $S$ and odor schema $\Theta$.
3. Query GPT-5.2 with $P$ for a scent plan.
4. Parse the textual output into a JSON list of olfactory_events.

Post-Processing

To refine the plan:

• Secondary filter: Discard $\theta_j^*$ with normalized $\alpha_j < \tau$ (e.g., $\tau = 0.2$).
• Temporal smoothing: Merge adjacent events with the same $\theta_k$ and overlapping intervals.
• Plausibility check: Remove events where the object's frame coverage $<$ $\sigma$ (e.g., $\sigma=0.1$ area), excluding visually minor objects from olfactory mapping.

6. Experimental Validation and Evaluation

Video Dataset and Baseline Methods

Evaluation utilized 10 video clips (5–10 s) rich in smell-relevant events (e.g., food, flowers, eating). For each, three plans were compared:

• System: The proposed two-stage pipeline.
• Over-inclusive baseline: Maps every semantic token to an odor descriptor (no prioritization).
• Naive baseline: Fixed mapping from visual categories (e.g., "food" → "sweet") with static timing.

Study 1: Semantic Selection (n=14)

Participants ranked the three plans for each clip per olfactory immersion:

System	Over-inclusive	Naive
μ_rank: 1.586	μ_rank: 1.871	μ_rank: 2.543
95% CI: [1.464,1.714]	[1.736,2.007]	[2.378,2.714]
First-place: 54.3%	32.1%	13.6%

Friedman test: $\chi^2(2)=19.36$, $p=6.26 \times 10^{-5}$. Wilcoxon signed-rank (Holm corrected): System $>$ Over ($p=0.0251$), System $>$ Naive ($p=0.000366$).

Study 2: Experiential Plausibility (n=8)

Participants compared System and Over-inclusive plans for three clips, rating Immersion, Coherence, Distraction (reverse coded), and Easy-to-Imagine (7-point scales):

Metric	Mean Diff (System–Baseline)	95% CI	Wilcoxon p (Holm)
Immersion	1.42	[1.04,1.79]	0.0313
Coherence	1.29	[0.62,1.83]	0.0313
Distraction	2.17	[1.71,2.83]	0.0313
Easy-to-Imagine	1.50	[0.92,2.08]	0.0313

Preference rates were $\geq 100\%$ in all categories.

Qualitative Successes and Failure Modes

• Successful cases included prioritizing a dominant odor source (e.g., durian pulp) and aligning scent onset with state_change=true intervals (e.g., cutting, opening).
• Failure cases involved over-detailed secondary materials (e.g., metal surfaces) and excessive descriptor intensity (e.g., "extremely pungent"), sometimes perceived as implausible or discomforting.

7. Limitations, Biases, and Future Research

Key limitations include:

• Absence of physical scent delivery in evaluation; actual user olfactory perception may diverge from planned timelines.
• The fixed odor schema may fail to accommodate culturally specific or rare olfactory categories.
• The pipeline inherits LLM biases (e.g., food-/chemical-centric interpretations), potentially overlooking environmental or emotional odor dimensions.

Future directions include integration with real-time scent delivery hardware to assess psychophysical synchrony, schema expansion via crowdsourced and cross-cultural descriptors, user-interactive controls for adjusting intensity or timing, and development of adaptive models jointly refining visual_timeline extraction with olfactory feedback (Wang et al., 27 Jan 2026).

---

The video-to-scent planning pipeline’s bifurcated architecture confers advantages in modularity, interpretability, and alignment with human olfactory expectations. Decoupling visual semantics from olfactory inference enables perceptual filtering, event prioritization, and effective use of LLM commonsense, outperforming over-inclusive and naive alternatives in both quantitative and qualitative evaluations. This approach provides a foundation for scalable olfactory media systems synchronized with video content.