- The paper introduces a 1,350-clip audiovisual benchmark with 99 scent classes, three odor tracks, and annotator-specific olfactory profiles, alongside the OAR model for profile-conditioned prediction.
- Structured field-wise profile conditioning improves performance over content-only prediction, raising Top-1 accuracy from 25.00% to 29.69%, Top-3 from 54.69% to 62.50%, and MRR from 0.454 to 0.508.
- The method delivers its strongest gains for Background and Emotion Odor and improves VR cinema and advertising evaluations, while revealing limitations involving Foreground Odor, single-annotator labels, and the 99-essence vocabulary.
Automated video-odor matching has conventionally been treated as a content-only recognition task: given visual or auditory cues from a clip, predict which scent should be released. This formulation implicitly assumes that odor labels are observer-independent. The authors of OlfactProfile argue that this assumption is restrictive, because odor judgment is shaped by individual olfactory characteristics—scent sensitivity, tolerance to unpleasant odors, and affective odor preference—which determine whether the same audiovisual input warrants a noticeable, acceptable, or emotionally fitting scent. A blood-like odor may suit a meat-cutting scene yet be inappropriate for a viewer with low aversion tolerance.
The paper formulates profile-conditioned odor prediction: given an audiovisual clip and an olfactory preference profile associated with its annotation, predict odor labels under that observer context. The central empirical claim is deliberately counterintuitive: profile information is not beneficial by default. Under controlled comparisons with matched feature backbones, naive profile-feature concatenation and uniform profile modulation both degrade prediction relative to a content-only baseline; only structured field-wise conditioning improves it. The paper's contribution therefore concerns not merely whether observer context is available, but how it is integrated into multimodal reasoning.
Benchmark construction
To enable this setting, the authors construct what they identify as the first audiovisual benchmark pairing temporally aligned odor annotations with annotator olfactory profiles. The benchmark comprises 1,350 video clips with synchronized audio drawn from public platforms (food preparation, natural environments, public spaces, emotionally expressive scenes), a 99-class scent vocabulary of aromatic essences, and three semantic odor tracks adapted from OdorAgent: Foreground Odor (salient visible sources), Background Odor (ambient environment), and Emotion Odor (scent reinforcing affective tone without a visible source). Sixty qualified volunteers annotated clips after completing a structured questionnaire covering six olfactory dimensions; each annotation is paired with the annotator's own profile rather than a hypothetical consensus label, preserving the link between who assigned a label and which profile shaped that judgment.
Compared against prior resources such as SniffyArt, ODOR, Beyond the Scent, and SmellNet, this benchmark is the first to combine video+audio modality, user profiles, multi-track annotation, and temporal structure in one resource. The authors acknowledge a vocabulary constraint: 99 essences do not cover all possible scents, though they judge the diversity sufficient for their experiments.
Method
OlfactProfile is built around OAR (Olfactory-Aware Routing), a fusion module combining two mechanisms:
- Track-aware audiovisual routing: relevance scores sv,sa are computed from concatenated visual, audio, profile, and track-embedding features, while a separate branch estimates signal reliability rv,ra; normalized products wv∝svrv, wa∝sara weight projected modality features. Speech, when present, is added through a gated auxiliary term.
- Field-wise profile modulation: rather than injecting the profile as a flat vector, distinct fields act through role-aligned paths. Scent sensitivity multiplicatively gates the routed representation across all tracks; tolerance to unpleasant odors contributes a track-selective bias on Foreground and Background tracks; emotional odor tendency contributes an affective bias restricted to the Emotion track; remaining dimensions enter via a shared bias term.
A Scent Skill Library (SSL) complements learned representations with learnable knowledge embeddings retrieved via attention over the profile-aware representation and track embedding, and additionally provides structured textual odor entries for downstream reasoning. The fused representation and retrieved embedding feed three independent classification heads trained with summed track-specific cross-entropy losses.
Controlled comparison: integration strategy determines value
The most important experiment holds the ResNet50 + HuBERT backbone fixed and varies only how profile information enters the model:
| Model |
Top-1 |
Top-3 |
Top-5 |
MRR |
| AV-only |
25.00% |
54.69% |
76.56% |
0.454 |
| AV+NaiveUser |
20.31% |
50.00% |
73.44% |
0.423 |
| MM-CLIP-Style |
25.00% |
48.44% |
60.94% |
0.414 |
| UniformProfile |
20.31% |
50.00% |
64.06% |
0.397 |
| OlfactProfile |
29.69% |
62.50% |
79.69% |
0.508 |
Three negative results are as informative as the positive one. Naive concatenation hurts (−4.69 Top-1 points); substituting a stronger generic cross-modal attention block does not help; and uniform modulation within the same OAR routing framework also underperforms. Only field-wise conditioning yields gains (+4.69 Top-1, +7.81 Top-3 over AV-only). This isolates the source of improvement to structured profile integration rather than profile availability, fusion capacity, or routing architecture per se.
Per-track analysis
Per-track results show gains concentrated where observer-dependent judgment matters most. On Background Odor, OlfactProfile raises Top-1 accuracy from 11.54% (AV-only) to 42.31%, with MRR improving from 0.316 to 0.577; on Emotion Odor, Top-1 rises from 5.00% to 25.00%. These tracks depend on ambient interpretation and affective association rather than directly visible odor sources, consistent with the design rationale. Notably, on Foreground Odor—where content-driven matching is comparatively strong—OlfactProfile achieves only 22.22% Top-1, below both AV-only (33.33%) and MM-CLIP-Style (38.89%), though it attains the best Top-3 (55.56%). This trade-off indicates that structured profile conditioning redistributes capacity toward observer-dependent tracks at some cost on source-grounded ones, a nuance the paper reports but does not deeply analyze.
Component ablations confirm complementarity between OAR and SSL, and dynamic routing outperforms audio-only, visual-only, and equal-weight fusion alternatives.
Comparison with MLLMs and human experts
Against general-purpose multimodal large models evaluated under the validated OdorAgent prompt protocol, OlfactProfile leads in both settings: 29.67% Top-1 / 62.5% Top-3 / 0.508 MRR with profile input versus GPT-4.1's 20.0% / 22.0% / 0.2673, and 25.0% / 50.0% / 0.435 without profile. Removing the profile reduces OlfactProfile's Top-1 by 4.67 points, providing direct evidence of predictive value in observer context. An interesting observation is that the strongest MLLM baseline differs across the two settings, suggesting current MLLMs exploit personalized textual context less reliably than feature-level structured conditioning.
In a small human study on 12 sampled test clips, OlfactProfile reaches 29.41% Top-1 versus 31.10% for five odor experts and 17.24% for lay participants, and attains the highest MRR (0.497 vs. 0.483 expert, 0.356 lay) plus the lowest intensity MAE. Given the small sample, this establishes competitiveness rather than superiority over experts.
Downstream application studies
Two within-subject studies tested transfer without task-specific fine-tuning, using open-source piezo-based mist-releasing devices and comparing against a GPT-4o-derived scent-plan baseline. In a VR cinema study (N=12), OlfactProfile-based plans produced higher Film IEQ scores (M=4.82 vs. $4.56$, p<0.001), with subscale gains in captivation, dissociation, and transportation and unchanged comprehension. In a desktop multimodal advertising scenario, participants rated the model's plans higher on intensity appropriateness (5.67 vs. 4.88) and acceptability (5.91 vs. 5.02), preferring them in 72.4% of pairwise comparisons. These results indicate that profile-conditioned predictions translate into measurable user-perceived improvements in deployed scent-enhanced media.
Limitations and open questions
The paper concedes a significant benchmark limitation: each clip is paired with a single annotator profile-label instance, so profile effects are not fully disentangled from between-clip variation. Extending the benchmark with multi-annotator annotations per clip is identified as necessary to study how odor judgments vary across users for the same scene. Additional constraints follow from the reported results themselves: the Foreground-track regression under profile conditioning is unexplained; the human comparison rests on only 12 clips; and the vocabulary of 99 essences bounds the label space. Whether field-wise conditioning generalizes beyond hand-designed field-to-role alignments—for example, learned or data-driven mappings between profile dimensions and perceptual roles—remains an open question the paper does not address.
Conclusion
OlfactProfile formulates profile-conditioned odor prediction, provides the first audiovisual benchmark pairing odor annotations with annotator olfactory profiles, and demonstrates through controlled experiments that the value of observer context depends entirely on its integration mechanism: naive concatenation and uniform modulation degrade performance, whereas track-aware routing with field-wise profile modulation consistently improves it, particularly on Background and Emotion Odor. Combined with competitive performance against MLLMs and odor experts and verified downstream benefits, the work establishes structured profile conditioning as a concrete design principle for personalized scent-enhanced media, while leaving multi-annotator supervision and generalized profile-field alignment as the principal unresolved issues.