New York Smells: A Large Multimodal Dataset for Olfaction

Abstract: While olfaction is central to how animals perceive the world, this rich chemical sensory modality remains largely inaccessible to machines. One key bottleneck is the lack of diverse, multimodal olfactory training data collected in natural settings. We present New York Smells, a large dataset of paired image and olfactory signals captured ``in the wild.'' Our dataset contains 7,000 smell-image pairs from 3,500 distinct objects across indoor and outdoor environments, with approximately 70$\times$ more objects than existing olfactory datasets. Our benchmark has three tasks: cross-modal smell-to-image retrieval, recognizing scenes, objects, and materials from smell alone, and fine-grained discrimination between grass species. Through experiments on our dataset, we find that visual data enables cross-modal olfactory representation learning, and that our learned olfactory representations outperform widely-used hand-crafted features.

• The paper introduces a large-scale olfaction dataset pairing e-nose time-series with visual data from urban NYC, achieving a 70-fold increase in diversity over lab settings.
• It employs synchronized sensor capture and deep contrastive learning to enable effective cross-modal retrieval, scene recognition, and fine-grained odor discrimination.
• Quantitative results demonstrate that models using raw sensor inputs significantly outperform legacy methods, validating advanced multimodal olfactory applications.

New York Smells: A Large Multimodal Dataset for Olfaction

Motivation and Dataset Construction

The paper introduces "New York Smells," a comprehensive multimodal dataset for olfaction centered on pairing electronic nose time-series signals with synchronized visual data captured in unconstrained, real-world urban environments across New York City. Existing machine olfaction datasets are typically constrained to lab settings and lack broad coverage and multimodal supervision, limiting their generalization and utility for modern ML techniques. This new resource comprises over 7,000 olfactory-visual sample pairs from approximately 3,500 distinct objects, representing a 70-fold increase in diversity and scale relative to contemporary lab-based datasets (e.g., SMELLNET (Feng et al., 30 May 2025)). Data acquisition leverages a mobile rig combining the Cyranose 320 e-nose with synchronized high-fidelity RGB and RGB-D cameras, temperature/humidity sensors, and ambient VOC analyzers.

Paired time-synchronized data is collected both during a baseline phase (ambient odor) and a sample phase (target object odor), yielding a $T\times32$ raw sensor matrix for each instance (Figure 1). Automatic object and material annotation are performed using visual LLMs and Matador taxonomy, while scene labels are assigned manually to ensure precise curation. This design achieves robust alignment across multiple sensing modalities and enables supervised and self-supervised multimodal learning protocols.

Figure 2: Aggregate distribution of objects and material categories across the dataset, supporting benchmark construction for olfaction models.

Figure 3: Mobile capture rig with e-nose and high-resolution camera deployed across diverse indoor and outdoor settings for paired data acquisition.

Multimodal Representation Learning Methods

To enable general-purpose machine olfaction, the authors formalize self-supervised and supervised learning benchmarks spanning three core tasks:

1. Cross-modal smell-to-image retrieval: Models learn joint embedding spaces for olfactory sensor signals and images via contrastive learning, facilitating retrieval of visual samples based solely on olfactory queries.
2. Recognition from smell alone: Using linear probes or end-to-end classifiers, models predict scene, object, and material categories from olfactory signals.
3. Fine-grained discrimination: Models are required to recognize subtle odor differences between instances of similar objects (e.g., grass species), a challenging classification benchmark relevant to real-world deployment.

Contrastive learning (COIP) is implemented using paired samples with a dual encoder architecture (CNN/Transformer for olfaction, CNN for image), minimizing a contrastive loss over the joint embedding space (Figure 4). The approach follows contemporary multimodal representation paradigms and leverages the rich cross-modal supervision present in the data.

Figure 1: Temporal response of the 32-dimensional e-nose signal during baseline and sample phases for an exemplar flower.

Figure 4: Schematic of contrastive multimodal olfaction-image training pipeline, enabling alignment of perceptual features across sight and smell.

Quantitative and Qualitative Results

Contrasting performance across input modalities and model architectures, the study demonstrates robust advantages for models trained directly on the raw olfactory sensor matrix versus traditional hand-engineered "smellprint" features. In cross-modal retrieval, raw-signal Transformers and CNNs reach recall@5 of 12.9–17.3%, recall@20 of 43.1%, and median ranks orders-of-magnitude better than chance or smellprint+MLP baselines.

Qualitative cross-modal retrieval analyses reveal strong semantic grouping in predicted matches, where olfactory queries for specific objects (e.g., books, moss, sticks) retrieve visually and materially congruent samples (Figure 5). This substantiates that deep models exploit the rich information in raw sensor time series to form transferable olfactory representations.

Figure 5: Cross-modal retrieval: Top-5 visual matches for diverse odor queries, illustrating successful semantic association between olfactory and visual inputs.

Scene, object, and material recognition from smell alone achieve substantial lift above chance and above classical smellprint-based MLPs, with end-to-end CNNs trained on raw data reaching 99.5% scene classification, and object/material accuracies of up to 19.8% and 14%, respectively—contradicting previous assumptions about the limited discriminative power of raw e-nose signals.

Figure 6: Representative top-3 predictions in scene, object, and material recognition from smell encoder, visualized for accuracy assessment.

Fine-grained discrimination (grass species) is also markedly better when training and probing on raw data, reaching 92.9% accuracy via SSL+linear probe, versus 52.4% for chance/random weights and significantly exceeding all smellprint-based alternatives. This provides compelling evidence for the utility of cross-modal supervision in encoding subtle, real-world olfactory distinctions.

Implications, Limitations, and Future Directions

The results highlight that representation learning frameworks informed by vision can drive high-performing models in olfaction, overcoming the inherent sparsity and noisiness of low-cost e-nose sensors. By demonstrating that cross-modal alignment is feasible and that deep architectures can robustly outperform legacy feature engineering approaches, this dataset opens pathways to new applications—real-time olfactory scene understanding, robot navigation, hazard detection, and multimodal sensory fusion for embodied intelligence. Potential future prospects include scaling multimodal fusion to temporal video, 3D environmental modeling, synesthetic multimodal retrieval, and few-shot rapid generalization in open-world and continuously evolving urban environments.

Limitations persist regarding sensor hardware sensitivity, VOC drift, annotation noise, and generalization to novel urban chemistries. The field will benefit from expanding benchmarks with richer chemical metadata, active sensing protocols, and from integration with advanced molecular datasets for psychophysical modeling. Further theoretical work may interrogate the dimensionality and structure of learned olfactory embeddings, particularly for computational neurobiology and artificial sensory cognition.

Conclusion

This work establishes "New York Smells" as the largest, most diverse real-world multimodal olfaction dataset, pairing thousands of naturalistic e-nose recordings with visual signals across the breadth of urban objects, scenes, and materials. The benchmarks and analyses provide strong evidence that end-to-end models leveraging raw sensor input and visual cross-modal supervision significantly surpass classical features for retrieval, recognition, and fine-grained olfactory discrimination. This foundation links computer vision and computational olfaction, catalyzing research in multimodal machine perception for both academic and industrial AI contexts.