New York Smells: Multimodal Olfaction Dataset

• New York Smells is a multimodal dataset pairing visual observations with electronic-nose readings captured in real NYC environments.
• The dataset comprises 7,000 smell-image pairs spanning diverse indoor and outdoor settings, supporting three benchmark tasks.
• Empirical results demonstrate that learned raw-signal features, especially with Transformer models, significantly outperform traditional smellprint baselines.

New York Smells is a large multimodal dataset for olfaction that pairs visual observations with electronic-nose measurements captured “in the wild” across New York City (Ozguroglu et al., 25 Nov 2025). It was introduced to address a central bottleneck in machine olfaction: the lack of diverse, multimodal olfactory training data collected in natural settings. The dataset contains 7,000 smell-image pairs from 3,500 distinct objects across indoor and outdoor environments, with approximately $70\times$ more objects than existing olfactory datasets, and it supports three benchmark tasks: cross-modal smell-to-image retrieval, recognition of scenes, objects, and materials from smell alone, and fine-grained discrimination between grass species. In the reported experiments, visual data enables cross-modal olfactory representation learning, and learned olfactory representations outperform widely used hand-crafted features.

1. Scope, setting, and dataset composition

New York Smells was collected across forty distinct locations in New York City, spanning 60 recording sessions over two months. Approximately 41% of samples are outdoor, including parks and streets, and 59% are indoor, including libraries, dining halls, gyms, apartments, and offices. The dataset contains 7,000 paired smell-image samples corresponding to 3,500 distinct objects, 49 object categories, 34 material categories, and 8 scene types. The object categories were auto-generated with GPT-4o, and the material categories use the Matador taxonomy with GPT-4o. The corpus also includes 70 hours of video and 196,000 smell timesteps. The split is 50/50 into train and validation, with both samples of each object placed in the same split (Ozguroglu et al., 25 Nov 2025).

These design choices position the dataset as an “in-the-wild” benchmark rather than a laboratory collection. A plausible implication is that the benchmark emphasizes environmental heterogeneity, object diversity, and nuisance variation more strongly than prior controlled olfactory datasets. The stated comparison to prior lab datasets, including Feng et al. 2025, further underscores that the scale increase is not only in sample count but also in the number of distinct objects.

2. Instrumentation and acquisition protocol

The olfactory sensor is a Cyranose 320 electronic nose with 32 nanocomposite gas sensors. It samples resistance at 2 Hz and yields a raw time series $x_S \in \mathbb{R}^{T \times 32}$. The primary visual sensor is an iPhone 12 RGB camera operating at $1920 \times 1080$ pixels and 15 FPS. Supplementary sensing includes an Intel RealSense D405 RGB-D camera at $1280 \times 720$ pixels and 15 FPS, a MiniPID2 VOC meter, and ambient temperature and humidity. A custom 3D-printed rig mounts the e-nose and cameras in alignment.

For each object, the protocol records a baseline purge and then two object “sniff” samples. The baseline purge uses ambient air through a side port for 10 s, yielding 14 timesteps. Two sniff samples of the object are then collected for 10 s each through the main inlet, producing an additional 14 timesteps. The concatenated raw smell signal is therefore $T = 28$ timesteps by 32 channels. Raw olfactory data are used without filtering beyond the sensor’s internal calibration, while visual data undergo standard center-crop and resizing; no heavy augmentation was applied in the reported experiments (Ozguroglu et al., 25 Nov 2025).

The acquisition stack is notable because it aligns smell sensing with image capture and supplementary environmental measurements in a single physical rig. This suggests that the dataset was designed not merely for supervised classification, but for multimodal representation learning under aligned sensing geometry.

3. Signal representations and preprocessing baselines

The reported work evaluates both hand-crafted and learned olfactory representations. The hand-crafted baseline is the widely used “smellprint.” After Savitzky–Golay smoothing of each channel, the baseline response and peak response are defined as

$$R_{0,i} = \frac{1}{|B|}\sum_{j \in B} R_{i,j}, \qquad R_{\max,i} = \max_{j \in S} R_{i,j}.$$

These statistics produce a 32-dimensional vector. By construction, the smellprint summarizes each sensor channel through baseline and maximal response rather than preserving the full temporal waveform.

The learned alternatives operate on the raw $28 \times 32$ olfactory signal. The paper explicitly contrasts this with the smellprint by noting that the smellprint discards second-order temporal and cross-sensor statistics, whereas end-to-end learning can exploit the full $T \times 32$ waveform. This distinction is central to the benchmark’s empirical findings: the comparison is not only between model classes, but between compressed hand-crafted descriptors and raw-signal representation learning.

A common misconception is that olfactory benchmarking is intrinsically limited to static feature engineering. New York Smells directly contests that view by formulating the electronic-nose stream as a structured temporal signal rather than only a per-sensor summary.

4. Benchmark tasks and learning objective

The benchmark defines three tasks, all evaluated on the held-out validation split.

First, cross-modal smell-to-image retrieval asks whether a query smell can retrieve the correct image from a gallery containing one true match and $N-1$ distractors. Two encoders are learned, $f_{\theta_S}: x_S \mapsto z_S$ and $x_S \in \mathbb{R}^{T \times 32}$0, and training uses a symmetric contrastive loss:

$x_S \in \mathbb{R}^{T \times 32}$1

with $x_S \in \mathbb{R}^{T \times 32}$2 defined analogously and optimization objective

$x_S \in \mathbb{R}^{T \times 32}$3

Evaluation uses mean rank, median rank, and Recall@$x_S \in \mathbb{R}^{T \times 32}$4.

Second, scene, object, and material recognition from smell alone probe the smell encoder using pseudo-labels from the visual modality. The three classification problems are 8 scene classes, 49 object classes, and 34 material classes. In these experiments, the smell encoder is frozen and a linear classifier is trained on its penultimate activations.

Third, fine-grained discrimination evaluates whether the learned smell representation can distinguish between two co-located grass species. For this task, an additional 256 samples were collected evenly across the two species, and a 42-sample session was held out for test. A linear probe is again trained on the smell encoder features (Ozguroglu et al., 25 Nov 2025).

The model families reported for olfactory encoding are an MLP on the 32-dimensional smellprint, a 1D-CNN on the raw $x_S \in \mathbb{R}^{T \times 32}$5 signal, and a Transformer on the raw $x_S \in \mathbb{R}^{T \times 32}$6 sequence. The visual encoder is a standard CNN of ResNet style processing static RGB frames. Both modalities use 512-dimensional embeddings during cross-modal training. The work refers to this contrastive olfaction-image pretraining as COIP.

5. Empirical results and performance profile

In retrieval, chance performance is $x_S \in \mathbb{R}^{T \times 32}$7 and $x_S \in \mathbb{R}^{T \times 32}$8. The MLP operating on smellprint reaches $x_S \in \mathbb{R}^{T \times 32}$9. The raw-signal CNN reaches $1920 \times 1080$0, and the raw-signal Transformer reaches $1920 \times 1080$1, which is the best reported result. Mean and median rank improve from approximately 467 under chance to approximately 104 and 28 with the Transformer.

In smell-only recognition, the reported probe accuracies are strongly task dependent. For Raw-CNN+scratch, scene classification reaches 99.5%, material classification 11.9%, and object classification 17.9%. For Raw-CNN+SSL-probe, the corresponding numbers are 95.0%, 12.3%, and 19.8%. For Smellprint-MLP+SSL-probe, the results are 32.5% for scene, 2.0% for material, and 4.96% for object. Random-initialized probes perform far worse; the example given is scene classification at approximately 72% versus 95%, which the paper interprets as evidence that cross-modal pretraining is key.

In fine-grained grass-species discrimination, Raw-SSL+probe reaches 92.9% accuracy, Smellprint-SSL+probe reaches 90.0%, and scratch training on raw signals reaches 52.4%, with chance at 50%. Across all tasks, learned raw-signal features greatly outperform the smellprint baseline. The qualitative examples described in the paper also report semantically coherent retrievals and top-3 smell-based predictions, such as a “flower” smell retrieving other flowers (Ozguroglu et al., 25 Nov 2025).

The paper does not report explicit $1920 \times 1080$2-values, but it reports large absolute gaps, including greater than 30 percentage points in retrieval recall and greater than 60 percentage points in scene classification. Errors are said to occur mainly among visually similar materials or objects with subtle olfactory differences, such as concrete versus asphalt. This suggests that the remaining failure modes are concentrated less in gross scene separation and more in semantically adjacent categories with weak olfactory separability.

6. Conceptual significance, limitations, and relation to urban smell reporting

The principal conclusion of New York Smells is that vision provides an effective supervisory signal for machine olfaction. More specifically, contrastive olfaction-image pretraining yields high-quality smell representations that outperform traditional smellprint features across retrieval, classification, and fine-grained tasks. The paper identifies several open challenges: leveraging temporal and 3D visual context such as video and depth, scaling to continuous ambient variations and moving sensors, and integrating other modalities such as sound and touch for richer environmental understanding. The authors plan to publicly release code and data at http://smell.cs.columbia.edu (Ozguroglu et al., 25 Nov 2025).

New York Smells should be distinguished from urban smell-reporting systems such as “Smell Pittsburgh” (Hsu et al., 2018). Smell Pittsburgh is a community-empowered mobile smell reporting system in which native iOS and Android apps record GPS and timestamps, collect a smell rating and optional text fields, and forward reports to the local health department. It also visualizes public smell reports on a map together with air-quality and wind data, and it trains predictive models for upcoming smell events. Over Nov 2016 to Sep 2018, the system recorded 17,280 smell reports from 3,917 unique Pittsburgh users, and its classification Extra-Trees model achieved precision $1920 \times 1080$3, recall $1920 \times 1080$4, and $1920 \times 1080$5 under time-series cross-validation.

The contrast between the two systems clarifies the scope of New York Smells. Smell Pittsburgh operationalizes odor as human report data embedded in public-health and civic-action workflows, whereas New York Smells operationalizes olfaction as aligned electronic sensing paired with visual context. A common misconception is to treat these as interchangeable. They are not: one is a citizen-sensing and notification system for smell events, and the other is a multimodal machine-learning benchmark for representation learning. A plausible implication is that the two paradigms are complementary. Community reporting captures subjective nuisance and public-health relevance, while paired electronic sensing and imagery capture the structured multimodal signals required for cross-modal olfactory representation learning.