New Paper: Connecting senses: The cross-modal associations between smell and vision in understanding urban environments

In a previous post we wrote about how one can mine social media to uncover smells and how they shapes peoples perceptions of urban spaces. Building off this work we (Qingqing Chen, Ate Poorthuis and myself) have a new paper entitled "Connecting senses: The cross-modal associations between smell and vision in understanding urban environments" published in Geographical Analysis. 

In this paper we build upon this but at the same time move away from social media to explore the relationship between smell and vision. We do this utilizing street view imagery from New York City, in this case we are using Mapillary. A subset of these images were labeled with pre-defined smell categories (e.g.,  ‘Nature’, ‘Food’, ‘Transportation & Fuel’) in order to develop a deep learning smell classifier capable of classifying perceived smells from the images. We then use  advanced image processing techniques (e.g., ResNet50, VGG16, Inception-V3, MobileNet and EfficientNet.) to extract visual cues from the street view imagery to predict smells. 

If this sounds of interest, below we provide an abstract to the paper along with some of the figures which show our workflow and accompanying results. At the bottom of the post you can see the full referece to the paper, while at https://figshare.com/s/94cdec3b14c206e6d225 we provide our code to allow others to replicate or extend this to other areas. 

Abstract:   

Smell is a crucial yet understudied sensory dimension in urban environments, bridging tangible elements (e.g., exhaust, flowers) with intangible impacts on emotions, social interactions and well-being. While geographical and urban research increasingly acknowledges multisensory experiences, much of geospatial analysis still emphasized the visual dimension. This research advances spatial thinking by examining cross-modal associations between smell and vision in urban environments. Specifically, we utilize advanced image processing techniques to extract visual cues from street view imagery (i.e., Mapillary) and apply causal analysis to examine their effects on smell expectations recorded from participants. The results show that visual cues can predict smells in straightforward urban settings (e.g., parks or less densely populated areas). However, in complex urban environments, the predictive power of visual cues diminishes as diverse and overlapping scents obscure specific smells, even in visually distinct areas. These findings underscore the importance of a multisensory approach in urban analytics, enhancing our understanding of the interplay between sensory experiences and informing urban design strategies that integrate multiple senses to create engaging and inclusive environments. This is especially important for individuals with sensory impairments, such as anosmia or visual impairments, who rely on other senses to compensate for their perception of urban environments. 

Keywords: Smell and Vision; Cross-modal Associations; Multisensory Experiences; Image Processing; Street View Imagery (SVI)

An overview of research workflow.

The visual feature extraction framework based on key patterns identified from questionnaires.

Spatial distribution of images with different perceived smells and example participant notes identifying potential smell sources responsible for perceiving the dominant smells.

Identified important features for smell categories that are relatively indirect to be inferred from visual cues. Left: Identified important features; Right: Examples of feature  contribution in individual images.

Full Referece: 

Chen, Q. Poorthuis, A. and Crooks A.T. (2026), Connecting senses: The cross-modal associations between smell and vision in understanding urban environments, Geographical Analysis, 58 (3): e70046.. Available at https://doi.org/10.1111/gean.70046. (pdf)