Sensecape: Urban & Digital Multimodal Landscape

• Sensecape is a multifaceted construct that models and analyzes multimodal urban and digital landscapes by integrating sensor data, subjective narratives, and environmental metrics.
• It employs layered data architectures and techniques such as clustering, spatial smoothing, and semantic zoom to facilitate real-time sensemaking and visualization.
• The framework supports interactive mapping via urban well-being analysis, soundscape captioning, and digital canvases, bridging physical sensing with subjective experience.

Sensecape refers to a multifaceted construct for representing, analyzing, and interacting with the multimodal and multi-level “landscape” of sensed, subjective, and informational experiences in both urban and digital domains. It is employed in three interrelated but technically distinct contexts: (1) as an extension of the “Third Infoscape” paradigm in urban informatics, (2) as an interactive software system for multilevel exploration and sensemaking with LLMs, and (3) as a technical and affective mapping of environmental data streams such as soundscapes and physiological sensor data. Across these domains, Sensecape is unified by its aggregation of distributed sensing (machines, people, sensors), multilevel data fusion, and abstractions that transcend conventional physical-symbolic dichotomies.

1. Evolution and Conceptual Foundations

Sensecape is grounded in the epistemological transformation from the First Infoscape (pre-industrial, material and symbolic urban information) to the Second Infoscape (industrial, infrastructure-mediated, transaction-based data) and ultimately to the Third Infoscape, which entails the “polyphonic sedimentation” of innumerable real-time micro-histories, emotions, sensor traces, and narratives generated by city dwellers and automated systems alike (Iaconesi et al., 2015).

Whereas traditional sensorial landscapes are predicated on fixed physical or representational strata (e.g., Cullen’s serial vision, Lynch’s legibility), the Third Infoscape redefines the urban Sensecape as a recombinant, real-time informational “body,” with every urban locus equally a node for data, affective signals, historical anecdotes, and cultural expressions. This is possible through technological enablers such as pervasive wireless sensing, smart dust, and participatory human reporting (Iaconesi et al., 2015).

2. Methodological Approaches and System Architectures

Technical instantiations of Sensecape span both physical space and digital information workflows, unified by layered data architectures and multilevel abstraction:

• Urban Well-being Mapping: Sensecape systems for urban analytics collect environmental (noise, light, barometric), physiological (HR, GSR), self-reported affect, and locational data at high temporal and spatial granularity. Preprocessing involves normalization, feature extraction (e.g., HRV, GSR peak count), clustering (DBSCAN, k-means), and spatial analytic frameworks such as Voronoi tessellation and kernel density estimation. Aggregation of well-being scores is achieved via weighted composite formulas,

$W_i = \sum_{m=1}^M \alpha_m\,\tilde{x}_{i,m},$

enabling spatial smoothing and choropleth or heatmap visualization (Johnson et al., 2020).

• Tri-modal Soundscape Mapping: Sensecape can be powered by systems such as GeoCLAP—jointly embedding overhead imagery (ViT), audio (HTSAT), and text (RoBERTa) into a shared 512-D space via symmetric InfoNCE objectives. Zero-shot retrieval is then performed by encoding queries (audio or text), scoring similarity with precomputed region embeddings, and constructing heatmaps over space, offering a scalable basis for global interactive soundscape mapping (Khanal et al., 2023).
• Automated Soundscape Captioning: The SoundSCaper pipeline extends the Sensecape paradigm into affective audio: a multi-scale CNN+graph network (SoundAQnet) extracts acoustic scene, events, and affective qualities (e.g., pleasantness, eventfulness) from audio, feeding these to an LLM for semantic description via chain-of-thought prompting. Evaluation shows human-comparable captions, anchoring the link between objective sensing and subjective experience (Hou et al., 2024).
• Multilevel Sensemaking Workflows: In the digital knowledge domain, Sensecape (as described in (Suh et al., 2023)) adopts a 2-layer hierarchical graph, organizing canvases (each a directed graph of information) recursively. A semantic-zoom feature, enabled by LLMs, allows on-demand abstraction or concretization—nodes are re-prompted at different granularity (keywords, summary, lines, full text). The system maintains both canvas and hierarchy views, supporting seamless transitions between foraging (explore) and sensemaking (structure/synthesize).

3. Key Mechanisms: Polyphonic Sedimentation, Recombinant Inventories, Telepathic Migration

• Polyphonic Sedimentation: Sensecape embodies continual layering of diverse data: sensor logs, stories, physiological streams, emotional annotations. Analogous to geological strata, this process enables composite readings of space or information environments.
• Recombinant Inventories: Any subset of these layers can be filtered, recombined, and rendered via bespoke overlays or analytic functions—facilitating dynamic construction of new services/maps (e.g., edible-plant maps, mood overlays).
• Telepathic Migration of Dust: Information flows across social, technical, and physical boundaries (devices, clouds, platforms) instantaneously, requiring mechanisms for curation and pattern recognition, as well as open infrastructure for remix (Iaconesi et al., 2015).

4. Visualization, Interaction, and Abstraction Techniques

Visualization and interaction in Sensecape systems are characterized by non-linear, multi-resolution, and multimodal representation:

• Geospatial Heatmaps/Choropleths: Interactive overlays encode aggregated well-being, mood, or sound event probabilities onto urban canvases. Spatial smoothing and clustering are used to project seamless data surfaces (Johnson et al., 2020, Khanal et al., 2023).
• Canvas and Hierarchy Views (Digital Domains): Infinite canvas models, with expand bars for one-click LLM-powered generative actions (explore questions, generate subtopics, explain), and a synchronized pseudo-3D tree for managing multilayered abstraction and semantic zoom (Suh et al., 2023).
• Soundscape Captioning Interfaces: Automated pipelines combine graphical acoustic feature analysis with LLM-generated free-text descriptions, evaluated via expert ratings on precision, recall, fluency, and conciseness (Hou et al., 2024).

5. Quantitative Evaluation, User Studies, and System Performance

• Urban Well-being Systems: Feature extraction and visualization approaches have enabled identification of emotional characteristics and stress patterns in small-scale urban deployments, though limitations include post-hoc analysis and non-generalizability beyond pilot sites (Johnson et al., 2020).
• Soundscape Mapping: On the SoundingEarth dataset, GeoCLAP achieves image→sound R@100 of 0.450 (median rank 143), a >55% improvement over prior approaches (Khanal et al., 2023).
• Soundscape Captioning: SoundAQnet + LLM captions are statistically indistinguishable in quality from human experts on in-domain tasks (THumBS 3.22 vs. 3.43, $p=0.128$), though human captions can exhibit higher recall for rare events (Hou et al., 2024).
• Sensemaking Workflows: Controlled studies show that Sensecape increases the number of unique domain concepts discovered ($M=68.3$ vs. $M=22.8$), hierarchical depth ($M=4.3$ vs. $M=2.6$), and revisitation of previous information ($M=12.8$ vs. $M=0.7$) compared to baseline interfaces (Suh et al., 2023).

6. Design Principles, Recommendations, and Implications

Design recommendations distilled from urban and digital Sensecape research include (Iaconesi et al., 2015, Johnson et al., 2020, Suh et al., 2023):

• Open Data Layers: Require open APIs and granular consent for all municipal and citizen-generated data streams.
• Participatory Curation: Provide interfaces for users to build and share overlayed data filters, demand algorithmic transparency for data weighting.
• Ambient Multimodal Interfaces: Employ AR/VR, auditory and haptic feedback, and non-intrusive displays to render informational and affective strata.
• Polyphonic Placemaking: Architecturally embed support for layering and contestation of competing community narratives or moods in physical and digital spaces.
• Commons Licensing: Apply “Ubiquitous Commons” or adapted Creative Commons mechanisms to facilitate remix, responsible governance, and privacy.

Implications span urban planners (treating data streams as infrastructure), architects (embedding “data conduits” into fabric), technologists (crafting seamless, attention-aware telepathic interfaces), and public health analysts (real-time stress/mood mapping).

7. Limitations and Future Directions

• Urban deployments: Current implementations lack city-scale real-time streaming; present pipelines are predominantly post-hoc and have limited sample sizes. Advancing toward edge-computing, live analytic “nudges,” and real-time feedback loops is suggested (Johnson et al., 2020).
• Soundscape/affective mapping: Event vocabularies are bounded (fixed taxonomy), spatial context is limited, and LLM prompting is generic; future work is anticipated in vocabulary expansion, spatial cue encoding, and fine-tuned prompt architectures (Hou et al., 2024).
• Collaborative and adaptive interfaces: Needed are longitudinal studies for Sensecape’s abstraction learning-curve, support for collaborative multiuser construction, and integration with context-aware recommendation or retrieval modules (Suh et al., 2023).

A plausible implication is that, as sensing capacity and participatory data stewardship mature, Sensecape frameworks will facilitate live, remixable, multisensory urban and knowledge environments—rendering the division between virtuality and physicality obsolete, and enabling both cities and digital domains to function as truly polyphonic, living landscapes of collective intelligence (Iaconesi et al., 2015).