ELSA: Urban Social Activity Benchmark

• ELSA is a benchmark dataset that evaluates open-vocabulary detection of social activities using 937 NYC street images annotated with over 4,300 multi-label bounding boxes.
• It leverages a theory-informed taxonomy from urban sociology to capture individual postures and group interactions in complex, real-world street environments.
• The dataset challenges detection models through issues of semantic consistency, phrasing sensitivity, and confidence calibration in overlapping multi-label scenarios.

ELSA, short for Evaluating Localization of Social Activities in Urban Streets, is a benchmark dataset and evaluation agenda for open-vocabulary detection in street-level imagery. It is designed to localize individual and group social activities in “in-the-wild” urban scenes, with particular emphasis on multi-label annotation, prompt sensitivity, semantic consistency, and confidence reliability. The resource is grounded in urban sociology and urban design, and consists of 937 street-level images of New York City with more than 4,300 multi-labeled bounding boxes spanning 34 atomic labels and 112 unique multi-label patterns (Hosseini et al., 2024).

1. Motivation and problem setting

ELSA was introduced at the intersection of two research pressures. One is substantive and urbanistic: urban scholars have long treated lively streets as a function of people’s interactions, but traditional observation-based studies are costly, error-prone, and difficult to scale to entire cities. The other is methodological: open-vocabulary detection models such as Grounding DINO, MDETR, OWL-ViT, and Detic promise recognition beyond fixed class inventories, yet in practice exhibit instability when labels are phrased differently, when multiple labels overlap, or when confidence estimates must reflect complex context (Hosseini et al., 2024).

The benchmark targets three shortcomings identified for current OVD systems. First, semantic consistency is fragile: the same object or activity may receive different labels under slight prompt changes. Second, phrasing sensitivity is substantial: formulations such as “group talking” and “talking group” may yield materially different detections. Third, confidence miscalibration is a recurring problem, especially under out-of-distribution conditions and in overlapping multi-label cases, where models may be overconfident despite limited contextual understanding (Hosseini et al., 2024).

Street imagery is a particularly demanding domain for this problem. It contains large numbers of people in heterogeneous public settings, including distant pedestrians, crowds, printed images, and mannequins. It also includes activities ranging from solitary states such as sitting or walking to collective interactions such as dining, shopping, and talking. ELSA is therefore positioned as a benchmark for fine-grained counting and localization of social activity patterns relevant to street design, land use, and social vibrancy (Hosseini et al., 2024).

2. Conceptual and taxonomic foundations

Rather than defining its ontology ad hoc, ELSA derives its label space from established urban sociology and public-space research. The paper explicitly situates the benchmark in relation to Webber, Whyte, Jacobs, Gehl, and Mehta, using those traditions to justify why street life should be represented through both individual disposition and collective interaction (Hosseini et al., 2024).

Gehl’s distinction between necessary and optional activities shaped the benchmark’s State and Action categories. Whyte’s account of “public life” and Jacobs’s “sidewalk ballet” motivated a representation that captures both the behavior of individual pedestrians and the dynamics of groups. The public-space observational method associated with Gehl also informed the use of multi-label annotation, reflecting the premise that a single bounding box may encode concurrent behaviors, such as a person walking while talking on a phone (Hosseini et al., 2024).

The resulting annotation space comprises 34 atomic labels organized into Condition, State, Action, and Other. Condition includes mutually exclusive descriptors such as “alone,” “two-person,” and “group.” State and Action are explicitly designed to co-occur. The benchmark therefore does not treat urban activity recognition as a single-label action-classification task; it treats it as a structured description problem in which posture, activity, and social grouping can coexist within one localized instance (Hosseini et al., 2024).

3. Dataset construction and annotation workflow

The dataset contains 937 street-level images of New York City, sampled from Microsoft Bing Side-view (time-stamped) and Google Street View. Annotation begins with spatial preprocessing rather than direct manual boxing: CitySurfaces semantic segmentation is used to isolate sidewalks and crop images to pedestrian areas. Candidate person boxes are then generated with YOLOv8, after which four trained annotators manually refine box locations and assign multi-label sets (Hosseini et al., 2024).

Quality control is central to the construction pipeline. An urban-planning specialist audits every image for semantic correctness. The annotation process also enforces 9 logical constraints intended to prevent internally inconsistent label combinations. The detailed account gives representative examples: a box labeled Alone may only have one State; “alone” and “group” cannot co-occur; and certain Actions imply a posture label. Any box violating a rule is re-annotated until it satisfies those constraints (Hosseini et al., 2024).

This workflow is significant because ELSA is not merely a collection of boxes with free-form tags. It is a manually corrected, theory-informed, and logically constrained benchmark in which annotation validity is treated as part of the research contribution. A plausible implication is that the dataset is intended not only to test raw detection accuracy, but also to test whether open-vocabulary systems can sustain structured semantic commitments across overlapping labels and socially meaningful categories (Hosseini et al., 2024).

4. Statistical profile and representational structure

ELSA includes more than 4,300 multi-labeled bounding boxes and 112 unique multi-label patterns. The Condition distribution is explicitly described as comprising “alone,” “two-person,” and “group” categories, with over 20% of boxes depicting groups of three or more. Group size ranges from solitary individuals to clusters of eight or more (Hosseini et al., 2024).

The benchmark’s multi-label structure is important because it differs from standard action-recognition settings built around tightly controlled videos or single dominant labels. ELSA uses still images captured in uncontrolled public environments, and its labels permit combinations across Condition, State, and Action. Examples in the detailed account include State labels such as sitting, standing, walking, running co-occurring with Action labels such as talking, dining, shopping, and pushing strollers (Hosseini et al., 2024).

This representational choice formalizes a view of public-space observation in which social activity is layered rather than atomic. A single detection may encode posture, social configuration, and purpose simultaneously. That design makes the benchmark unusually sensitive to semantic overlap, prompt composition, and confidence calibration, especially when multiple plausible textual formulations can refer to nearly the same scene content (Hosseini et al., 2024).

5. Evaluation agenda, manuscript scope, and stated limitations

The paper advocates benchmarking OVD models in closed- and open-vocabulary settings with multi-label prompts of increasing complexity at CS, CSA, and CSAO levels. It also names models such as Grounding DINO, MDETR, OWL-ViT, and Detic as intended targets for evaluation. However, the detailed account of the manuscript states that the current document does not define custom metrics, does not report formal evaluation results, and does not provide confidence-calibration curves, precision, recall, or mAP results (Hosseini et al., 2024).

The same qualification applies to two algorithmic elements announced in the abstract. The abstract states that the work introduces a confidence score computation method called NLSE and a Dynamic Box Aggregation (DBA) algorithm. The detailed account, however, states that the manuscript does not propose an explicit NLSE formula, does not describe a DBA procedure or pseudocode, and does not include experiments or quantitative benchmarks. In its current published form, the contribution is therefore the dataset and benchmarking problem formulation rather than a completed evaluation suite or a new detection algorithm (Hosseini et al., 2024).

The discussion section frames several anticipated failure modes for existing OVD systems. These include prompt variation, overlapping labels, distant or occluded pedestrians, and negative distractors such as billboard cutouts and mannequins. The authors also identify future directions: extending the resource with temporal sequences, designing loss functions or prompt-aggregation schemes to enforce consistency across related queries, and exploring calibration techniques such as temperature scaling and Bayesian dropout for out-of-distribution sidewalk scenes (Hosseini et al., 2024).

6. Disambiguation and acronym usage

In arXiv usage, ELSA is not unique to urban-activity localization. The acronym has been used for unrelated methods and systems in multiple fields, which can create bibliographic ambiguity. Examples include “Enhanced Location Spoofing Detection using Audibility” in Internet-of-Things security (Koh et al., 2016), “Enhanced Local Self-Attention” for Vision Transformers (Zhou et al., 2021), “Efficient Label Shift Adaptation” in semiparametric domain adaptation (Tian et al., 2023), “Exact Linear-Scan Attention” for Transformer inference (Hsu et al., 26 Apr 2026), and “European Leadership in Space Astrometry” in the Gaia astrometry program (Brown, 2010).

For the urban-vision literature, ELSA therefore refers specifically to Evaluating Localization of Social Activities in Urban Streets using Open-Vocabulary Detection. Its distinctive contribution lies in joining a theory-derived taxonomy of public life with manually audited, multi-label street-level annotations for OVD benchmarking, while explicitly exposing unresolved issues of semantic stability, phrasing sensitivity, and confidence reliability in socially complex urban scenes (Hosseini et al., 2024).