iOSPointMapper: On-Device Sidewalk Mapping

• iOSPointMapper is a real-time, privacy-conscious sidewalk mapping system that integrates on-device semantic segmentation, LiDAR depth estimation, and sensor fusion for precise feature localization.
• It employs user-initiated capture events with temporal fusion and a human-in-the-loop annotation interface to generate anonymized geodata for pedestrian accessibility mapping.
• The system demonstrates robust performance with a mean IoU of 0.7782 and low localization RMSE in urban settings, ensuring enhanced mapping accuracy and scalability.

iOSPointMapper is a real-time, privacy-conscious sidewalk mapping system deployed as a mobile application for recent-generation iPhones and iPads. By integrating on-device semantic segmentation (BiSeNetv2), LiDAR-based depth estimation, and GPS/IMU fusion, iOSPointMapper enables the detection, localization, and human validation of sidewalk-relevant features, transmitting anonymized geodata to improve pedestrian accessibility datasets at scale (Naidu et al., 26 Dec 2025).

1. System Architecture and Data Processing Pipeline

The iOSPointMapper architecture orchestrates three primary on-device computational streams at launch:

• Semantic segmentation on each RGB camera frame (using BiSeNetv2 via CoreML)
• LiDAR depth capture (via ARKit) for per-pixel depth maps
• GPS/IMU pose estimation (via CoreLocation / ARKit) for device localization and camera extrinsics $(R, t)$

Capture Event Workflow

Each user-initiated "capture" event triggers a data pipeline:

1. User framing: The Camera View overlays segmentation masks on a live RGB feed to facilitate accurate framing.
2. Data acquisition: Tapping "Capture" records the current RGB frame $I$, depth map $D$, intrinsics $K$, pose $(R, t)$, and GPS $(\phi_0, \lambda_0)$.
3. Temporal fusion: Segmentation masks from the current and preceding $N$ frames are registered via homographies $H$ (from pose) and fused by per-pixel majority vote.
4. Contour extraction: Discrete object instances $\{\Gamma_k\}$ are extracted from the fused mask.
5. 3D localization: Each $\Gamma_k$ instance undergoes feature localization (see Section 4), projecting image centroids into world and then geographic coordinates.
6. Sidewalk geometry processing: Sidewalk regions are further processed for attributes such as width.
7. User annotation interface: Human-in-the-loop vetting enables interactive confirmation, rejection, and editing of detected features.
8. Data integration: Finalized nodes and ways (OpenSidewalks format) are grouped into a TDEI changeset and uploaded via a REST API.

This architecture enables efficient, low-latency on-device mapping while enforcing privacy constraints and supporting user oversight.

2. Semantic Segmentation and LiDAR Depth Estimation

Semantic Segmentation

• Network: BiSeNetv2, with a dual-branch architecture supporting spatial detail and semantic context.
• Output: $C = 8$ classes (5 point-of-interest features + 3 background classes).
• Training: Pretrained on a 4,300-image Mapillary Vistas subset emphasizing sidewalk features, then fine-tuned using a custom RGB-D pedestrian-perspective dataset (200 train, 50 val images).
• Optimization: Per-pixel cross-entropy loss,

$$\mathcal{L}_{seg} = - \sum_{i=1}^{h}\sum_{j=1}^{w}\sum_{c=1}^{C} y_{ij}^{c} \log(p_{ij}^{c}),$$

with selective fine-tuning and Adam optimizer (initial LR = $1\text{e-}3$, polynomial decay, 100 epochs).

LiDAR Depth Estimation

• Calibration: Camera intrinsics from ARKit, and depth-to-camera extrinsics embedded in $(R, t)$.
• Depth mapping: The raw point cloud $\mathcal{P} = \{\mathbf{X}_k \in \mathbb{R}^3\}$ is projected into the image plane, and for each pixel $(i, j)$, the minimum depth $d_{ij}$ is chosen.
• Filtering: A bilateral filter is applied to $D$ to denoise while preserving edges:

$$D'(i, j) = \frac{\sum_{m, n} w_s(m,n) w_r(D(i,j) - D(m,n)) D(m,n)}{\sum_{m, n} w_s(m,n) w_r(D(i,j) - D(m,n))}$$

This fusion of segmentation and depth enables scene geometry to inform high-fidelity mapping.

3. Sensor Fusion and Georeferencing

iOSPointMapper leverages ARKit's built-in GPS/IMU fusion infrastructure, conceptually modeled as an Extended Kalman Filter (EKF):

• State vector: $$\mathbf{x} = [\mathbf{p}, \mathbf{v}, \mathbf{b}_a, \mathbf{b}_\omega]^T$$ (position, velocity, accelerometer and gyro biases).
• Process model: Follows standard IMU-driven equations,

$$\dot{\mathbf{p}} = \mathbf{v},\quad \dot{\mathbf{v}} = R(\mathbf{q})(\mathbf{a}_m - \mathbf{b}_a - \eta_a) + \mathbf{g},\quad \dot{\mathbf{b}_a} = \eta_{b_a},\quad \dot{\mathbf{b}_\omega} = \eta_{b_\omega}$$

• Discrete prediction and GPS update: Standard EKF updates, with GPS updates every 0.5–1 s.

The fusion outputs geodetic $(\phi, \lambda)$ and 6-DoF pose $(R, t)$ for downstream localization and mapping.

4. Feature Detection, 3D Localization, and Human Annotation

Detection operates on both current and recent frames, temporally fusing segmentation via pose-aligned homographies and pixel-wise majority voting. Contour extraction yields candidate instances for each class.

Feature Localization Algorithm

For each detected region $\Gamma$, the algorithm:

1. Computes the centroid $(u, v)$.
2. Averages the depth across $\Gamma$.
3. Back-projects to camera space: $\mathbf{x}_{cam} = d K^{-1} [u, v, 1]^T$.
4. Transforms to world coordinates: $\mathbf{x}_{world} = R \mathbf{x}_{cam} + t$.
5. Converts spatial offset $\Delta = \mathbf{x}_{world} - t$ to geodetic coordinates using haversine-based spherical formulas.

Pseudo-code for this procedure is supplied in the original work.

Human-in-the-Loop Workflow

A user-guided annotation interface presents each detection for class-wise and instance-wise confirmation (“Agree”, “Discard”, “Missing” for classes; “Accept”, “Reject” for individual instances). For sidewalks, geometric attributes such as width and slope flags are additionally editable.

5. Data Privacy, Anonymization, and Integration Protocols

All computationally intensive operations—semantic segmentation, depth processing, and feature mapping—are performed on-device. Raw images and depth maps are immediately deleted post-inference. Only the following outputs are retained and transmitted:

• A sparse set of nodes $$\{(\text{lat}, \text{lon}, \text{class}, \text{attrs})\}$$ and ordered sidewalk geometries.
• Data conforming to the OpenSidewalks schema.
• Changesets tagged with pseudonymized user and workspace IDs.
• No imagery, personally identifiable information (PII), or unneeded metadata is ever uploaded.

Transmission to the TDEI utilizes a REST API. Users retain veto authority over each mapped element.

6. Quantitative Performance and System Limitations

Segmentation and Localization Metrics

Performance on a held-out pedestrian-perspective validation set demonstrates improved segmentation and localization over baselines. BiSeNetv2-iOS achieves mean IoU for sidewalk class of $0.7782$ and recall of $0.9708$. Localization errors for buildings, traffic signs, and sidewalk elements across three cities are summarized below.

City	Building RMSE (m)	Traffic Sign RMSE (m)	Sidewalk Location RMSE (m)	Sidewalk Width RMSE (m)
Seattle	1.1275	1.2135	0.7579	1.3070
Bellevue	0.8606	1.1781	0.7327	1.1840
Redmond	1.1420	1.2632	0.7614	1.2249

Inference latency on iPhone 16 Pro is approximately 50 ms per frame (20 FPS).

Limitations

• objects $>$ 5 m from the sensor, including traffic lights and poles, are difficult to localize accurately due to LiDAR range.
• Segmentation performance may degrade in adverse conditions (occlusion, poor lighting, environmental clutter).
• Certain infrastructure classes (curb ramps, surface condition) are not yet recognized.

7. Prospective Enhancements and Research Directions

The initial release omits several relevant classes and mapping augmentations. Planned work includes:

• Extending semantic classes to curb ramps, crosswalk markings, surface defects.
• Augmenting mapping range with photogrammetry or multi-view stereo.
• Dead-reckoning and on-device SLAM integration for reliability in areas with weak GPS (urban canyons).
• Overlaying external municipal asset datasets within the annotation interface for in-field validation and longitudinal change detection.
• Exposing slope and cross-slope measurements by combining depth and inertial cues.

These directions target increased coverage, accuracy, and robustness in pedestrian infrastructure mapping for accessibility and multimodal transportation applications (Naidu et al., 26 Dec 2025).