---
title: 'iOSPointMapper: On-Device Sidewalk Mapping'
url: https://www.emergentmind.com/topics/iospointmapper
type: topic
---

# iOSPointMapper: On-Device Sidewalk Mapping

iOSPointMapper is a real-time, privacy-conscious sidewalk mapping system deployed as a mobile application for recent-generation iPhones and iPads. By integrating on-device semantic segmentation (BiSeNetv2), LiDAR-based depth estimation, and GPS/IMU fusion, iOSPointMapper enables the detection, localization, and human validation of sidewalk-relevant features, transmitting anonymized geodata to improve pedestrian accessibility datasets at scale [2512.22392].

## 1. System Architecture and Data Processing Pipeline

The iOSPointMapper architecture orchestrates three primary on-device computational streams at launch:
- **Semantic segmentation** on each RGB camera frame (using BiSeNetv2 via CoreML)
- **LiDAR depth capture** (via ARKit) for per-pixel depth maps
- **GPS/IMU pose estimation** (via CoreLocation / ARKit) for device localization and camera extrinsics $(R, t)$

### Capture Event Workflow

Each user-initiated "capture" event triggers a data pipeline:
1. **User framing:** The Camera View overlays segmentation masks on a live RGB feed to facilitate accurate framing.
2. **Data acquisition:** Tapping "Capture" records the current RGB frame $I$, depth map $D$, intrinsics $K$, pose $(R, t)$, and GPS $(\phi_0, \lambda_0)$.
3. **Temporal fusion:** Segmentation masks from the current and preceding $N$ frames are registered via homographies $H$ (from pose) and fused by per-pixel majority vote.
4. **Contour extraction:** Discrete object instances $\{\Gamma_k\}$ are extracted from the fused mask.
5. **3D localization:** Each $\Gamma_k$ instance undergoes feature localization (see Section 4), projecting image centroids into world and then geographic coordinates.
6. **Sidewalk geometry processing:** Sidewalk regions are further processed for attributes such as width.
7. **User annotation interface:** Human-in-the-loop vetting enables interactive confirmation, rejection, and editing of detected features.
8. **Data integration:** Finalized nodes and ways (OpenSidewalks format) are grouped into a TDEI changeset and uploaded via a REST API.

This architecture enables efficient, low-latency on-device mapping while enforcing privacy constraints and supporting user oversight.

## 2. Semantic Segmentation and LiDAR Depth Estimation

### Semantic Segmentation

- **Network:** BiSeNetv2, with a dual-branch architecture supporting spatial detail and semantic context.
- **Output:** $C = 8$ classes (5 point-of-interest features + 3 background classes).
- **Training:** Pretrained on a 4,300-image Mapillary Vistas subset emphasizing sidewalk features, then fine-tuned using a custom RGB-D pedestrian-perspective dataset (200 train, 50 val images).
- **Optimization:** Per-pixel cross-entropy loss,
  $$
  \mathcal{L}_{seg} = - \sum_{i=1}^{h}\sum_{j=1}^{w}\sum_{c=1}^{C} y_{ij}^{c} \log(p_{ij}^{c}),
  $$
  with selective fine-tuning and Adam optimizer (initial LR = $1\text{e-}3$, polynomial decay, 100 epochs).

### LiDAR Depth Estimation

- **Calibration:** Camera intrinsics from ARKit, and depth-to-camera extrinsics embedded in $(R, t)$.
- **Depth mapping:** The raw point cloud $\mathcal{P} = \{\mathbf{X}_k \in \mathbb{R}^3\}$ is projected into the image plane, and for each pixel $(i, j)$, the minimum depth $d_{ij}$ is chosen.
- **Filtering:** A bilateral filter is applied to $D$ to denoise while preserving edges:
  $$
  D'(i, j) = \frac{\sum_{m, n} w_s(m,n) w_r(D(i,j) - D(m,n)) D(m,n)}{\sum_{m, n} w_s(m,n) w_r(D(i,j) - D(m,n))}
  $$

This fusion of segmentation and depth enables scene geometry to inform high-fidelity mapping.

## 3. Sensor Fusion and Georeferencing

iOSPointMapper leverages ARKit's built-in GPS/IMU fusion infrastructure, conceptually modeled as an Extended Kalman Filter (EKF):

- **State vector:** $\mathbf{x} = [\mathbf{p}, \mathbf{v}, \mathbf{b}_a, \mathbf{b}_\omega]^T$ (position, velocity, accelerometer and gyro biases).
- **Process model:** Follows standard IMU-driven equations,
  $$
  \dot{\mathbf{p}} = \mathbf{v},\quad 
  \dot{\mathbf{v}} = R(\mathbf{q})(\mathbf{a}_m - \mathbf{b}_a - \eta_a) + \mathbf{g},\quad  
  \dot{\mathbf{b}_a} = \eta_{b_a},\quad
  \dot{\mathbf{b}_\omega} = \eta_{b_\omega}
  $$
- **Discrete prediction and GPS update:** Standard EKF updates, with GPS updates every 0.5–1 s.

The fusion outputs geodetic $(\phi, \lambda)$ and 6-DoF pose $(R, t)$ for downstream localization and mapping.

## 4. Feature Detection, 3D Localization, and Human Annotation

Detection operates on both current and recent frames, temporally fusing segmentation via pose-aligned homographies and pixel-wise majority voting. Contour extraction yields candidate instances for each class.

### Feature Localization Algorithm

For each detected region $\Gamma$, the algorithm:
1. Computes the centroid $(u, v)$.
2. Averages the depth across $\Gamma$.
3. Back-projects to camera space: $\mathbf{x}_{cam} = d K^{-1} [u, v, 1]^T$.
4. Transforms to world coordinates: $\mathbf{x}_{world} = R \mathbf{x}_{cam} + t$.
5. Converts spatial offset $\Delta = \mathbf{x}_{world} - t$ to geodetic coordinates using haversine-based spherical formulas.

Pseudo-code for this procedure is supplied in the original work.

### Human-in-the-Loop Workflow

A user-guided annotation interface presents each detection for class-wise and instance-wise confirmation (“Agree”, “Discard”, “Missing” for classes; “Accept”, “Reject” for individual instances). For sidewalks, geometric attributes such as width and slope flags are additionally editable.

## 5. Data Privacy, Anonymization, and Integration Protocols

All computationally intensive operations—semantic segmentation, depth processing, and feature mapping—are performed on-device. Raw images and depth maps are immediately deleted post-inference. Only the following outputs are retained and transmitted:

- A sparse set of nodes $\{(\text{lat}, \text{lon}, \text{class}, \text{attrs})\}$ and ordered sidewalk geometries.
- Data conforming to the OpenSidewalks schema.
- Changesets tagged with pseudonymized user and workspace IDs.
- No imagery, personally identifiable information (PII), or unneeded metadata is ever uploaded.

Transmission to the TDEI utilizes a REST API. Users retain veto authority over each mapped element.

## 6. Quantitative Performance and System Limitations

### Segmentation and Localization Metrics

Performance on a held-out pedestrian-perspective validation set demonstrates improved segmentation and localization over baselines. BiSeNetv2-iOS achieves mean IoU for sidewalk class of $0.7782$ and recall of $0.9708$. Localization errors for buildings, traffic signs, and sidewalk elements across three cities are summarized below.

| City     | Building RMSE (m) | Traffic Sign RMSE (m) | Sidewalk Location RMSE (m) | Sidewalk Width RMSE (m) |
|----------|-------------------|-----------------------|----------------------------|-------------------------|
| Seattle  | 1.1275            | 1.2135                | 0.7579                     | 1.3070                  |
| Bellevue | 0.8606            | 1.1781                | 0.7327                     | 1.1840                  |
| Redmond  | 1.1420            | 1.2632                | 0.7614                     | 1.2249                  |

Inference latency on iPhone 16 Pro is approximately 50 ms per frame (20 FPS).

### Limitations

- objects $>$ 5 m from the sensor, including traffic lights and poles, are difficult to localize accurately due to LiDAR range.
- Segmentation performance may degrade in adverse conditions (occlusion, poor lighting, environmental clutter).
- Certain infrastructure classes (curb ramps, surface condition) are not yet recognized.

## 7. Prospective Enhancements and Research Directions

The initial release omits several relevant classes and mapping augmentations. Planned work includes:

- Extending semantic classes to curb ramps, crosswalk markings, surface defects.
- Augmenting mapping range with photogrammetry or multi-view stereo.
- Dead-reckoning and on-device SLAM integration for reliability in areas with weak GPS (urban canyons).
- Overlaying external municipal asset datasets within the annotation interface for in-field validation and longitudinal change detection.
- Exposing slope and cross-slope measurements by combining depth and inertial cues.

These directions target increased coverage, accuracy, and robustness in pedestrian infrastructure mapping for accessibility and multimodal transportation applications [2512.22392].

Source: https://www.emergentmind.com/topics/iospointmapper