Semantic Cost Map: Spatial Risk Modeling

• Semantic Cost Map is a spatial representation that encodes heterogeneous navigational risks by converting semantic segmentation outputs into cost surfaces.
• It leverages Bayesian occupancy grids and deep learning architectures like PyrOccNet to fuse sensor data and compute context-sensitive costs for autonomous navigation.
• Empirical evaluations on datasets such as Cityscapes and NuScenes demonstrate improved segmentation metrics and highlight trade-offs in ethical cost matrix configurations.

A semantic cost map is a spatial representation that encodes heterogeneous navigational costs based on semantic segmentation outputs, explicitly quantifying the expected risk or cost associated with traversing each region or cell in a scene. These maps convert probabilistic semantic occupancy grids—often derived from deep learning models—into structured, planning-ready cost surfaces, supporting applications such as autonomous navigation, path planning, and safety-critical robotic decision-making. Their construction leverages class probabilities, confusion cost matrices, and Bayesian inference to impose context-sensitive penalties for entering regions associated with static and dynamic object classes, reflecting both geometric and ethical risk preferences (Roddick et al., 2020, Chan et al., 2019).

1. Formal Foundations: Semantic Occupancy and Cost Formulation

Semantic cost maps are built upon the semantic Bayesian occupancy grid formulation, in which the workspace (e.g., the ground-plane) is discretized into a 2D grid of cells indexed by $i$. Each cell $i$ at time $t$ maintains a vector of binary random variables $m_i^c$ indicating the presence $(m_i^c=1)$ or absence $(m_i^c=0)$ of semantic class $c\in\{1,\dots,C\}$. The posterior $p(m_i^c|z_{1:t})$ quantifies the probability that class $c$ occupies cell $i$ given all observations $z_{1:t}$ (typically images or sensor data up to time $t$).

Belief updates are handled using log-odds within a Bayesian filtering framework:

• For class $c$ in cell $i$ at time $t$, the inverse-sensor model produces $p(m_i^c|z_t)$.
• Log-odds are updated recursively as $l_{i,1:t}^c = l_{i,1:t-1}^c + l_{i,t}^c - l_0^c$
• The fused posterior is retrieved as $p(m_i^c | z_{1:t}) = \sigma( l_{i,1:t}^c )$ with $\sigma(\cdot)$ the sigmoid function (Roddick et al., 2020).

Semantic cost maps transform these occupancy probabilities into cost values used by downstream planners. A canonical continuous formulation is:

$$C(i) = w_{\textrm{free}}\,p_i^r + \sum_{c\in D} w_c\,p_i^c + w_{\textrm{obs}}\,p_{\textrm{occ}}(i),$$

where

• $p_i^r$ is the probability of drivable road,
• $D$ is the set of dynamic classes (vehicles, pedestrians, cyclists),
• $p_{\textrm{occ}}(i) = 1-p_i^r$ is the occupancy probability (cell not free road),
• $w_{\textrm{free}}\ll w_{\textrm{obs}}<w_c$ are weightings encoding the penalty structure (Roddick et al., 2020).

Alternatively, a thresholded cost yields discrete semantic labels (free, dynamic, obstacle) using class-wise probability cutoffs.

2. Network Architectures and Probabilistic Inference

End-to-end deep learning networks, specifically Pyramid Occupancy Networks (PyrOccNet), operationalize the creation of semantic occupancy grids directly from monocular images (Roddick et al., 2020). PyrOccNet integrates:

• Encoder backbone with a ResNet-50 FPN extracting multi-scale features,
• Dense transformer layers that map features to polar and then Cartesian BEV coordinates,
• Multi-scale pyramid transformers handling different depth ranges,
• A top-down decoder producing a BEV grid over a spatial extent (e.g., $50\,\textrm{m}\times50\,\textrm{m}$ at $0.25\,\textrm{m/pixel}$).

Multiclass sigmoid activations output per-class posterior probabilities for each cell. Information is accumulated temporally and across multiple cameras using Bayesian fusion:

• Per-camera maps are transformed into a common frame via known extrinsics and combined by summing log-odds,
• Temporal integration leverages recursive log-odds fusion for smooth, history-aware occupancy maps.

This framework supports class-wise fusion, non-exclusive occupancy, and spatially resolved cost surfaces vital for semantic cost mapping.

3. Cost Matrix Specification and Decision Rules

The mapping of semantic beliefs to costs is governed by the choice of a confusion cost matrix $C\in\mathbb{R}_{\ge0}^{N\times N}$, where $C_{\hat k,k}$ quantifies the penalty for predicting class $\hat k$ when the true class is $k$ (Chan et al., 2019). The general cost-based (Bayesian risk-minimizing) decision rule at each pixel $(i,j)$ is:

$$d(x;C)_{ij} = \arg\min_{\hat k} \sum_{k'} C_{\hat k, k'}\,p_{ij}(k'|x)$$

where $p_{ij}(k'|x)$ is the softmax-probability for class $k'$.

Three notable cost matrices are typically analyzed:

• Symmetric (Robotistic) cost $C_R$: All misclassifications equally penalized.
• Altruistic cost $C_A$: High penalties for confusing “human” with any other class.
• Egoistic cost $C_E$: High penalties for misclassifying road/flat/static with each other, but lesser for “human” errors.

A convex combination $$C(\alpha,\beta,\gamma) = \alpha C_R + \beta C_A + \gamma C_E$$ interpolates ethical attitudes. The expected cost volume $M_{ij}(k) = \sum_{k'} C_{k,k'}\,p_{ij}(k'|x)$ enables cost surface visualization and label selection based on minimizing risk.

4. Semantic Cost Surfaces for Planning

Semantic cost maps are designed for downstream planners to evaluate the relative desirability of traversing each cell. Cost weights (e.g., $w_{\textrm{free}} = 1$, $w_{\textrm{obs}} = 50$, $w_{\textrm{vehicle}} = 100$, $w_{\textrm{pedestrian}} = 200$, $w_{\textrm{cyclist}} = 150$) encode domain-specific risk aversion: free regions are cheap, static obstacles are costly, dynamic agents incur the highest penalty (Roddick et al., 2020).

Discrete and continuous cost constructions provide flexibility:

• Discrete: Assigning “safe,” “dynamic,” or “obstacle” labels by thresholding probabilities.
• Continuous: Summing weighted probabilities for all relevant classes to express a nuanced, passage-specific cost.

This cost surface is directly used by motion-planning algorithms to favor safer, more efficient navigation routes.

5. Empirical Evaluation and Quantitative Results

Semantic cost mapping techniques are benchmarked using metrics such as Intersection-over-Union (IoU) per class, mean IoU, and Cityscapes-class mean IoU (Roddick et al., 2020). The PyrOccNet model achieves substantial improvements:

• On Argoverse, Cityscapes-mean IoU increases by $22.3\%$ relative compared to the best previous baseline.
• On NuScenes, it achieves a $9.1\%$ relative IoU improvement.

Cost-based segmentation rules affect precision-recall tradeoffs:

• Altruistic costs result in person recall climbing to $\approx 99.8\%$, but precision drops to $\approx 40\%$.
• Egoistic costs yield higher person precision ($\approx 94\%$), but lower recall ($\approx 70\%$) (Chan et al., 2019).

The table below summarizes key results for pixel-wise semantic segmentation under distinct cost matrices in two regions of interest (RoI):

Cost matrix	Class	RoI	Precision	Recall
Altruistic	Person	1	41.1%	99.8%
Robotistic	Person	1	89.9%	94.9%
Egoistic	Person	1	93.9%	70.1%
Altruistic	Building	1	22.6%	93.7%
Robotistic	Building	1	81.0%	94.9%
Egoistic	Building	1	15.2%	99.9%

These results demonstrate the sensitivity of semantic cost surfaces—and thus planning outcomes—to domain, ethics, and context encoded in the cost matrix.

6. Applications, Context, and Ethical Dimensions

Semantic cost maps enable semantic-aware motion planning, risk-sensitive navigation, and principled tradeoffs between safety and efficiency. The mapping from semantic grid probabilities to planning costs is not uniquely defined and may be altered based on application-specific safety envelopes, societal priorities, or regulatory constraints.

A key consideration is the explicit encoding of ethical stances in the cost matrix. For urban driving, over-prioritizing physical obstacles may reduce collision risk for vehicles but endanger vulnerable road-users if their semantic class is undervalued. Conversely, “altruistic” matrices can reduce collision risk for non-vehicle actors, at a cost to efficiency or overconservative planning (Chan et al., 2019).

The selection or tuning of $C$ thus implicates nontrivial ethical judgments, and exploring interpolations between robotistic, altruistic, and egoistic cost matrices reveals how performance, error patterns, and planning outcomes depend on these choices.

7. Limitations and Open Directions

Semantic cost mapping is fundamentally constrained by the underlying segmentation quality, uncertainty propagation, and the subjective nature of cost assignments. Limitations include:

• Sensitivity to sensor occlusions or out-of-distribution artifacts,
• Non-exclusivity of cell occupancy complicating assignment,
• Absence of a “principled” universal cost matrix, leading to reliance on application-specific or regulatory heuristics.

A plausible implication is that research should continue to formalize ethical frameworks for cost specification and explore calibration techniques that make the tradeoffs between safety, efficiency, and social responsibility explicit (Chan et al., 2019). Future directions include data-driven or participatory cost matrix design and robust uncertainty propagation in semantic cost maps.

---

Semantic cost maps constitute a principled bridge from probabilistic semantic perception to risk-aware action, their form and function critically shaped by both mathematical formalism and explicit or implicit ethical design (Roddick et al., 2020, Chan et al., 2019).