Perception Error Modeling Overview

• Perception Error Modeling is a framework that mathematically and empirically characterizes discrepancies between sensed data and the true environment using probabilistic and stochastic methods.
• It employs techniques like Bayesian decision theory, Markov models, and optimal transport to analyze error propagation in downstream tasks such as decision making and control.
• Data-driven approaches and human-centered calibrations complement theoretical models to improve safety and robustness in autonomous and perception-dependent systems.

Perception Error Modeling encompasses mathematical, algorithmic, and empirical frameworks for capturing, analyzing, and minimizing discrepancies between sensed or inferred representations of the environment and their ground-truth referents. These models characterize perception errors not only as isolated, static phenomena (e.g., detection misses or parameter noise), but also as temporally and contextually structured events that propagate through downstream tasks—decision making, control, planning, or human interpretation. Contemporary research formalizes these errors using tools from probabilistic logic, optimal transport, Bayesian decision theory, stochastic processes, and human-perception studies, enabling the design and validation of robust perception-driven systems.

1. Formal Definitions and Taxonomies

Perception Error Models (PEMs) are typically defined as computational surrogates that transform a true world state $W=\{o_1, ..., o_n\}$ into a perceived world $\widehat{W} = W + E$, where $E$ denotes the aggregate error process (Piazzoni et al., 2023). PEMs abstract away the costly simulation of raw sensor data by modeling the statistics of detection misses (false negatives), hallucinations (false positives), misclassification (label confusions), parameter noise (regression errors), and temporally correlated failures (dropouts, intermittent losses) (Piazzoni et al., 2020, Piazzoni et al., 2022).

A common structural taxonomy separates:

• Geometry-based errors: Positional, orientation, or size deviations, typically modeled as additive or multiplicative noise in a geometric parameter space (Piazzoni et al., 2023).
• Categorical/misclassification errors: Probabilistic label flips, parameterized by misclassification matrices (Piazzoni et al., 2020).
• Temporal errors: Markovian or more complex dependencies across time, as in detection loss with key dependence on sojourn times (Piazzoni et al., 2020).
• Measurement noise: Additive Gaussian noise, possibly context-dependent (Yan et al., 15 Sep 2025).
• Composite errors: Coupling between error modes, e.g., occlusion-induced detection loss in cooperative settings (Piazzoni et al., 2022).

In sequential or time-structured systems, PEMs often instantiate as Markov chains or hidden Markov models (HMMs) controlling detection and parameter evolution (Piazzoni et al., 2023, Piazzoni et al., 2020).

2. Mathematical and Probabilistic Frameworks

Rigorous perception-error modeling relies on probabilistic signal temporal logic (PSTL), stochastic process theory, and Bayesian risk metrics.

Probabilistic Signal Temporal Logic (PSTL):

CogSense(Kwon et al., 2021) extends traditional signal temporal logic by defining predicates over sliding windows of heterogeneous probe functions (geometry, dynamics, image quality) and constrains their empirical distributions using probabilistic bands. Formally, for a probe $f_j$, the axiom

$$\Phi_j \equiv \Box_{[t-M, t]} \Pr(a_j \leq f_j(x_\tau) \leq b_j) \geq P_{TP}^j$$

is derived to ensure statistical validity with respect to true positive (TP) empirical data.

Stochastic Process Models:

Measurement and misdetection errors are modeled via jump-diffusion systems, combining continuous-time Markov chains (CTMCs) to represent mode-switching (e.g., detection/misdetection) with Wiener processes for measurement noise (Yan et al., 15 Sep 2025). Stability and boundedness of the closed-loop system under PEM-augmented models are analyzed via Lyapunov theory and linear matrix inequalities (LMIs).

Risk-Driven and Bayesian Formulations:

Risk-driven design (Corso et al., 2022, Liu et al., 2023) considers the downstream impact of a perception error $ε$ at state $x$ by computing the conditional value-at-risk (CVaR) of the cost-to-go: $R(ε, x) = \text{CVaR}_\alpha[Z_{\pi_ε}(x, ε)]$ where $Z_{\pi_ε}$ is the cumulative cost under a perception-error policy. This leads to both risk-sensitive loss functions in training and targeted data-collection schemes that emphasize dangerous regimes.

3. Data-Driven and Generative Modeling

Parametric and Data-driven PEMs:

Practical PEMs for AV testing are constructed by fitting context-dependent detection and parameter-noise models to real or simulated detector outputs. Common methods include:

• Partitioning the state-space (e.g., polar occupancy grid × occlusion bins)
• Fitting Markov transition matrices for detection/loss events
• Estimating context-dependent bivariate noise models for parameter outputs, regularized across space via conditional autoregressive (CAR) priors (Piazzoni et al., 2023)

Generative and Set-based Models:

EMPERROR (Hanselmann et al., 2024) advances PEMs by learning the full distribution $P(\hat{y}|y)$ over detector outputs given ground-truth. This is implemented as a conditional variational autoencoder (CVAE) with transformer-based architecture, capturing correlation and error modes (including false positives and scene-level dependencies) across objects.

Adaptive and Adversarial Error Synthesis:

Boundary-attack approaches (Sadeghi et al., 2023) expose planner-failure-inducing perturbations that evade standard perception score metrics. These adversarial PEMs are essential to reveal the limitations of average-case PEM training and the “narrowness” of safety-compromising error modes.

4. Perceptual and Human-Centered Error Metrics

Perceptual error modeling seeks to align objective errors with human-perceived degradation, guiding both algorithmic optimization and evaluation.

• Perceptual Metric Derivation: Large-scale human studies quantify error visibility, yielding mappings from parameter errors to empirically observed confusion scores. For example, in camera calibration, a learned perceptual metric $D_{perp}$ is built from user response data and fitted as a $k$-NN regressor in error-context space (Hold-Geoffroy et al., 2017).
• HVS-based Metrics in Rendering: In Monte Carlo image and animation rendering, errors are filtered through human visual system (HVS) models—e.g., Gaussian point-spread functions, contrast sensitivity functions—to minimize the perceptual (rather than numerical) impact of estimation error. The optimization explicitly distributes error as blue noise, both spatially and temporally, resulting in substantially improved perceived fidelity (Chizhov et al., 2020, Korać et al., 2023).
• Rate–Distortion–Perception Theory: In semantic communication and video compression, perceptual loss functions (frame-marginal, joint-distribution, or self-adaptive) are incorporated into rate-distortion tradeoffs, with the monotonicity property of the perception-error function with respect to channel error proved theoretically and observed numerically (Xu et al., 2024, Salehkalaibar et al., 2023, Salehkalaibar et al., 15 Feb 2025).

5. Closed-Loop Impact and Safety Assessment

Comprehensive perception error models reveal nontrivial pathways by which upstream errors cascade into downstream failures in control and planning systems.

• End-to-End Simulation: PEMs allow decoupled, efficient study of the effect of perception errors on closed-loop AV safety and decision outcomes without simulating raw sensor data, supporting scenario- and context-specific robustness studies (Piazzoni et al., 2020, Piazzoni et al., 2023, Piazzoni et al., 2022).
• Adaptive Sampling for Rare Events: State-dependent adaptive importance sampling, powered by learned PEMs, enables practical estimation of the probability of rare safety violations in black-box perceptual-control systems (Innes et al., 2022). Here, the interplay between the PEM, proposal distribution, and safety metric crucially determines both the convergence and the interpretability of safety evaluations.
• Contextual and Semantic Perception Risk: Risk-aware metrics (e.g., CVaR of action costs) can drive both system design and operational decision making, e.g., by influencing high-level action choices in the presence of ambiguous/failure-prone perception, as in symbolic perception risk modeling for traffic sign recognition (Liu et al., 2023).

6. Human-in-the-Loop and Psychophysical Error Calibration

Perception error modeling is not limited to artificial systems. Computational frameworks for error signals in human perception, such as the mismatch negativity (MMN) in predictive coding, formalize context-dependent, trial-by-trial perceptual learning error using Bayesian inference, Kalman filtering, and surprise-based models (Lecaignard et al., 2023). These frameworks allow decomposition of aggregate prediction errors into interpretable, hierarchical learning signals, and enable neurophysiological mapping and clinical interpretation.

In stereoscopic display, geometric PEMs predict bias in perceived depth or distance via explicit algebraic relationships between rendering/viewing misalignments and disparity cues, validated by both psychophysical data and closed-loop recalibration experiments (Zhu et al., 29 May 2025).

---

References:

• CogSense: A Cognitively Inspired Framework for Perception Adaptation (Kwon et al., 2021)
• Risk-Driven Design of Perception Systems (Corso et al., 2022)
• Symbolic Perception Risk in Autonomous Driving (Liu et al., 2023)
• Control Analysis and Design for Autonomous Vehicles Subject to Imperfect AI-Based Perception (Yan et al., 15 Sep 2025)
• Perceptual error optimization for Monte Carlo animation rendering (Korać et al., 2023)
• Modeling Perception Errors towards Robust Decision Making in Autonomous Vehicles (Piazzoni et al., 2020)
• Attacking Motion Planners Using Adversarial Perception Errors (Sadeghi et al., 2023)
• From Input Perception to Predictive Insight (Mi et al., 24 Sep 2025)
• CoPEM: Cooperative Perception Error Models for Autonomous Driving (Piazzoni et al., 2022)
• PEM: Perception Error Model for Virtual Testing of Autonomous Vehicles (Piazzoni et al., 2023)
• Generative Semantic Communications with Foundation Models (Xu et al., 2024)
• Errors in Stereo Geometry Induce Distance Misperception (Zhu et al., 29 May 2025)
• On the Choice of Perception Loss Function for Learned Video Compression (Salehkalaibar et al., 2023)
• An experiment in price perception error (Berry, 24 Jun 2025)
• Testing Rare Downstream Safety Violations via Upstream Adaptive Sampling of Perception Error Models (Innes et al., 2022)
• Mismatch Negativity: time for deconstruction (Lecaignard et al., 2023)
• On Self-Adaptive Perception Loss Function for Sequential Lossy Compression (Salehkalaibar et al., 15 Feb 2025)
• A Perceptual Measure for Deep Single Image Camera Calibration (Hold-Geoffroy et al., 2017)
• EMPERROR: A Flexible Generative Perception Error Model for Probing Self-Driving Planners (Hanselmann et al., 2024)