PEMs: Modeling Errors in Perception Systems

• Perception Error Models (PEMs) are statistical constructs that represent empirical error patterns from black-box AI perception systems, focusing on misdetections and noise.
• They employ stochastic frameworks such as switched linear stochastic differential equations and Markov chains to quantify errors in robotics and autonomous vehicles.
• PEMs facilitate robust control synthesis, closed-loop safety validation, and error diagnostics in engineering and NLP, offering actionable insights for safety-critical applications.

A Perception Error Model (PEM) is a formal construct for representing the statistical structure of errors introduced by imperfect, often black-box, perception systems within larger autonomous or intelligent systems. PEMs do not attempt to dissect the internal mechanisms of AI-based perception modules; instead, they aim to characterize, inject, or anticipate the empirical error patterns that such modules introduce. This paradigm shift—from modeling perception systems to modeling their error output—enables principled analysis, robust control synthesis, and closed-loop safety guarantees in safety-critical domains such as autonomous vehicles and robotics (Yan et al., 15 Sep 2025). PEMs have also been adapted in NLP and vision-language modeling to preempt and diagnose input comprehension failures or to provide token-level error supervision (Mi et al., 24 Sep 2025, Min et al., 27 Apr 2026). Their scope spans stochastic system modeling, robust safety validation, virtual testing, and even application to human perceptual phenomena.

1. Formalization of Perception Error Models

A PEM is defined by a stochastic mapping from true world state $\mathcal W$ to the observable output of a perception system, with error $\varepsilon$ encapsulating all sources of deviation: $$\mathrm{PEM}(\mathcal W) = \mathcal W + \varepsilon.$$ The modeling of $\varepsilon$ is domain-specific but generally aspires to capture both instance-level random noise (e.g., additive Gaussian perturbations) and structured errors such as missed detections or misclassifications. In control-theoretic settings, $\varepsilon$ is typically composed of:

• Misdetection: Entire feature or object omissions, often modeled as finite-state, continuous-time Markov chains ($r(t)\in\mathcal I$) with transition rate matrix $Q$ (Yan et al., 15 Sep 2025).
• Measurement noise: Additive white Gaussian noise represented via Wiener processes, modeling persistent estimation jitter.

Alternative domains, such as language modeling, operationalize PEMs as classifiers over pre-generation input features (e.g., token-level surprisals) to identify high-risk prompts subject to semantic misinterpretation (Mi et al., 24 Sep 2025).

2. Representative Mathematical Models

A central feature of PEMs is their use of rigorous stochastic frameworks to reflect real error processes. Key examples include:

• Switched linear stochastic differential equations for AVs:

$$\begin{aligned} \dot x(t) &= A(r(t))x(t) + B(r(t))u(t) \ y(t) &= C(r(t))x(t) + D(r(t))\omega(t) \end{aligned}$$

with $r(t)$ a Markovian mode (e.g., object detected/missed), and $\omega(t)$ standard i.i.d. Gaussian noise (Yan et al., 15 Sep 2025).

• Markovian error processes for detection:

$\varepsilon$0

where $\varepsilon$1 is the generator matrix dictating transitions between error states.

• Input-referenced error features for NLP:

$\varepsilon$2

clusters of which, or aggregate statistics thereof, act as features for preemptively flagging interpretive “blind spots” (Mi et al., 24 Sep 2025).

• Probabilistic surrogates in simulation:

PEMs can be constructed from low-dimensional surrogates $\varepsilon$3 trained to match the conditional error distribution of a real detector $\varepsilon$4, aligning simulated miss/hit rates to fielded perception systems (Innes et al., 2022).

3. Analysis and Synthesis Techniques

For systems incorporating PEMs, formal analysis and robust control synthesis become tractable via stochastic methods:

• Closed-Loop Stochastic Stability: Stability is guaranteed in the mean-square sense by verifying Lyapunov criteria in each error mode and enforcing stochastic Lyapunov inequalities, e.g.,

$\varepsilon$5

This ensures ultimate bounded stability (UBS) of the closed loop even under switching and noise.

• Performance-Guaranteed Control (PGC): Controller gains $\varepsilon$6 are synthesized via convex optimization (semidefinite programming with Linear Matrix Inequality constraints) to guarantee explicit steady-state error bounds and convergence rates,

$\varepsilon$7

with $\varepsilon$8 the optimal value of the convex program over system and error parameters (Yan et al., 15 Sep 2025).

• Input-only and Pre-deployment Error Diagnostics: In NLP, PEMs function as lightweight statistical flaggers for high-risk inputs, enabling pipeline-side