PEM-Augmented Driving Model

• PEM-Augmented Driving Models are architectures that extend nominal driving stacks with auxiliary modules to inject additional structure, uncertainty, context, or memory.
• They encompass various implementations such as pedestrian environment modeling, perception error injection, and past-experience memory to address specific shortcomings.
• Empirical evaluations show that these augmentations can reduce estimation errors and improve safety metrics without replacing the baseline models.

Searching arXiv for the cited paper and closely related PEM/driving-model literature to ground the article in current arXiv records. A PEM-Augmented Driving Model is a driving architecture in which a nominal perception, behavior, planning, or control stack is extended by an auxiliary PEM module that injects additional structure, uncertainty, context, or memory into downstream decision-making. In the arXiv literature, however, PEM is not a standardized acronym. It can denote a Pedestrian Environment Model (Holzbock et al., 2023), a Perception Error Model for virtual testing and planner stress-testing (Piazzoni et al., 2023, Hanselmann et al., 2024), a psychological or passenger evaluation model embedded in local planning (Deng et al., 2023), or a Past-Experience Memory used for retrieval-augmented maneuver recommendation (Gan et al., 25 Jun 2025). Closely related work also includes Gaussian process augmented state-space models for driver behavior learning (Kullberg et al., 2020), augmented driver behavior models for high-fidelity simulation (Jami et al., 2022), and physics-augmented ACC/car-following models (He et al., 2021). Taken together, these works define a family of augmentation strategies rather than a single canonical model.

1. Terminological scope and architectural variants

The term “PEM-Augmented Driving Model” is best understood as an umbrella expression for architectures that preserve a baseline driving model while adding an auxiliary module that compensates for missing information, unmodeled structure, or evaluation blind spots. The augmentation target differs across subfields: some PEMs enrich environment representation, some reproduce perception failures, some encode human comfort or memory, and some augment longitudinal or behavioral dynamics.

PEM meaning	Role in the stack	Representative paper
Pedestrian Environment Model	Adds tracked skeletons and monocular 3D pedestrian positions	(Holzbock et al., 2023)
Perception Error Model	Maps ground truth to perceived objects for safety evaluation	(Piazzoni et al., 2023)
Generative Perception Error Model	Samples correlated detector failures for planner probing	(Hanselmann et al., 2024)
Psychological/Passenger Evaluation Model	Penalizes low-confidence or psychologically unsafe trajectories in MPC	(Deng et al., 2023)
Past-Experience Memory	Retrieves prior safety-critical cases for LLM decision support	(Gan et al., 25 Jun 2025)
Prototype-based Efficient MaskFormer	Efficient perception module for semantic and panoptic segmentation	(Cavagnero et al., 2024)

A common misconception is that PEM necessarily means Perception Error Model. The literature does not support that restriction. Another misconception is that PEM augmentation always replaces the nominal model. In most of these works, the PEM is instead a surrogate, corrective, or conditioning layer placed around an existing planner, controller, estimator, or perception stack.

2. Behavioral and physical augmentation of driving dynamics

One line of work augments driver or vehicle dynamics directly. “Learning Driver Behaviors Using A Gaussian Process Augmented State-Space Model” presents an inference method for Gaussian process augmented state-space models, described as a class of grey-box models that incorporate domain knowledge while remaining flexible enough to learn partially unknown model dynamics and inputs. To support online inference, it introduces a sparse approximation of the Gaussian process based upon inducing points. In an intersection example involving cars passing through the scene, the augmented model produced reduced estimation error and bias relative to using only the state-space model (Kullberg et al., 2020).

A different but structurally related formulation appears in “Physics-augmented models to simulate commercial adaptive cruise control (ACC) systems,” where a baseline car-following or ACC law is augmented by perception delay, vehicle dynamics, and acceleration constraints (He et al., 2021). In that framework, delayed measurements

$$\hat{v}(t)=v(t-\tau_p),\quad \hat{s}(t)=s(t-\tau_p),\quad \Delta \hat{v}(t)=v_\ell(t-\tau_p)-v(t-\tau_p)$$

feed either IDM, Gipps, or linear ACC controllers, after which the commanded acceleration is filtered by actuation dynamics such as

$\tau_a \dot{a}(t)+a(t)=u(t).$

Across ninety models formed from five base models and incremental physics extensions, perception delay and linear vehicle dynamics were the extensions that mostly increased modeling accuracy when applied singly, whereas IDM-based models were reported as the most robust in validation, showing almost no crash when calibrated parameters were transferred to different trajectories (He et al., 2021).

“Augmented Driver Behavior Models for High-Fidelity Simulation Study of Crash Detection Algorithms” pushes the same idea into hybrid traffic simulation. It augments a longitudinal IDM controller with perception-reaction latency, field-of-view-based situation awareness, communication-driven sensing with random packet losses and motion prediction, and smoothing/low-pass filtering of actuation and sensed acceleration (Jami et al., 2022). The driver model remains human-interpretable because its structure is decomposed into IDM decision-making, a Mamdani fuzzy inference system with a PD stabilizer, explicit reaction times, and tunable driver-class parameters derived from NGSIM I-80 trajectories. In evaluation, the no-warning baseline with 3% distracted drivers yielded an average of 42 collisions, whereas FCW configurations reduced at-fault collisions differently across aggressive, normal, and conservative drivers; notably, NHTSA Early achieved 87.0% positive warnings for conservative drivers and reduced their at-fault rear-end collisions to zero under the tested conditions (Jami et al., 2022).

These model-side augmentations share a common principle: the nominal controller is preserved, but perceptual, dynamic, or stochastic effects are injected at interfaces where simplified models are known to fail. This suggests that, in driving dynamics, augmentation is most effective when it targets identifiable structural deficits rather than replacing the entire model class.

3. Environment augmentation through pedestrian-centric representation

In “Pedestrian Environment Model for Automated Driving,” PEM denotes a Pedestrian Environment Model that extends an automated driving environment model with both pedestrian position and pose information using only monocular camera images and vehicle localization (Holzbock et al., 2023). The pipeline combines a CID human pose estimator with HRNet-W32 backbone, a tracking stage based on the Hungarian algorithm with negative generalized IoU cost, ego-motion compensation using yaw change, and a multi-frame geometric reconstruction procedure that estimates 3D position from rays across consecutive frames.

The model output at each time step includes a 2D skeletal pose with 17 COCO joints, a derived 2D bounding box, a tracking identity, and a 3D world position $L_{t,r,f}=[x,y,z]$ smoothed with a Kalman filter (constant-velocity model). The stored skeleton is explicitly intended as a substrate for gesture recognition, pedestrian behavior prediction, and pose forecasting, but the work does not implement an explicit intent classifier (Holzbock et al., 2023).

Dataset	Overall position error	Note
CARLA	$e_{abs}=9.123$ m, $e_{rel}=16.85\%$	12,771 pedestrian instances
nuScenes	$e_{abs}=2.524$ m, $e_{rel}=15.66\%$	194 pedestrian instances

The reported relative position error of around 16% on both datasets is one of the central quantitative results. On nuScenes, error stayed below 2 m up to around 13 m and then rose as detections thinned out. Removing the refinement step increased $e_{abs}$ from 2.524 m to 6.601 m, showing that the image-height reprojection refinement is not merely cosmetic but central to the geometric pipeline (Holzbock et al., 2023).

A second environmental interpretation of PEM appears in “Prototype-based Efficient MaskFormer,” where PEM is not a driving model per se but an efficient segmentation architecture intended to serve real-time perception stacks on edge devices (Cavagnero et al., 2024). It replaces dense masked cross-attention with prototype-based cross-attention and couples it to an efficient multi-scale pixel decoder. On Cityscapes panoptic segmentation with a ResNet-50 backbone, it achieved 61.1 PQ at 13.8 FPS, 237 GFLOPs, 35.6M parameters, compared with 62.1 PQ at 4.1 FPS, 519 GFLOPs, 44.0M params for Mask2Former (Cavagnero et al., 2024). In a PEM-augmented driving stack, such a module is best interpreted as an efficient perception front-end rather than a behavioral or planning augmenter.

4. Perception-error augmentation for planner evaluation and robustness

A major branch of the literature uses PEM to denote a Perception Error Model. “PEM: Perception Error Model for Virtual Testing of Autonomous Vehicles” formalizes PEMs as compact, data-driven surrogates for the sensing-plus-perception subsystem that map the ground-truth world to a perceived world by injecting realistic detection and state-estimation errors at the object level (Piazzoni et al., 2023). In the reported implementation, objects are represented in ego-centric polar coordinates, detection is governed by a context-conditioned HMM, and localization errors are drawn from Gaussian emissions conditioned on zone and occlusion partitions. The transition matrix

$$A^c=\begin{bmatrix} a_{00}^c & a_{01}^c\ a_{10}^c & a_{11}^c \end{bmatrix}$$

is learned by maximum likelihood, while emissions $(\epsilon_r,\epsilon_\theta)$ are smoothed across spatial partitions via Conditional Autoregressive priors.

The practical significance of this formulation lies in its interface: PEM replaces explicit sensor simulation with a lower-latency object-level surrogate situated between simulator ground truth and the autonomy stack. In SVL–Apollo experiments, 500 runs per scenario were executed for each sensor configuration. In a jaywalking scenario, camera-only PEM led to 99.2% of runs with $\tau_a \dot{a}(t)+a(t)=u(t).$0 m, whereas LiDAR-only and fusion reached 100% runs with $\tau_a \dot{a}(t)+a(t)=u(t).$1 m. The paper argues that mAP, IoU, and detection rates are insufficient as safety metrics because they do not explain downstream braking and collision behavior (Piazzoni et al., 2023).

“EMPERROR: A Flexible Generative Perception Error Model for Probing Self-Driving Planners” generalizes this direction by modeling full-scene detector behavior with a transformer CVAE (Hanselmann et al., 2024). Rather than perturbing objects independently, EMPERROR learns

$\tau_a \dot{a}(t)+a(t)=u(t).$2

where $\tau_a \dot{a}(t)+a(t)=u(t).$3 is the ground-truth dynamic object set, $\tau_a \dot{a}(t)+a(t)=u(t).$4 is a BEV map, and $\tau_a \dot{a}(t)+a(t)=u(t).$5 is a set of per-query latent variables. It explicitly models false negatives, false positives, regression perturbations, duplicate detections, and correlated context-dependent errors. Training uses a reconstruction term plus a skew-geometric Jensen–Shannon divergence rather than a KL term, with $\tau_a \dot{a}(t)+a(t)=u(t).$6, $\tau_a \dot{a}(t)+a(t)=u(t).$7, $\tau_a \dot{a}(t)+a(t)=u(t).$8, and $\tau_a \dot{a}(t)+a(t)=u(t).$9 in the default configuration.

The central finding is not only detector-fidelity improvement but planner vulnerability. Against DETR3D, BEVFormer, and StreamPETR, EMPERROR reported consistently lower cumulative absolute difference area over recall curves than baseline PEMs. When used adversarially, it increased planner collision rate by up to 85%: for BEVFormer, collision rate rose from 3.20 on detector inputs to 6.20 at $L_{t,r,f}=[x,y,z]$0; for StreamPETR, from 3.40 to 6.28; for DETR3D, from 3.40 to 5.88 (Hanselmann et al., 2024). The important controversy here is methodological: planner evaluation on ground-truth state alone can conceal long-tail perception failures, but excessively simplistic PEMs can also underrepresent those failures. EMPERROR addresses this by moving from per-object perturbation to scene-level generative modeling.

5. Human factors, comfort, and memory augmentation

Another usage of PEM is explicitly human-centered. “Evaluation and Control Model Design of Human Factors for Autonomous Driving Systems” defines PEM as a combined subjective/objective evaluation model and psychological safety boundary that augments local path planning and tracking via MPC (Deng et al., 2023). The evaluation model maps objective vehicle-dynamics features—such as longitudinal speed, lateral acceleration, yaw rate, changing rate of lateral acceleration, and yaw angular acceleration—to three subjective classes: good, normal, and poor. Four classifiers were evaluated, with Mahalanobis distance performing best at 68.18% accuracy.

The same work introduces the PLPTS boundary, a speed-dependent psychologically last point to steer. The measured distances were 7.67 m at 10 km/h, 8.65 m at 30 km/h, 15.27 m at 60 km/h, and 18.15 m at 80 km/h. These quantities enter an MPC cost of the form

$L_{t,r,f}=[x,y,z]$1

where obstacle and road terms are implemented through artificial potential fields reflecting psychological pressure near obstacles and road boundaries. In co-simulation with MATLAB/Simulink and CarSim, increasing the obstacle potential weight $L_{t,r,f}=[x,y,z]$2 from 30 to 80 caused earlier steering farther from the obstacle, while the classifier-integrated planner increased the number of trajectory points classified good and reduced poor points (Deng et al., 2023). This formulation treats augmentation as a direct incorporation of human comfort into optimization rather than as a post hoc evaluation metric.

A more recent interpretation appears in “Case-based Reasoning Augmented LLM Framework for Decision Making in Realistic Safety-Critical Driving Scenarios,” where PEM is mapped to a Past-Experience Memory (Gan et al., 25 Jun 2025). The memory is defined as

$L_{t,r,f}=[x,y,z]$3

with case descriptions $L_{t,r,f}=[x,y,z]$4, embeddings $L_{t,r,f}=[x,y,z]$5, maneuver labels $L_{t,r,f}=[x,y,z]$6, and risk tags $L_{t,r,f}=[x,y,z]$7. Retrieval is performed by cosine similarity, constrained by risk type, and the selected cases are injected into the prompt of an LLM that outputs one maneuver from a discrete set including Emergency Braking, Evasive Steering Left, and combined maneuvers. On a 1000-sample real-world near-miss dataset, Llama3.3 70B achieved 0.937 maneuver micro-accuracy in risk-unaware zero-shot prompting and 0.941 with risk-aware prompting; in the CBR setting, similarity-based retrieval consistently outperformed random selection across 1/3/5-shot settings for several models, and Llama3.3 with similarity-based 5-shot reached maneuver accuracy of approximately 0.94 with strong text metrics (Gan et al., 25 Jun 2025).

These human-facing PEMs differ from perception-error PEMs in purpose. They do not primarily degrade or reconstruct observations; rather, they bias planning toward psychologically acceptable trajectories or ground high-level decisions in retrieved prior cases.

6. Evaluation themes, limitations, and recurrent misconceptions

Across the literature, PEM augmentation is unified less by acronym expansion than by a repeated systems pattern: a baseline model is retained, and a PEM module is inserted to expose hidden uncertainty, encode neglected human factors, or improve representational richness. That pattern recurs in GP-augmented state estimation (Kullberg et al., 2020), longitudinal behavior simulation (Jami et al., 2022), perception-error injection (Piazzoni et al., 2023, Hanselmann et al., 2024), pedestrian representation (Holzbock et al., 2023), psychologically weighted MPC (Deng et al., 2023), and memory-augmented maneuver recommendation (Gan et al., 25 Jun 2025).

Several limitations are explicit. The pedestrian PEM does not explicitly model 3D position uncertainty, does not implement an explicit intent classifier, assumes average person height 1.7 m in refinement, and neglects translation during ego-motion compensation at urban speeds ≤ 50 km/h (Holzbock et al., 2023). The original HMM-based perception PEM does not generate false positives and depends on calibration choices such as the 10 m association threshold (Piazzoni et al., 2023). EMPERROR improves fidelity but does not explicitly model time, does not explicitly represent tracking identity switches, and is evaluated in open-loop rather than closed-loop simulation (Hanselmann et al., 2024). The psychological evaluation PEM is limited by a small field-test dataset, manually adjusted discriminant functions, and undisclosed exact potential-field and MPC horizon settings (Deng et al., 2023). The augmented crash-detection simulator focuses on longitudinal car-following and emergency braking, leaving lane changing and lateral decision-making outside the reported experiments (Jami et al., 2022).

Three misconceptions are especially persistent. First, richer perception metrics do not necessarily imply planner safety; both PEM and EMPERROR argue that object-centric metrics can miss downstream failure modes (Piazzoni et al., 2023, Hanselmann et al., 2024). Second, pose-aware environment modeling is not equivalent to intent recognition; the pedestrian PEM explicitly provides the substrate for downstream inference rather than the inference itself (Holzbock et al., 2023). Third, augmentation is not synonymous with end-to-end replacement; many of the strongest results arise from grey-box, modular, or weakly coupled integration, where the nominal model is preserved and only its failure modes are parameterized or corrected (Kullberg et al., 2020, He et al., 2021, Jami et al., 2022).

The broader implication is that “PEM-Augmented Driving Model” names a design philosophy rather than a fixed algorithm. Its concrete realization depends on which part of the autonomy stack is considered structurally deficient: observation, prediction, control, human-factors modeling, or high-level reasoning. The literature to date shows that these augmentations can reduce estimation bias, improve comfort-aware planning, enrich pedestrian representation, and expose planner fragility under realistic perception failures; it also shows that the validity of each PEM remains tightly coupled to its calibration regime, interface definition, and target deployment context.