Dual-Modal Driver Model (Dual-DM)

Updated 7 July 2026

Dual-Modal Driver Model (Dual-DM) is a family of models that employ non-adversarial and adversarial driving modalities to generate evolving, realistic yet challenging scenarios for autonomous driving tests.
It utilizes a two-stage training process with multi-agent reinforcement learning to balance traffic realism with increased adversarial interactions and diversified safety-critical scenarios.
Dual-DM principles extend to multi-view and multimodal sensing applications in distraction and fatigue detection as well as mode control, highlighting the importance of tailored sensor fusion and decision architectures.

Searching arXiv for the cited Dual-DM and closely related papers. Dual-Modal Driver Model (Dual-DM) most explicitly denotes a driver model with two driving modalities—non-adversarial and adversarial—trained by multi-agent reinforcement learning (MARL) to control background vehicles in evolving scenario generation for autonomous-driving testing (Wu et al., 4 Aug 2025). In adjacent driving research, closely related dual formulations appear in three distinct forms: stacked driver-facing and road-facing video for distraction detection (Dontoh et al., 23 Dec 2025), computer vision plus physiological signal analysis for fatigue detection (K et al., 23 May 2025), and mode switching between autonomous driving, human driving, and emergency fallback in level-3/4 automated driving systems (Insua et al., 2021). The term therefore refers not to a single canonical architecture, but to a family of dual-structured models in which the pairing may involve behavior modalities, sensing modalities, visual viewpoints, or driving-control modes.

1. Terminology and research scope

The most direct use of the name Dual-DM appears in evolving scenario generation, where the model determines the driving maneuvers of background vehicles and is explicitly trained to switch between realistic traffic behavior and adversarial interaction near the subject vehicle. Related work uses the same dual logic for different purposes: context-aware distraction recognition, fatigue estimation through multimodal sensing, and human–automation mode management.

Research line	What is “dual”	Primary function
(Wu et al., 4 Aug 2025)	non-adversarial and adversarial driving modalities	evolving scenario generation
(Dontoh et al., 23 Dec 2025)	driver-facing and road-facing video	distraction detection
(K et al., 23 May 2025)	computer vision and physiological signals	fatigue detection
(Insua et al., 2021)	AUTON and MANUAL, with EMERG fallback	driving mode management

A recurring misconception is that all such systems are interchangeable because they are “dual.” The literature does not support that simplification. In one line of work, duality refers to policy modalities for background vehicles; in another, it refers to sensor fusion; in another, to multi-view video input; and in another, to control arbitration between automation and the human driver. This suggests that the term should be interpreted with careful attention to the problem formulation and the locus at which the two information streams or modes are combined.

In evolving scenario generation, Dual-DM is designed to address a specific tension: generated traffic should preserve high fidelity, while still producing high adversariality / test efficiency and high complexity and diversity. The model therefore assigns each background vehicle one of two driving modalities. The non-adversarial driving modality is used when a background vehicle is far from the SV and is trained for realistic behavior, including car-following, lane changing, on-ramp merging, speed control, and maintaining traffic legality and plausibility. The adversarial driving modality activates when a background vehicle enters the adversarial driving area, defined as 45 meters longitudinally around the SV—22.5 m in front and 22.5 m behind—and Five lanes laterally: the SV’s lane plus two lanes on each side (Wu et al., 4 Aug 2025).

The adversarial modality is not defined as direct collision seeking. Instead, background vehicles progressively compress the SV’s drivable space by reducing safe forward space, reducing lane-changing opportunities, and creating risky or difficult decisions through lane occupation, speed adjustment, and cooperative positioning. If multiple background vehicles are inside the adversarial area, they can act cooperatively. The paper emphasizes that adversariality is implemented in a controlled, realistic way and that background vehicles should not directly collide with the SV as a strategy.

Behavioral constraints are used to suppress irrational or illegal maneuvers. The listed constraints are: No left lane change in the leftmost lane; No right lane change in the rightmost lane; No lane change left if vehicles are within 10 m in front and behind in the left lane; No lane change right if vehicles are within 10 m in front and behind in the right lane; No lane changes from main road to merging ramps. These constraints are intended to preserve traffic legality, realism, and meaningful adversarial interactions.

This construction distinguishes Dual-DM from purely naturalistic models and purely adversarial models. A plausible implication is that the model’s “duality” is not merely architectural; it is operational, since the policy changes according to the vehicle’s relation to the SV and to the local interaction geometry.

3. Learning architecture, observation design, and reward structure

Dual-DM is trained in two stages. Stage 1: Non-adversarial pretraining uses a single-vehicle DRL method from the authors’ previous work, with an Actor-Critic style DRL algorithm and hierarchical training based on level-k theory. Stage 2: MARL adversarial training extends the pretrained model to the adversarial setting with the SUT-in-the-loop: the SUT is connected to the training environment, and background vehicles learn to interact adversarially against that specific SUT. The algorithm in both stages is TD3, i.e., Twin Delayed Deep Deterministic Policy Gradient (Wu et al., 4 Aug 2025).

The training setup is described as CTDE-style, but with an important modification: all agents share the same Actor network and Critic network, and a shared cooperative reward is used. Stage 1 uses 1 trained model vehicle and 12 other vehicles in the environment. Stage 2 uses 12 trained model vehicles and 1 SUT vehicle. The simulation engine is CARLA, and vehicle types are randomly selected from Audi A2, Dodge Challenger, and Lincoln MKZ. The Stackelberg model is used as the SUT during adversarial training.

The observation space is an ego-centric grid-based observation. Each background vehicle observes 100 m ahead, 20 m behind, and five lanes laterally. The environment is discretized into 5 × 3.5 m cells, forming a 25 × 5 grid, and the full observation tensor is 25 × 5 × 5. The five layers are the Spatial drivability layer $P$ , Relative longitudinal velocity layer $\Delta v_s$ , Relative lateral velocity layer $\Delta v_t$ , Relative yaw angle layer $\Delta h$ , and Adversarial driving determination factor layer $M$ . The layer $P$ takes $P=1$ if a grid cell contains a vehicle and $P=0$ otherwise. The layer $M$ encodes the SV’s position relative to the background vehicle when the background vehicle is inside the adversarial area.

The action space is continuous in both longitudinal and lateral dimensions:

$\mathcal{A}_{long} \in [-1,1], \qquad \mathcal{A}_{lat} \in [-1,1].$

The longitudinal action controls acceleration or braking. The lateral action is converted into

$\Delta v_s$ 0

with thresholds $\Delta v_s$ 1, $\Delta v_s$ 2, and otherwise $\Delta v_s$ 3.

Reward design combines an individual reward $\Delta v_s$ 4 for realism and driving competence with a cooperative reward $\Delta v_s$ 5 for adversarial space compression. The cooperative term is

$\Delta v_s$ 6

where $\Delta v_s$ 7 and $\Delta v_s$ 8 compress the SV’s left and right lane-changing space and $\Delta v_s$ 9 compresses forward space in the SV’s lane. Reward sharing is defined as

$\Delta v_t$ 0

where $\Delta v_t$ 1 is the number of background vehicles in the adversarial area. The reported individual-reward parameters are collision reward $\Delta v_t$ 2, $\Delta v_t$ 3, $\Delta v_t$ 4, $\Delta v_t$ 5, $\Delta v_t$ 6, $\Delta v_t$ 7, $\Delta v_t$ 8, illegal-driving reward $\Delta v_t$ 9, with $\Delta h$ 0 and $\Delta h$ 1 for the cooperative reward terms.

The reported TD3 hyperparameters are training rounds: 90,000, replay buffer size: 40,000, actor learning rate: 0.0001, critic learning rate: 0.0001, batch size: 10, training iterations per round: 2, soft update coefficient: 0.095, and discount factor: 0.9. Reward curves show convergence around 60,000 training rounds for level-1 non-adversarial training, level-2 non-adversarial training, and adversarial training.

4. Scenario generation workflow and empirical performance

Dual-DM is deployed in a continuous simulated traffic environment in which the SV is controlled by the SUT and 15 BVs are controlled by an NPC model, for 16 vehicles total. The map is a highway-like 4-lane road of approximately 2400 m, including straight sections, curves, and an on-ramp. Two interfaces are defined: an Observation signal interface, which provides map information and surrounding vehicle states, and a Driving behavior signal interface, which outputs decision or control signals. If the model outputs strategic decisions, they are converted into control actions via PID controllers. The lateral steering signal is

$\Delta h$ 2

with $\Delta h$ 3 and $\Delta h$ 4 (Wu et al., 4 Aug 2025).

The simulation runs for 500 rounds. A scenario is considered safety-critical if a collision occurs between SV and a BV or if the SV’s TTC < 0.5 s. For each such event, the 3.5 s trajectory preceding the event is extracted, including position, velocity, and acceleration for the SV and all BVs.

Evaluation uses four axes: fidelity, efficiency, complexity, and diversity. Fidelity is benchmarked against the HighD dataset using Jensen-Shannon divergence on vehicle velocity and TTC under lane-change behavior. Dual-DM yields velocity: 0.0074 and TTC under lane-change: 0.1323, both below 0.15, corresponding to more than 85% similarity. Test efficiency is measured by $\Delta h$ 5, with results after 500 rounds of Nilsson: 129 scenarios, $\Delta h$ 6, Social-DRL: 145 scenarios, $\Delta h$ 7, and Dual-DM: 429 scenarios, $\Delta h$ 8. The paper reports that Dual-DM improves efficiency by about 195% over the baselines. Complexity results are Nilsson: 0.34, Social-DRL: 0.40, and Dual-DM: 0.45, making Dual-DM 32.35% higher than Nilsson and 12.5% higher than Social-DRL.

The count of safety-critical scenarios is also reported in crash and near-crash form: Nilsson: 62 crash scenarios and 67 near-crash scenarios, total 129; Social-DRL: 104 crash scenarios and 41 near-crash scenarios, total 145; Dual-DM: 206 crash scenarios and 223 near-crash scenarios, total 429. Diversity is evaluated only for Dual-DM through statistical analysis of cooperative reward outcomes and case studies. Among the 429 critical scenarios, 414 have positive cooperative rewards. The number of background vehicles participating in adversarial interactions is 1 BV: 64.49%, 2 BVs: 30.68%, and 3 BVs: 4.83%. Within the adversarial area divided into 10 subregions, the paper reports 2-BV scenarios: 127 scenarios, 23 distinct patterns and 3-BV scenarios: 20 scenarios, 16 distinct patterns.

These results position Dual-DM as a mechanism for generating complex, diverse and strong interactive safety-critical scenarios without reported degradation in scenario fidelity (>85% similarity to real-world scenarios) and complexity (complexity metric: 0.45, +32.35% and +12.5% over two baselines), while improving the efficiency of generating safety-critical scenarios (efficiency metric: 0.86, +195% over two baselines).

5. Dual-view and dual-sensing driver models in monitoring systems

A separate but closely related research line studies whether driver models should integrate multiple sensing streams rather than rely on a single driver-facing view. In naturalistic driving recordings from blackbox sensors installed in personal vehicles, synchronized driver-facing and road-facing video streams were split into 5-second clips, sampled at 25 FPS, giving 125 frames per clip. Two input configurations were compared: driver-only, implemented by cropping the lower half of the stacked video, and stacked dual-view, in which the road-facing view is on top and the driver-facing view is on the bottom. The clips were labeled into six classes: hands off the wheel, head turned, listening to radio or music, normal driving, personal grooming, and stopped. The evaluated architectures were SlowFast-R50, X3D-M, and SlowOnly-R50 (Dontoh et al., 23 Dec 2025).

The quantitative result is architecture-dependent rather than uniformly favorable to dual input. X3D-M was the best overall performer with driver-only test accuracy = 55.3%, while stacked dual-view produced a 2.0% decrease. SlowFast-R50 showed an accuracy dropped by 7.2%, from 0.53 accuracy in driver-only form to 0.46 accuracy in dual-view form. SlowOnly-R50 showed the strongest positive result, improving from 0.43 accuracy to 0.53 accuracy, i.e., a 9.8% improvement. The paper attributes the degradation in SlowFast to representational conflicts, motion interference, and ambiguous signals introduced by naïve early fusion. The main conclusion is explicit: simply adding visual context is not sufficient and may lead to interference unless the architecture is specifically designed to support multi-view integration.

Another dual formulation combines computer vision with physiological signal analysis for fatigue detection. The architecture comprises a Computer vision module, a Physiological signal module, and a Multi-modal fusion module. The vision branch performs face detection and tracking, eye-closure analysis, micro-expression analysis, yawn detection, and head-nodding / fatigue-related facial behavior analysis. A stated handcrafted feature is the eye aspect ratio (EAR):

$\Delta h$ 9

The physiological branch uses non-intrusive wearable sensors and processes EEG, EOG, and HRV, with noise filtering, normalization, and artifact removal, including a 0.5–40 Hz bandpass filter and z-score normalization

$M$ 0

Feature preprocessing and reduction include adaptive histogram equalization and Gaussian filtering, mutual information and recursive feature elimination, and PCA retaining 95% of the variance. Fusion occurs at both the feature and decision levels:

$M$ 1

$M$ 2

where $M$ 3 and $M$ 4 are determined by signal quality (K et al., 23 May 2025).

The dataset includes 50 participants, age: 32.4 ± 10.2 years, 60% male, 40% female, each completing two 2-hour simulated driving sessions, yielding over 200 hours of synchronized visual and physiological data. Sampling rates are 30 FPS for visual data and 256 Hz for physiological signals, with synchronization using NTP and maximum drift: 10 ms. The split is 70% training, 15% validation, and 15% test, corresponding to Hours: 140 / 30 / 30. Training uses 5-fold cross-validation, early stopping, learning rate scheduling, and Bayesian optimization.

Reported performance is Computer Vision Only: 92% accuracy, 91% precision, 90% recall, 90.5% F1-score; Physio Signal Only: 94% accuracy, 93% precision, 92% recall, 92.5% F1-score; Fusion (Ours): 98% accuracy, 97% precision, 98% recall, 97.5% F1-score. The abstract also states 98% accuracy in controlled environments and 94% in real-world conditions. Efficiency metrics are average latency: 85 ms per frame, CPU utilization: 28%, and memory usage: 180 MB. This line of work presents a different sense of Dual-DM: not policy modality switching, but confidence-weighted multimodal sensing for fatigue estimation.

6. Dual-mode control management, misconceptions, and limitations

A further related but conceptually distinct line of work treats driving as a mode-management problem in level-3 and level-4 automated driving systems. The framework manages three modes: $M$ 5: AUTON, $M$ 6: MANUAL, and $M$ 7: EMERG. The ADS forecasts whether future conditions will force a mode change and compares the expected utility of autonomous and manual driving,

$M$ 8

issuing a Request-to-Intervene (RtI) when manual mode is preferable and using EMERG when the ODD boundary is too close or the driver is not a safe takeover candidate (Insua et al., 2021).

The architecture comprises five probabilistic components: ODD monitoring, environment monitoring, driver state monitoring, trajectory planning, and Driver intervention performance assessment (DIPA). It uses Bayesian forecasting plus decision analysis, and it is explicitly organized around the management-by-exception principle: routine monitoring and forecasting continue without interruption until forecasted conditions cross warning thresholds, at which point warnings and mode reassessment are triggered. The workflow is summarized as: Read sensors; Forecast environment $M$ 9 steps ahead; Forecast driver state $P$ 0 steps ahead; Compute trajectory plan; Assess driving modes using expected utility; Issue warnings if probabilities cross thresholds; Manage driving mode transition; Perform DIPA if an RtI occurred.

Taken together, these research lines clarify several misconceptions. First, dual-modal does not have a single stable meaning across the driving literature. Second, dual input does not imply automatic gains: stacked driver-facing and road-facing video can improve one architecture and degrade another. Third, multimodal sensing gains depend on the fusion strategy: the fatigue-detection model uses both feature-level fusion and decision-level fusion, whereas the distraction-detection study relies on simple stacked early fusion and reports representational conflicts in SlowFast. Fourth, dual-mode control management is not equivalent to multimodal perception; it is a probabilistic decision architecture over AUTON, MANUAL, and EMERG.

The limitations reported across the literature are correspondingly heterogeneous. In Dual-DM scenario generation, future work is directed toward more diverse road topologies and more advanced SUTs. In dual-sensing fatigue detection, limitations include extreme lighting, sensor misplacement, individual physiological variability, the fact that wearable sensors may affect long-term comfort, and the possibility that the dataset may not fully represent all cultures and environments. In dual-view distraction detection, the central limitation is that naive stacking is early fusion and may introduce interference unless the architecture explicitly learns which view matters when, how to align temporal information, and how to suppress irrelevant view-specific noise. A plausible implication is that future uses of the term Dual-DM will continue to diverge unless authors specify whether duality refers to sensing, viewpoint, behavior policy, or human–automation control mode.