---
title: Dual-Modal Driver Model (Dual-DM)
url: https://www.emergentmind.com/topics/dual-modal-driver-model-dual-dm
type: topic
---

# Dual-Modal Driver Model (Dual-DM)

Searching arXiv for the cited Dual-DM and closely related papers.
Dual-Modal Driver Model (Dual-DM) most explicitly denotes a driver model with two driving modalities—**non-adversarial** and **adversarial**—trained by **multi-agent reinforcement learning (MARL)** to control background vehicles in evolving scenario generation for autonomous-driving testing [2508.02027]. In adjacent driving research, closely related dual formulations appear in three distinct forms: **stacked driver-facing and road-facing video** for distraction detection [2512.20025], **computer vision plus physiological signal analysis** for fatigue detection [2505.17392], and **mode switching between autonomous driving, human driving, and emergency fallback** in level-3/4 automated driving systems [2107.00280]. The term therefore refers not to a single canonical architecture, but to a family of dual-structured models in which the pairing may involve behavior modalities, sensing modalities, visual viewpoints, or driving-control modes.

## 1. Terminology and research scope

The most direct use of the name **Dual-DM** appears in evolving scenario generation, where the model determines the driving maneuvers of background vehicles and is explicitly trained to switch between realistic traffic behavior and adversarial interaction near the subject vehicle. Related work uses the same dual logic for different purposes: context-aware distraction recognition, fatigue estimation through multimodal sensing, and human–automation mode management.

| Research line | What is “dual” | Primary function |
|---|---|---|
| [2508.02027] | non-adversarial and adversarial driving modalities | evolving scenario generation |
| [2512.20025] | driver-facing and road-facing video | distraction detection |
| [2505.17392] | computer vision and physiological signals | fatigue detection |
| [2107.00280] | AUTON and MANUAL, with EMERG fallback | driving mode management |

A recurring misconception is that all such systems are interchangeable because they are “dual.” The literature does not support that simplification. In one line of work, duality refers to **policy modalities** for background vehicles; in another, it refers to **sensor fusion**; in another, to **multi-view video input**; and in another, to **control arbitration** between automation and the human driver. This suggests that the term should be interpreted with careful attention to the problem formulation and the locus at which the two information streams or modes are combined.

## 2. Dual-DM as a dual-modal background-vehicle policy

In evolving scenario generation, Dual-DM is designed to address a specific tension: generated traffic should preserve **high fidelity**, while still producing **high adversariality / test efficiency** and **high complexity and diversity**. The model therefore assigns each background vehicle one of two driving modalities. The **non-adversarial driving modality** is used when a background vehicle is **far from the SV** and is trained for realistic behavior, including **car-following, lane changing, on-ramp merging, speed control, and maintaining traffic legality and plausibility**. The **adversarial driving modality** activates when a background vehicle enters the **adversarial driving area**, defined as **45 meters longitudinally** around the SV—**22.5 m in front** and **22.5 m behind**—and **Five lanes laterally**: the SV’s lane plus two lanes on each side [2508.02027].

The adversarial modality is not defined as direct collision seeking. Instead, background vehicles **progressively compress the SV’s drivable space** by reducing safe forward space, reducing lane-changing opportunities, and creating risky or difficult decisions through **lane occupation, speed adjustment, and cooperative positioning**. If multiple background vehicles are inside the adversarial area, they can act **cooperatively**. The paper emphasizes that adversariality is implemented in a **controlled, realistic way** and that background vehicles should **not directly collide** with the SV as a strategy.

Behavioral constraints are used to suppress irrational or illegal maneuvers. The listed constraints are: **No left lane change in the leftmost lane; No right lane change in the rightmost lane; No lane change left if vehicles are within 10 m in front and behind in the left lane; No lane change right if vehicles are within 10 m in front and behind in the right lane; No lane changes from main road to merging ramps**. These constraints are intended to preserve **traffic legality**, **realism**, and **meaningful adversarial interactions**.

This construction distinguishes Dual-DM from purely naturalistic models and purely adversarial models. A plausible implication is that the model’s “duality” is not merely architectural; it is operational, since the policy changes according to the vehicle’s relation to the SV and to the local interaction geometry.

## 3. Learning architecture, observation design, and reward structure

Dual-DM is trained in **two stages**. **Stage 1: Non-adversarial pretraining** uses a **single-vehicle DRL** method from the authors’ previous work, with an **Actor-Critic** style DRL algorithm and hierarchical training based on **level-k theory**. **Stage 2: MARL adversarial training** extends the pretrained model to the adversarial setting with the **SUT-in-the-loop**: the SUT is connected to the training environment, and background vehicles learn to interact adversarially against that specific SUT. The algorithm in both stages is **TD3**, i.e., **Twin Delayed Deep Deterministic Policy Gradient** [2508.02027].

The training setup is described as **CTDE-style**, but with an important modification: all agents **share the same Actor network and Critic network**, and a **shared cooperative reward** is used. Stage 1 uses **1 trained model vehicle** and **12 other vehicles in the environment**. Stage 2 uses **12 trained model vehicles** and **1 SUT vehicle**. The simulation engine is **CARLA**, and vehicle types are randomly selected from **Audi A2**, **Dodge Challenger**, and **Lincoln MKZ**. The **Stackelberg model** is used as the SUT during adversarial training.

The observation space is an **ego-centric grid-based observation**. Each background vehicle observes **100 m ahead**, **20 m behind**, and **five lanes** laterally. The environment is discretized into **5 × 3.5 m** cells, forming a **25 × 5** grid, and the full observation tensor is **25 × 5 × 5**. The five layers are the **Spatial drivability layer** \(P\), **Relative longitudinal velocity layer** \(\Delta v_s\), **Relative lateral velocity layer** \(\Delta v_t\), **Relative yaw angle layer** \(\Delta h\), and **Adversarial driving determination factor layer** \(M\). The layer \(P\) takes \(P=1\) if a grid cell contains a vehicle and \(P=0\) otherwise. The layer \(M\) encodes the SV’s position relative to the background vehicle when the background vehicle is inside the adversarial area.

The action space is continuous in both longitudinal and lateral dimensions:
$$
\mathcal{A}_{long} \in [-1,1], \qquad \mathcal{A}_{lat} \in [-1,1].
$$
The longitudinal action controls acceleration or braking. The lateral action is converted into
$$
\mathcal{D}_{LaneChange}\in \{-1,0,1\},
$$
with thresholds \(\mathcal{A}_{lat} > 0.5 \Rightarrow \mathcal{D}_{LaneChange}=1\), \(\mathcal{A}_{lat} < -0.5 \Rightarrow \mathcal{D}_{LaneChange}=-1\), and otherwise \(0\).

Reward design combines an **individual reward** \(r_{ind}\) for realism and driving competence with a **cooperative reward** \(r_{coop}\) for adversarial space compression. The cooperative term is
$$
r_{coop} = r_l + r_r + r_f,
$$
where \(r_l\) and \(r_r\) compress the SV’s left and right lane-changing space and \(r_f\) compresses forward space in the SV’s lane. Reward sharing is defined as
$$
r_i =
\begin{cases}
r_{ind}, & \text{non-adversarial driving modality} \\
r_{ind} + \frac{r_{coop}}{n}, & \text{adversarial driving modality}
\end{cases}
$$
where \(n\) is the number of background vehicles in the adversarial area. The reported individual-reward parameters are **collision reward \(r_c = -3\)**, **\(\mu = 1\)**, **\(\lambda = 10\)**, **\(d_{desired} = 15\)**, **\(\delta_1 = 8000\)**, **\(\delta_2 = 3000\)**, **\(p_{lc} = -0.1\)**, **illegal-driving reward \(r_{rv} = -0.3\)**, with **\(\eta_1 = 400\)** and **\(\eta_2 = 4000\)** for the cooperative reward terms.

The reported TD3 hyperparameters are **training rounds: 90,000**, **replay buffer size: 40,000**, **actor learning rate: 0.0001**, **critic learning rate: 0.0001**, **batch size: 10**, **training iterations per round: 2**, **soft update coefficient: 0.095**, and **discount factor: 0.9**. Reward curves show convergence around **60,000 training rounds** for **level-1 non-adversarial training**, **level-2 non-adversarial training**, and **adversarial training**.

## 4. Scenario generation workflow and empirical performance

Dual-DM is deployed in a **continuous simulated traffic environment** in which the **SV** is controlled by the SUT and **15 BVs** are controlled by an NPC model, for **16 vehicles total**. The map is a **highway-like 4-lane road** of approximately **2400 m**, including **straight sections**, **curves**, and **an on-ramp**. Two interfaces are defined: an **Observation signal interface**, which provides map information and surrounding vehicle states, and a **Driving behavior signal interface**, which outputs decision or control signals. If the model outputs strategic decisions, they are converted into control actions via **PID controllers**. The lateral steering signal is
$$
\mathcal{U}_{steering} = \text{CLIP} [\alpha \text{PID}(d) + \beta \text{PID}(e), s_{min}, s_{max}]
$$
with **\(\alpha = 1.5\)** and **\(\beta = 1.0\)** [2508.02027].

The simulation runs for **500 rounds**. A scenario is considered safety-critical if **a collision occurs between SV and a BV** or if the SV’s **TTC < 0.5 s**. For each such event, the **3.5 s trajectory** preceding the event is extracted, including **position, velocity, and acceleration for the SV and all BVs**.

Evaluation uses four axes: **fidelity**, **efficiency**, **complexity**, and **diversity**. Fidelity is benchmarked against the **HighD dataset** using **Jensen-Shannon divergence** on **vehicle velocity** and **TTC under lane-change behavior**. Dual-DM yields **velocity: 0.0074** and **TTC under lane-change: 0.1323**, both below **0.15**, corresponding to **more than 85% similarity**. Test efficiency is measured by \(E\), with results after 500 rounds of **Nilsson: 129 scenarios, \(E=0.26\)**, **Social-DRL: 145 scenarios, \(E=0.29\)**, and **Dual-DM: 429 scenarios, \(E=0.86\)**. The paper reports that Dual-DM improves efficiency by **about 195% over the baselines**. Complexity results are **Nilsson: 0.34**, **Social-DRL: 0.40**, and **Dual-DM: 0.45**, making Dual-DM **32.35% higher than Nilsson** and **12.5% higher than Social-DRL**.

The count of safety-critical scenarios is also reported in crash and near-crash form: **Nilsson: 62 crash scenarios and 67 near-crash scenarios, total 129**; **Social-DRL: 104 crash scenarios and 41 near-crash scenarios, total 145**; **Dual-DM: 206 crash scenarios and 223 near-crash scenarios, total 429**. Diversity is evaluated only for Dual-DM through **statistical analysis of cooperative reward outcomes** and **case studies**. Among the **429** critical scenarios, **414** have positive cooperative rewards. The number of background vehicles participating in adversarial interactions is **1 BV: 64.49%**, **2 BVs: 30.68%**, and **3 BVs: 4.83%**. Within the **adversarial area** divided into **10 subregions**, the paper reports **2-BV scenarios: 127 scenarios, 23 distinct patterns** and **3-BV scenarios: 20 scenarios, 16 distinct patterns**.

These results position Dual-DM as a mechanism for generating **complex, diverse and strong interactive safety-critical scenarios** without reported degradation in **scenario fidelity (>85% similarity to real-world scenarios)** and **complexity (complexity metric: 0.45, +32.35% and +12.5% over two baselines)**, while improving **the efficiency of generating safety-critical scenarios (efficiency metric: 0.86, +195% over two baselines)**.

## 5. Dual-view and dual-sensing driver models in monitoring systems

A separate but closely related research line studies whether driver models should integrate multiple sensing streams rather than rely on a single driver-facing view. In **naturalistic driving recordings** from blackbox sensors installed in personal vehicles, synchronized **driver-facing** and **road-facing** video streams were split into **5-second clips**, sampled at **25 FPS**, giving **125 frames per clip**. Two input configurations were compared: **driver-only**, implemented by cropping the lower half of the stacked video, and **stacked dual-view**, in which the **road-facing view is on top** and the **driver-facing view is on the bottom**. The clips were labeled into six classes: **hands off the wheel**, **head turned**, **listening to radio or music**, **normal driving**, **personal grooming**, and **stopped**. The evaluated architectures were **SlowFast-R50**, **X3D-M**, and **SlowOnly-R50** [2512.20025].

The quantitative result is architecture-dependent rather than uniformly favorable to dual input. **X3D-M** was the **best overall performer** with **driver-only test accuracy = 55.3%**, while stacked dual-view produced a **2.0% decrease**. **SlowFast-R50** showed an **accuracy dropped by 7.2%**, from **0.53 accuracy** in driver-only form to **0.46 accuracy** in dual-view form. **SlowOnly-R50** showed the strongest positive result, improving from **0.43 accuracy** to **0.53 accuracy**, i.e., a **9.8% improvement**. The paper attributes the degradation in SlowFast to **representational conflicts**, **motion interference**, and ambiguous signals introduced by naïve early fusion. The main conclusion is explicit: **simply adding visual context is not sufficient and may lead to interference unless the architecture is specifically designed to support multi-view integration**.

Another dual formulation combines **computer vision** with **physiological signal analysis** for fatigue detection. The architecture comprises a **Computer vision module**, a **Physiological signal module**, and a **Multi-modal fusion module**. The vision branch performs **face detection and tracking**, **eye-closure analysis**, **micro-expression analysis**, **yawn detection**, and **head-nodding / fatigue-related facial behavior analysis**. A stated handcrafted feature is the **eye aspect ratio (EAR)**:
$$
EAR = \frac{\|p_2 - p_6\| + \|p_3 - p_5\|}{2\|p_1 - p_4\|}.
$$
The physiological branch uses non-intrusive wearable sensors and processes **EEG**, **EOG**, and **HRV**, with **noise filtering**, **normalization**, and **artifact removal**, including a **0.5–40 Hz bandpass filter** and **z-score normalization**
$$
x_{norm} = \frac{x - \mu}{\sigma}.
$$
Feature preprocessing and reduction include **adaptive histogram equalization and Gaussian filtering**, **mutual information and recursive feature elimination**, and **PCA retaining 95% of the variance**. Fusion occurs at both the feature and decision levels:
$$
F_{fusion} = [F_{vision} ; F_{physio}],
$$
$$
S_{fusion} = w_{v} S_{vision} + w_{p} S_{physio},
$$
where \(w_v\) and \(w_p\) are determined by signal quality [2505.17392].

The dataset includes **50 participants**, **age: 32.4 ± 10.2 years**, **60% male**, **40% female**, each completing **two 2-hour simulated driving sessions**, yielding **over 200 hours of synchronized visual and physiological data**. Sampling rates are **30 FPS** for visual data and **256 Hz** for physiological signals, with synchronization using **NTP** and **maximum drift: 10 ms**. The split is **70% training**, **15% validation**, and **15% test**, corresponding to **Hours: 140 / 30 / 30**. Training uses **5-fold cross-validation**, **early stopping**, **learning rate scheduling**, and **Bayesian optimization**.

Reported performance is **Computer Vision Only: 92% accuracy, 91% precision, 90% recall, 90.5% F1-score**; **Physio Signal Only: 94% accuracy, 93% precision, 92% recall, 92.5% F1-score**; **Fusion (Ours): 98% accuracy, 97% precision, 98% recall, 97.5% F1-score**. The abstract also states **98% accuracy in controlled environments** and **94% in real-world conditions**. Efficiency metrics are **average latency: 85 ms per frame**, **CPU utilization: 28%**, and **memory usage: 180 MB**. This line of work presents a different sense of Dual-DM: not policy modality switching, but **confidence-weighted multimodal sensing** for fatigue estimation.

## 6. Dual-mode control management, misconceptions, and limitations

A further related but conceptually distinct line of work treats driving as a **mode-management problem** in level-3 and level-4 automated driving systems. The framework manages three modes: \(d_0\): **AUTON**, \(d_1\): **MANUAL**, and \(d_2\): **EMERG**. The ADS forecasts whether future conditions will force a mode change and compares the expected utility of autonomous and manual driving,
$$
\psi(d_0) \quad \text{vs.} \quad \psi(d_1),
$$
issuing a **Request-to-Intervene (RtI)** when manual mode is preferable and using **EMERG** when the ODD boundary is too close or the driver is not a safe takeover candidate [2107.00280].

The architecture comprises five probabilistic components: **ODD monitoring**, **environment monitoring**, **driver state monitoring**, **trajectory planning**, and **Driver intervention performance assessment (DIPA)**. It uses **Bayesian forecasting plus decision analysis**, and it is explicitly organized around the **management-by-exception** principle: routine monitoring and forecasting continue without interruption until forecasted conditions cross warning thresholds, at which point warnings and mode reassessment are triggered. The workflow is summarized as: **Read sensors; Forecast environment \(k\) steps ahead; Forecast driver state \(k\) steps ahead; Compute trajectory plan; Assess driving modes using expected utility; Issue warnings if probabilities cross thresholds; Manage driving mode transition; Perform DIPA if an RtI occurred**.

Taken together, these research lines clarify several misconceptions. First, **dual-modal** does not have a single stable meaning across the driving literature. Second, dual input does not imply automatic gains: stacked **driver-facing and road-facing** video can improve one architecture and degrade another. Third, multimodal sensing gains depend on the **fusion strategy**: the fatigue-detection model uses both **feature-level fusion** and **decision-level fusion**, whereas the distraction-detection study relies on simple stacked **early fusion** and reports **representational conflicts** in SlowFast. Fourth, dual-mode control management is not equivalent to multimodal perception; it is a probabilistic decision architecture over **AUTON**, **MANUAL**, and **EMERG**.

The limitations reported across the literature are correspondingly heterogeneous. In Dual-DM scenario generation, future work is directed toward **more diverse road topologies** and **more advanced SUTs**. In dual-sensing fatigue detection, limitations include **extreme lighting**, **sensor misplacement**, **individual physiological variability**, the fact that **wearable sensors may affect long-term comfort**, and the possibility that the dataset **may not fully represent all cultures and environments**. In dual-view distraction detection, the central limitation is that **naive stacking is early fusion** and may introduce interference unless the architecture explicitly learns **which view matters when**, **how to align temporal information**, and **how to suppress irrelevant view-specific noise**. A plausible implication is that future uses of the term **Dual-DM** will continue to diverge unless authors specify whether duality refers to sensing, viewpoint, behavior policy, or human–automation control mode.

Source: https://www.emergentmind.com/topics/dual-modal-driver-model-dual-dm