RadHARSimulator V2: Video-to-Radar HAR

• The paper introduces a video-to-Doppler generator that maps monocular video frames into radar observables, preserving natural motion, occlusion, and timing.
• It integrates advanced computer vision for detection, tracking, and 3D pose recovery with LFM echo simulation and through-the-wall propagation modeling.
• Experimental results show high PSNR in RTM/DTM and robust SPNet classification, confirming its practical impact on radar-based human activity recognition research.

RadHARSimulator V2 is an end-to-end simulator for radar-based human activity recognition (HAR) that converts ordinary recorded video into simulated radar micro-Doppler spectra and downstream recognition features. It couples a computer vision stack for detection, tracking, and pose recovery with a radar signal chain that generates echoes in free-space and through-the-wall (TTW) settings, forms range-time and Doppler-time representations, denoises them, extracts ridge features, and supports classification through a hybrid parallel–serial neural network termed SPNet. The system is positioned as an alternative to model-based simulators and motion-capture-driven pipelines by leveraging real video to preserve natural pose variation, occlusion patterns, and motion timing while retaining explicit radar-domain processing and TTW propagation effects (Gao, 12 Nov 2025).

1. Scope, problem setting, and research context

Radar HAR has long faced a simulator-design tension between realism and controllability. Existing software is described as being developed from analytical models or motion-captured data, which yields limited flexibility; RadHARSimulator V2 addresses that limitation by directly generating Doppler spectra from recorded video footage (Gao, 12 Nov 2025). The stated contribution is not merely a visual front-end attached to a classifier, but a full “video to Doppler generator” that maps monocular video frames to physically modeled radar observables and then to HAR outputs.

The simulator is intended to solve several concrete problems. It improves realism and diversity by using real video to capture natural pose variations, timing, occlusions, and motion complexity beyond simplified articulated models. It improves flexibility because it does not require motion labs or motion-capture suits and can operate on open-source datasets and in-the-wild videos. It explicitly supports TTW HAR through attenuation, delay, and multipath modeling, and it integrates DnCNN denoising into both range-time and Doppler-time processing stages (Gao, 12 Nov 2025).

In the literature landscape, this places RadHARSimulator V2 between two established simulator families. Relative to WiFi passive-radar simulators such as SimHumalator, which synthesize IEEE 802.11g transmissions and use marker-based motion capture plus primitive scattering models, V2 replaces motion-capture dependence with video-driven pose recovery and operates through a different end-to-end sensing pipeline (Vishwakarma et al., 2021). Relative to high-fidelity mmWave ray-tracing simulators for hand-pose imaging, which focus on photogrammetry-based meshes, coherent SFCW synthesis, and 3D backprojection, RadHARSimulator V2 targets full-body human activity micro-Doppler generation rather than hand imaging (Bräunig et al., 2023). This suggests that V2 occupies a distinct methodological niche: vision-driven kinematic realism combined with radar-domain feature synthesis for activity recognition rather than purely kinematic, MoCap, or ray-tracing paradigms.

2. End-to-end system organization

The simulator accepts monocular video frames $I_{t_f} \in \mathbb{R}^{H \times W \times C}$, supports a variable number of targets, and handles occlusions through tracker counters. In TTW operation, the scene additionally includes wall-slab parameters such as position, size, $\epsilon_r$, and $\tan\delta$. The complete pipeline proceeds from detection and tracking, through 2D and 3D pose estimation, temporal smoothing, radar-timing resampling, echo simulation, range-time map (RTM) formation, Doppler-time map (DTM) formation, ridge extraction, and activity classification (Gao, 12 Nov 2025).

Module	Main methods	Primary outputs
Computer vision	RTMDet, GNN tracking, HRNet, nearest matching, Kalman filtering	Tracks, 2D poses, smoothed 3D poses
Radar	Savitzky–Golay, LFM echo simulation, FFT, MTI, STFT, DnCNN, MLE	RTM, DTM, ridge curves
HAR	SPNet	Activity predictions

Several internal data structures are explicitly defined. Tracks are denoted $Tr_{m,t_f}$ with state

$$st_m = [B c_x, B c_y, B w, B h, \dot{(B c_x)}, \dot{(B c_y)}, \dot{(B w)}, \dot{(B h)}]^T.$$

The 2D pose is represented as $P_{2D} \in \mathbb{R}^{K \times 2}$, while the 3D pose library is stored as $Pool_{3D} \in \mathbb{R}^{N \times K \times 3}$ and $Pool_{2D} \in \mathbb{R}^{N \times K \times 2}$. Smoothed metric trajectories are denoted $P_{meters}(j,p,t_f)$ and the radar-resampled trajectories $P''(t_m)$. Radar-domain outputs include $\epsilon_r$0, $\epsilon_r$1, $\epsilon_r$2, and ridge curves $\epsilon_r$3 (Gao, 12 Nov 2025).

The architecture is explicitly modular in three senses. First, the computer-vision subpipeline is separable from the radar module. Second, free-space and TTW propagation share the same pose-driven source trajectories but different echo-generation stages. Third, HAR can consume RTM, DTM, channel-merged inputs, or ridge images. A common misconception is that the simulator directly predicts labels from video; in fact, the recognition stage is downstream of a substantial radar synthesis and processing stack, and the paper emphasizes intermediate radar-domain products rather than only end-task accuracy (Gao, 12 Nov 2025).

3. Computer vision pipeline

The visual front-end begins with RTMDet for person detection and a global nearest neighbor (GNN) tracker for identity association. RTMDet is described as using a CSPNeXt backbone with large-kernel depthwise separable convolutions, a top-down/bottom-up neck with multi-scale fusion, and a detection head with shared weights across scales and independent batch normalization per scale. Its loss is

$\epsilon_r$4

For tracking, the state transition is

$\epsilon_r$5

and data association uses the IoU-derived cost

$\epsilon_r$6

with assignment optimized by the Hungarian algorithm under one-to-one constraints (Gao, 12 Nov 2025).

Pose estimation is split into 2D recovery and 3D retrieval. HRNet maintains a high-resolution subnet throughout, uses parallel multi-resolution branches with periodic fusion, and predicts per-joint Gaussian heatmaps. Its loss is

$\epsilon_r$7

The model estimates 17 joints, with 14 used downstream for 3D matching. The 3D stage then performs nearest matching against a Human3.6M-derived library with $\epsilon_r$8 poses:

$\epsilon_r$9

Metric scaling is based on human height,

$\tan\delta$0

and body depth is scaled using $\tan\delta$1 (Gao, 12 Nov 2025).

After retrieval, the 3D pose is aligned to radar coordinates. The pose is hip-centered, the $\tan\delta$2 axis is inverted to match radar convention, the body is grounded through the ankle-based offset

$\tan\delta$3

and global translation is accumulated from hip displacement. Temporal smoothing then applies a joint-wise Kalman filter in each coordinate with

$\tan\delta$4

The process and measurement equations are

$\tan\delta$5

with standard prediction and update recursions. The paper states that $\tan\delta$6 and $\tan\delta$7 are chosen empirically. This computer-vision design implies that the fidelity of the final radar signatures is partly constrained by monocular 2D detection quality, nearest-neighbor 3D retrieval, and the assumptions embedded in height and depth scaling (Gao, 12 Nov 2025).

4. Radar signal generation, TTW modeling, and spectral products

The radar module first resamples the smoothed pose trajectories from video timestamps $\tan\delta$8 to radar pulse repetition instants $\tan\delta$9 by linear interpolation:

$Tr_{m,t_f}$0

for $Tr_{m,t_f}$1. A Savitzky–Golay filter is then applied independently to each coordinate time series. For a window length $Tr_{m,t_f}$2 and polynomial order $Tr_{m,t_f}$3, the fitted polynomial

$Tr_{m,t_f}$4

minimizes

$Tr_{m,t_f}$5

and the center value produces $Tr_{m,t_f}$6 (Gao, 12 Nov 2025).

Echo generation is based on linear frequency-modulated (LFM) signaling. The transmit waveform is

$Tr_{m,t_f}$7

$Tr_{m,t_f}$8

$Tr_{m,t_f}$9

For a single joint,

$$st_m = [B c_x, B c_y, B w, B h, \dot{(B c_x)}, \dot{(B c_y)}, \dot{(B w)}, \dot{(B h)}]^T.$$0

and the dechirped intermediate-frequency signal is

$$st_m = [B c_x, B c_y, B w, B h, \dot{(B c_x)}, \dot{(B c_y)}, \dot{(B w)}, \dot{(B h)}]^T.$$1

With bistatic transmitter and receiver positions $$st_m = [B c_x, B c_y, B w, B h, \dot{(B c_x)}, \dot{(B c_y)}, \dot{(B w)}, \dot{(B h)}]^T.$$2 and $$st_m = [B c_x, B c_y, B w, B h, \dot{(B c_x)}, \dot{(B c_y)}, \dot{(B w)}, \dot{(B h)}]^T.$$3 and a joint position $$st_m = [B c_x, B c_y, B w, B h, \dot{(B c_x)}, \dot{(B c_y)}, \dot{(B w)}, \dot{(B h)}]^T.$$4,

$$st_m = [B c_x, B c_y, B w, B h, \dot{(B c_x)}, \dot{(B c_y)}, \dot{(B w)}, \dot{(B h)}]^T.$$5

The Doppler relation is described as $$st_m = [B c_x, B c_y, B w, B h, \dot{(B c_x)}, \dot{(B c_y)}, \dot{(B w)}, \dot{(B h)}]^T.$$6 for monostatic settings, with geometry-dependent projection for bistatic radar (Gao, 12 Nov 2025).

TTW propagation adds wall transmission, reflection, attenuation, and refractive delay. The wall material is modeled through

$$st_m = [B c_x, B c_y, B w, B h, \dot{(B c_x)}, \dot{(B c_y)}, \dot{(B w)}, \dot{(B h)}]^T.$$7

$$st_m = [B c_x, B c_y, B w, B h, \dot{(B c_x)}, \dot{(B c_y)}, \dot{(B w)}, \dot{(B h)}]^T.$$8

At normal incidence,

$$st_m = [B c_x, B c_y, B w, B h, \dot{(B c_x)}, \dot{(B c_y)}, \dot{(B w)}, \dot{(B h)}]^T.$$9

with the corresponding reverse-direction coefficients. The direct TTW two-way delay is

$P_{2D} \in \mathbb{R}^{K \times 2}$0

and the attenuation factor is

$P_{2D} \in \mathbb{R}^{K \times 2}$1

The paper further includes internal wall multipath and reflection multipath using the mirror method, with image-source delays such as $P_{2D} \in \mathbb{R}^{K \times 2}$2, $P_{2D} \in \mathbb{R}^{K \times 2}$3, and $P_{2D} \in \mathbb{R}^{K \times 2}$4. The total IF signal is the coherent superposition

$P_{2D} \in \mathbb{R}^{K \times 2}$5

where $P_{2D} \in \mathbb{R}^{K \times 2}$6 is complex white Gaussian noise (Gao, 12 Nov 2025).

Signal processing then produces several radar-domain representations. RTM is obtained by FFT-based pulse compression,

$P_{2D} \in \mathbb{R}^{K \times 2}$7

followed by an MTI two-pulse canceller,

$P_{2D} \in \mathbb{R}^{K \times 2}$8

DnCNN denoising is applied to RTM and later to DTM. The DTM is formed after range summation,

$P_{2D} \in \mathbb{R}^{K \times 2}$9

and short-time Fourier transform:

$Pool_{3D} \in \mathbb{R}^{N \times K \times 3}$0

Ridge extraction then selects the top-$Pool_{3D} \in \mathbb{R}^{N \times K \times 3}$1 local maxima in each time frame,

$Pool_{3D} \in \mathbb{R}^{N \times K \times 3}$2

using the maximum local energy method (Gao, 12 Nov 2025).

The simulator reports two concrete radar configurations. In free-space it uses $Pool_{3D} \in \mathbb{R}^{N \times K \times 3}$3 GHz, $Pool_{3D} \in \mathbb{R}^{N \times K \times 3}$4 GHz, $Pool_{3D} \in \mathbb{R}^{N \times K \times 3}$5 Hz, $Pool_{3D} \in \mathbb{R}^{N \times K \times 3}$6 MHz, and $Pool_{3D} \in \mathbb{R}^{N \times K \times 3}$7, with TX and RX at $Pool_{3D} \in \mathbb{R}^{N \times K \times 3}$8 m and $Pool_{3D} \in \mathbb{R}^{N \times K \times 3}$9 m, antenna gain $Pool_{2D} \in \mathbb{R}^{N \times K \times 2}$0 dBi, isolation $Pool_{2D} \in \mathbb{R}^{N \times K \times 2}$1 dB, and SNR $Pool_{2D} \in \mathbb{R}^{N \times K \times 2}$2 dB. In TTW mode it uses $Pool_{2D} \in \mathbb{R}^{N \times K \times 2}$3 GHz, $Pool_{2D} \in \mathbb{R}^{N \times K \times 2}$4 GHz, $Pool_{2D} \in \mathbb{R}^{N \times K \times 2}$5 Hz, $Pool_{2D} \in \mathbb{R}^{N \times K \times 2}$6 MHz, and the same $Pool_{2D} \in \mathbb{R}^{N \times K \times 2}$7, with a wall centered at $Pool_{2D} \in \mathbb{R}^{N \times K \times 2}$8 m, dimensions $Pool_{2D} \in \mathbb{R}^{N \times K \times 2}$9, $P_{meters}(j,p,t_f)$0, and $P_{meters}(j,p,t_f)$1 (Gao, 12 Nov 2025).

5. HAR model, training protocol, and reported results

The recognition module is SPNet, a hybrid parallel–serial convolutional architecture designed for radar HAR. It accepts RTM, DTM, channel-merged R/DTM, or ridge-curves rendered as images. The network begins with a $P_{meters}(j,p,t_f)$2 convolution plus batch normalization, then uses dense blocks with multiple $P_{meters}(j,p,t_f)$3 convolutions, dense connections, $P_{meters}(j,p,t_f)$4 channel-compression and fusion layers, residual additions, and three repeated serial extraction stages. The final stage applies another $P_{meters}(j,p,t_f)$5 convolution, concatenation, global average pooling, a fully connected layer, and softmax (Gao, 12 Nov 2025).

Training uses categorical cross-entropy,

$P_{meters}(j,p,t_f)$6

with batch size $P_{meters}(j,p,t_f)$7, epochs $P_{meters}(j,p,t_f)$8, initial learning rate $P_{meters}(j,p,t_f)$9, Adam, L2 regularization, best-epoch checkpointing, $P''(t_m)$0 training samples, $P''(t_m)$1 validation samples, and an NVIDIA RTX 3060 OC GPU. The paper reports Matlab R2025b for training and validation. DnCNN itself is configured as a residual denoiser with an initial $P''(t_m)$2 convolution with $P''(t_m)$3 channels and ReLU, followed by $P''(t_m)$4 repeated $P''(t_m)$5 convolution + batch normalization + ReLU blocks, then a final $P''(t_m)$6 convolution to a single-channel output. Its loss is

$P''(t_m)$7

The paper attributes part of SPNet’s performance to the quality improvement introduced by these denoising stages (Gao, 12 Nov 2025).

The reported results combine spectral-fidelity measurements, classifier behavior, robustness analyses, and ablations. On OSSet, average RTM PSNR is approximately $P''(t_m)$8 dB in free-space and $P''(t_m)$9 dB in TTW; processed RTM PSNR is approximately $\epsilon_r$00 dB in free-space and $\epsilon_r$01 dB in TTW; DTM PSNR is approximately $\epsilon_r$02 dB in free-space and $\epsilon_r$03 dB in TTW. On RWSet, RTM PSNR increases to approximately $\epsilon_r$04 dB in free-space and $\epsilon_r$05 dB in TTW; processed RTM PSNR is approximately $\epsilon_r$06 dB in free-space and $\epsilon_r$07 dB in TTW; DTM PSNR is approximately $\epsilon_r$08 dB in free-space and $\epsilon_r$09 dB in TTW (Gao, 12 Nov 2025).

At the recognition level, confusion matrices reportedly show that SPNet achieves the highest validation accuracy among compared methods on RWSet and is the only one exceeding $\epsilon_r$10 there, while on OSSet it attains near-top performance and is second to TTW-ADFE-Net in some categories. The paper further states that SPNet is robust under SNR degradation: as SNR decreases by $\epsilon_r$11 dB on OSSet, its validation accuracy declines the least among compared methods. On RWSet at $\epsilon_r$12 dB, MC-GCLU temporarily exceeds SPNet, but SPNet remains robust as SNR drops further (Gao, 12 Nov 2025).

The ablation studies are central to interpreting the simulator. RTMDet+GNN tracking yields the best PSNR, while ACF and MOSSE degrade tracking quality and downstream spectra. HRNet with nearest matching and Kalman filtering outperforms CPM and TCN alternatives for pose-driven synthesis. Removing wall attenuation or multipath significantly lowers TTW spectra fidelity. DnCNN outperforms Perona–Malik and KSVD denoising on processed RTM and DTM, improving micro-Doppler clarity and HAR accuracy. These findings support the claim that V2’s performance is not reducible to a single component such as SPNet; rather, the paper presents it as an integrated visual-to-radar pipeline in which detection, pose estimation, TTW propagation, denoising, and classification are mutually coupled (Gao, 12 Nov 2025).

6. Limitations, comparisons, and future directions

Several limitations are stated explicitly. The 3D pose stage is monocular and nearest-matching based, so it lacks forward-looking depth inference and can be sensitive to 2D pose inaccuracies. Ridge extraction can fail when DTM signatures are discontinuous, because maximum local energy ridge tracking may break. The wall model is a homogeneous slab with planar reflections; complex indoor multipath and oblique incidence are not fully modeled. Adapting the simulator to very different radar systems, including UWB impulse radar or MIMO angle-Doppler sensing, would require extension of the waveform and array models (Gao, 12 Nov 2025).

These limitations clarify what the simulator is and is not. It is not a full electromagnetic solver, nor is it a general-purpose radar scene renderer. The TTW model is physically grounded through attenuation coefficients, refractive delay, and mirror-method multipath, but it remains a reduced scene model. Likewise, the vision front-end yields realistic kinematic diversity from ordinary video, but the 3D reconstruction stage depends on library matching rather than end-to-end learned geometry. A plausible implication is that the strongest use case is controlled synthesis of micro-Doppler and HAR benchmarks where natural motion variability is more important than full-scene wave propagation fidelity.

The paper situates V2 against prior HAR simulators by emphasizing three differences. Compared with RadHARSimulator V1, which used a fixed 13-scatterer kinematic model for 12 activities and simplified motion and multipath, V2 replaces analytical motion modeling with video-driven pose and adds TTW multipath, dual-stage DnCNN, and ridge features (Gao, 12 Nov 2025). Compared with SimHumalator, which uses WiFi passive radar, IEEE 802.11g waveform generation, motion-capture-derived animation, and primitive body scattering, V2 leverages recorded video and a different radar-generation chain, thereby improving applicability to diverse natural motions and TTW scenarios (Vishwakarma et al., 2021). Relative to realistic radar ray-tracing simulators for hand-pose imaging, which prioritize near-field coherent modeling, photogrammetry-based mesh realism, and 3D backprojection at 72–82 GHz, V2 remains focused on activity-level micro-Doppler rather than high-resolution hand imaging (Bräunig et al., 2023).

Future work in the paper points toward improved monocular 3D pose estimation with learned geometry or temporal constraints, more robust ridge extraction through graph-based tracking or dynamic programming, and richer TTW propagation models with angle-dependent Fresnel coefficients and 3D ray tracing. The paper also suggests that sequence models, including temporal CNNs, RNNs, Transformers, or graph neural networks operating on ridge sequences, may improve interpretability and generalization. Open-source code is reported at the GitHub repository linked in the paper, but the license is not specified there in the paper text, so repository-level terms remain the authoritative source for reuse conditions (Gao, 12 Nov 2025).