HumanRF Framework: RFMask for Silhouette Segmentation

• HumanRF Framework is an end-to-end system that uses FMCW radar and neural attention for high-resolution human silhouette segmentation under challenging visual conditions.
• It processes dual-plane RF heatmaps via CNN encoders and multi-head cross-view fusion to generate precise 2D mask predictions from millimeter-wave signals.
• Experimental evaluations show superior IoU and AP metrics over vision-only systems, particularly in low-light and occlusion scenarios.

The HumanRF framework, in the context of radio-based and cross-modal human sensing and segmentation, denotes a class of end-to-end systems that leverage frequency-modulated continuous-wave (FMCW) radar and neural attention-based architectures for high-resolution person silhouette segmentation, even in visually degraded scenarios. The precise term “HumanRF Framework” is operationalized as the “RFMask” architecture for human silhouette segmentation using millimeter-wave radio signals, as introduced and evaluated by Zhang et al. (Wu et al., 2022). This paradigm aims to overcome the fundamental limitations of conventional optical modalities in low-illumination and occlusion by harnessing RF’s penetrative and lighting-invariant properties.

1. System Architecture and Signal Processing

The HumanRF (RFMask) pipeline is a modular architecture structured in three sequential stages:

1. Signal-Processing Module: Dual FMCW radars (horizontal and vertical plane) collect raw complex-valued sweeps, which are then transformed into spatial domain heatmaps representing AoA (angle of arrival) and ToF (time of flight) per plane. Let $s_{k,m}(t)\in\mathbb{C}$ denote the baseband return for range sample $k$ on antenna $m$ at time $t$. The full multistatic imaging response at spatial coordinates $(x, y, z)$ and time $t$ is:

$$y(x,y,z,t) = \sum_{k=1}^K \sum_{m=1}^M s_{k,m}(t) \exp\left(j 2\pi \frac{d_m(x,y,z)}{\lambda_k}\right)$$

Dimensionality is reduced by slicing into two planes, yielding:

$$y_{hor}(x, y, t) = \sum_{k,m} s_{k,m}(t) \exp\left(j 2\pi \frac{d_m(x, y)}{\lambda_k}\right)$$

$$y_{ver}(y, z, t) = \sum_{k,m} s_{k,m}(t) \exp\left(j 2\pi \frac{d_m(y, z)}{\lambda_k}\right)$$

Static multipath is mitigated via frame-differencing: $$\hat{y}_{hor}(x, y, t) = y_{hor}(x, y, t) - y_{hor}(x, y, t-1)$$ (analogously for the vertical branch).

1. Human-Detection Module: Each branch utilizes a CNN-based encoder (ResNet with feature pyramid network) to process sequences of heatmaps, yielding feature maps $F_{hor}, F_{ver}\in\mathbb{R}^{C\times H\times W}$. Detection and region proposal are performed (including RPN on horizontal), with vertical range fixed per subject, defining a 3D cuboid aligned with the physical location of the human reflector. Feature crops are extracted using RoIAlign from both planes.
2. Attention-Based Mask Generation: The core innovation is multi-head (MH) fusion of the two feature crops. Inputs $H\in \mathbb{R}^{C\times h\times w_h}$ and $V\in \mathbb{R}^{C\times h\times w_v}$ are tokenized and concatenated, forming a sequence subject to four layers of MH self-attention. The cross-branch attention block explicitly attends between spatial regions aligned across the two perpendicular sensing planes, producing a fused feature representation $F_{fusion}$ that is upsampled by a convolutional decoder to yield the predicted binary mask.

2. Geometric Modeling and Projection

Projection from 3D detection cuboid to the imaging plane $P:Z=r$ is formulated as:

$$\begin{pmatrix} x_p \ y_p \ 1 \end{pmatrix} = \begin{bmatrix} r & 0 & p_x \ 0 & r & p_y \ 0 & 0 & 1 \end{bmatrix} \begin{pmatrix} x \ y \ z \end{pmatrix}$$

This enables all mask predictions to be composited and evaluated on a consistent 2D reference plane regardless of scene structure.

3. Training Objectives and Optimization

Objective function is the sum of detection and segmentation terms:

$$\mathcal{L} = \mathcal{L}_{detect} + \mathcal{L}_{mask}$$

The detection module employs a hybrid of binary cross-entropy and Smooth-L1 regression as in classical RPN training, while the mask branch optimizes a pixel-wise cross-entropy over predicted silhouette masks versus Mask R-CNN-derived ground truth:

$$\mathcal{L}_{mask} = -\frac{1}{N_{box}} \sum_{i=1}^{N_{box}} \sum_{u,v} [ m^*_{i}(u,v)\log \hat{m}_{i}(u,v) + (1-m^*_{i}(u,v))\log(1-\hat{m}_{i}(u,v)) ]$$

4. Experimental Protocols and Quantitative Performance

Hardware and Data: The HumanRF system was evaluated on a dataset comprising 804,760 RF frames (dual FMCW, 77–80 GHz, 1.23 GHz bandwidth, 192 virtual antenna array, 20 fps) and 402,380 multi-camera (13×Raspberry Pi v2) frames at 10 fps across ten environments. Rigid synchronization (NTP) and geometric calibration (Zhang’s method) enable pixel-accurate mask alignment.

Ground Truth: 2D silhouette masks were derived using Mask R-CNN on a reference view; full 3D keypoints obtained via OpenPose and triangulation inform occlusion and action labels.

Performance Metrics: Segmentation IoU (intersection-over-union), AP$_{50:95}$ (COCO-style), AP$_{50}$, AP$_{75}$, recall@IoU=0.5. Key results:

Backbone & T	AP$_{50:95}$	AP$_{50}$	AP$_{75}$	IoU (Single-P)	IoU (Multi-P)	IoU (Action)
ResNet-18, 4	0.586	0.966	0.678	0.681	0.682	0.681
ResNet-50,12	0.632	0.967	0.824	0.706	0.711	0.705

This demonstrates substantial gains with increased input sequence length, multi-head attention, and two-plane processing. Notably, RFMask maintains segmentation performance under $0\,\mathrm{lux}$ and severe occlusion, unlike vision-only systems (Wu et al., 2022).

5. Ablation Analyses and Comparative Insights

Single-Branch vs Dual-Branch: Using only the horizontal view results in lower IoUs ($\sim 0.634$ vs $0.681$ for single-person); simple concatenation yields minor improvement; MH fusion delivers a further $\sim$4–7 points.

Effect of Input Sequence Length: $T=12$ frames outperforms $T=4$ by $\sim$2–3 IoU points; further increases show diminishing returns.

Comparison to RFPose: HumanRF (RFMask) outperforms RFPose on all settings, especially in multi-person and complex action scenarios, attributed to superior cross-view attention and mask decoding.

6. Dataset Composition, Annotation Protocols, and Evaluation Procedure

• Scenarios: Random walks (single/multi-person), actions (sit, squat, etc.), structured and unstructured occlusion, and lighting from daylight to complete darkness.
• Annotation: Automated pipeline using Mask R-CNN and multiview triangulation for masks and 3D skeletons, respectively. Masks for occluded views are synthesized by reprojection.
• Precision–Recall Curves: Full curves illustrate robustness at multiple IoU thresholds ($0.5, 0.65, 0.75$).

7. Limitations, Open Challenges, and Prospective Extensions

Current Limitations: Spatial resolution is coarser than optical; fine limb details may be lost. Heavy multipath in cluttered environments degrades SNR, challenging the static suppression pipeline. Segmentation is limited to 2D imaging planes, not volumetric (3D) masks.

Future Work:

• Enhanced clutter cancellation (e.g., subspace projection, adaptive filtering)
• 3D voxel masks via learnable back-projectors
• Multi-radar sensor fusion for larger coverage areas
• Self-supervised/unsupervised RF pretraining to reduce dependency on optical labels
• Extension to activity recognition and tracking

A plausible implication is that HumanRF architectures can generalize far beyond conventional vision pipelines in adverse sensing conditions while retaining real-time performance and robust segmentation accuracy. These properties render the framework attractive for security, assistive, and smart-environment scenarios where lighting, occlusion, or privacy are critical factors (Wu et al., 2022).