Fine-Grained Dynamic Perception (FDP)

• FDP is a comprehensive framework that uses 3D part-guided data augmentation to simulate rare vehicle states and enhance perception for autonomous driving.
• It integrates a dual-backbone, multi-task Mask-RCNN architecture with multi-stream fusion to improve detection and segmentation accuracy by over 8%.
• The framework provides critical semantic cues by recognizing dynamic vehicle states and human-vehicle interactions, aiding in real-time risk assessment and planning.

The Fine-Grained Dynamic Perception (FDP) Framework defines a comprehensive methodology for vehicle perception that addresses the accurate parsing of 3D movable parts, the modeling of vehicle–human interactions, and the recognition of uncommon vehicle states using deep neural networks. This approach encompasses data-centric augmentation, multi-task network architectures, new datasets, and experimental protocols specifically tailored for autonomous driving scenarios with a focus on semantic safety-critical cues.

1. 3D Part-Guided Data Augmentation and Synthesis

Central to FDP is a fully automatic training data generation pipeline that leverages 3D vehicle models with annotated dynamic parts. Beginning with 2D/3D-aligned vehicle data (ApolloCar3D), each target vehicle is matched to its corresponding CAD model, which is semantically segmented into static (e.g., headlights, taillights) and dynamic/movable parts (four doors, trunk, bonnet).

For dynamic part modeling, motion axes and their respective transformation ranges are manually annotated. The pipeline then fits the 3D CAD model to content in real traffic images by calculating and aligning the camera’s six degrees of freedom (6-DoF pose). Once aligned, the system simulates various uncommon vehicle states (vehicles in uncommon states, VUS) by reconfiguring part geometries—such as opening doors or trunks—and reprojecting the modified 3D geometry onto the 2D image plane:

• For pixel $u$ with depth $D(u)$, the corresponding 3D point is $P = R_g^{-1} [D(u) K^{-1} \hat{u} - t_g]$;
• For a rotated part along a motion axis (rotation $R_o$, translation $t_o$): $$u' = \lfloor \pi(K[R_g (R_o(P - t_o) + t_o) + t_g]) \rfloor$$.

Hole-filling in projected regions utilizes a linear blend method, where color is interpolated using normalized weights among $K$ nearest neighbors, with post–bilateral filtering for smoothness.

This pipeline enables large-scale generation of synthetic images featuring not only typical but also rare vehicle configurations, including detailed annotation for aforementioned dynamic states and the simulation of human–vehicle interactions (VHI) in static images—crucial for training high-capacity DNNs.

2. Multi-Task and Multi-Stream Neural Network Design

For VUS parsing, FDP adopts a multi-task learning architecture based on Mask-RCNN, augmented to support dynamic part segmentation and state vector inference. Notably, it employs dual backbone networks (both ResNet50-FPN): the main backbone is pre-trained on vehicle-centric datasets (ApolloCar3D, CityScapes), the auxiliary backbone on the COCO dataset. Both are frozen during training to mitigate domain adaptation issues and synthetic bias overfitting.

The mask branch is extended to output $(K+1) m^2$-dimensional masks (for $K$ part classes plus one overall part segmentation channel). The loss for dynamic part segmentation ($L_{\text{part}}$) and state description ($L_{\text{state}}$)—the latter regressing binary abnormal/open indicators for each dynamic part—are both computed via pixel-wise sigmoid cross-entropy.

For VHI parsing, a dedicated multi-stream fusion network integrates:

• Human ROI features + 17-keypoint pose vector (34D)
• Vehicle features from VUS parser and state description
• Spatial relationship encoding via binary two-channel tensors, including human–vehicle bounding box overlaps and extra dynamic part highlighting
• Local CNN fusion of stream outputs for fine-grained classification of interaction semantics (e.g., “getting on/out,” “luggage handling”)

This network configuration enables the simultaneous learning and inference of detection, segmentation, part-level state, and interaction labels.

3. Dataset Construction and Evaluation Protocols

The VUS dataset, as established by FDP, contains 1441 real images and 1850 finely annotated vehicle instances, with comprehensive labeling for:

• 2D bounding boxes
• Instance segmentation
• Dynamic part segmentation (10 movable parts)
• State description of uncommon states (open doors/trunk, active signals, headlight flash, etc.)
• Optional human bounding boxes with precise VHI annotations

Experimental protocols measure improvement over Mask-RCNN and Faster-RCNN based baselines using metrics such as mAP (IoU), instance segmentation accuracy, and VHI detection. FDP achieves >8% margin improvement on both 2D detection and instance segmentation after synthetic data augmentation. Ablation studies indicate VHI stream inclusion yields further 3.2% detection and 1.9% state description gains; interaction mAP rises from 0.634 to 0.678 when compared with dedicated HOI detectors.

4. Semantic Implications for Autonomous Driving

FDP’s annotation and inference of part-level functional states and human-vehicle interactions provide vital semantic cues that safety-critical autonomous driving modules can utilize. For example, the recognition of doors being open or a person’s impending egress from a parked vehicle supplies actionable warnings for risk assessment and route planning, unattainable from bounding box-only detectors. FDP’s coverage of rare events dramatically increases system robustness in complex urban environments.

By enabling real-time parsing of rare but semantically rich vehicle states, FDP bridges a key gap in traffic scene understanding, which is essential for safe navigation amidst unpredictable maneuvers and interactions.

5. Practical Applications and System Integration

The fully automatic data augmentation pipeline and specialized multi-task networks described in FDP can be readily integrated into self-driving perception modules for risk detection and context-sensitive planning. This technique addresses annotated data scarcity in “uncommon” cases and alleviates manual labeling burdens, supporting continual improvement of autonomous vehicle algorithms as new edge cases arise in deployment.

Furthermore, the framework can be generalized to other mobile robotic systems facing semantic ambiguity due to dynamic object state changes or multi-agent interactions.

6. Limitations and Future Directions

While the FDP-generated dataset and neural architectures advance fine-grained perception substantially, several challenges persist:

• Synthetic-to-real domain gap remains, particularly for appearance and part occlusions.
• Manual motion axis annotation is necessary for the initial CAD part segmentation.
• The framework currently focuses on vehicle parts; extension to other dynamic environmental actors (bicycles, construction vehicles, etc.) will require further modeling.

Ongoing research might focus on unsupervised motion axis extraction, domain adaptation, and broader application across multimodal dynamic perception scenarios.

---

In summary, the Fine-Grained Dynamic Perception (FDP) Framework integrates automatic 3D part-guided augmentation, advanced multi-task/multi-stream network design, and robust dataset protocols to enable precise, semantic understanding of dynamic vehicle states and interactions. This significantly enhances the fidelity, coverage, and safety impact of autonomous agent visual perception in real-world environments (Lu et al., 2020).