4D Point Cloud Grounding with TPCNet

• The paper introduces TPCNet, a prompt-guided multi-modal pipeline that fuses LiDAR and 4D radar data for robust 3D bounding box prediction.
• It employs Bidirectional Agent Cross-Attention and Dynamic Gated Graph Fusion to adaptively integrate geometric and motion cues from synchronized sensor data.
• Experimental studies on Talk2Radar and Talk2Car demonstrate significant performance gains in accurately localizing moving objects in dynamic outdoor environments.

4D point cloud grounding defines the task of localizing objects in 3D space, over time, based on natural language prompts, using sensor data that encodes not just geometric (x,y,z) and intensity or reflectance information, but also temporal and motion cues (“4D”), particularly from mmWave radar and LiDAR. Central to recent progress is the fusion of multi-modal spatio-temporal point cloud data (LiDAR for static geometry, 4D radar for velocity and dynamic cues) under prompt guidance. This paradigm enables grounded object detection in dynamic, outdoor, real-world environments for autonomous agents, especially autonomous driving scenarios (Guan et al., 11 Mar 2025).

1. 4D Point Cloud Representation and Grounding Objective

The 4D point cloud grounding framework operates on two primary synchronized modalities per timestamp $t$: LiDAR, $$L_t = \{p_i^l = (x_i, y_i, z_i, i_i)\}_{i=1}^{N_l} \subset \mathbb{R}^4$$ with intensity $i_i$, and 4D mmWave radar, $$R_t = \{p_j^r = (x_j, y_j, z_j, r_j, v_j)\}_{j=1}^{N_r} \subset \mathbb{R}^5$$, where $r_j$ encodes radar cross-section and $v_j$ is radial velocity. Radar frames are typically accumulated over a temporal window to emphasize motion, resulting in $R = \bigcup_{t=1}^T R_t$.

Given a natural language prompt $P$ (e.g., “the car moving toward us on the right”) and synchronized $(L, R)$, the task is to output a 3D bounding box $B^* = \{p_c, \ell, w, h, \theta\}$ with center $$L_t = \{p_i^l = (x_i, y_i, z_i, i_i)\}_{i=1}^{N_l} \subset \mathbb{R}^4$$0, dimensions $$L_t = \{p_i^l = (x_i, y_i, z_i, i_i)\}_{i=1}^{N_l} \subset \mathbb{R}^4$$1, and orientation $$L_t = \{p_i^l = (x_i, y_i, z_i, i_i)\}_{i=1}^{N_l} \subset \mathbb{R}^4$$2, that grounds the object described by $$L_t = \{p_i^l = (x_i, y_i, z_i, i_i)\}_{i=1}^{N_l} \subset \mathbb{R}^4$$3.

The grounding is cast as minimizing a loss $$L_t = \{p_i^l = (x_i, y_i, z_i, i_i)\}_{i=1}^{N_l} \subset \mathbb{R}^4$$4, integrating a heatmap-based focal loss and $$L_t = \{p_i^l = (x_i, y_i, z_i, i_i)\}_{i=1}^{N_l} \subset \mathbb{R}^4$$5 terms for center, size, and rotation:

$$L_t = \{p_i^l = (x_i, y_i, z_i, i_i)\}_{i=1}^{N_l} \subset \mathbb{R}^4$$6

2. End-to-End Architecture: TPCNet

TPCNet implements a prompt-guided, multi-modal 4D point cloud grounding pipeline consisting of four principal stages:

• Input Encoding: Both LiDAR and radar point clouds are voxelized into pillar-based BEV representations, yielding multi-scale feature maps $$L_t = \{p_i^l = (x_i, y_i, z_i, i_i)\}_{i=1}^{N_l} \subset \mathbb{R}^4$$7, $$L_t = \{p_i^l = (x_i, y_i, z_i, i_i)\}_{i=1}^{N_l} \subset \mathbb{R}^4$$8. The text prompt $$L_t = \{p_i^l = (x_i, y_i, z_i, i_i)\}_{i=1}^{N_l} \subset \mathbb{R}^4$$9 is encoded by PointCLIP into token features $i_i$0.
• Two-Stage Heterogeneous Modal Adaptive Fusion: Bidirectional Agent Cross-Attention (BACA) fuses $i_i$1 and $i_i$2 at each scale, adaptively mediating information transfer between sensors according to the query structure.
• Dynamic Gated Graph Fusion (DGGF): DGGF incorporates language-guided gating and dynamically constructs spatial graphs over BEV cells, ensuring the region-of-interest localization is modulated by the prompt semantics.
• 3D Bounding Box Prediction: A feature pyramid network (FPN) merges fused BEV maps, and the C3D-RECHead module regresses the 3D bounding box, anchoring at the object edge nearest the ego-vehicle.

3. Two-Stage Heterogeneous Modal Adaptive Fusion (BACA)

BACA fuses geometry-rich LiDAR and motion/velocity-rich radar information using bidirectional cross-attention. Key elements include:

• Linear projections generate $i_i$3 for both LiDAR and radar features.
• Adaptive pooling yields compact “agent” features $i_i$4, $i_i$5.
• Stage 1: LiDAR queries radar to extract motion cues; attention is computed $i_i$6, lower than full cross-attention $i_i$7.
• Stage 2: Radar queries LiDAR to extract spatial/geometry structure.
• Outputs are summed to produce the fused token $i_i$8.

BACA adaptively prioritizes cues relevant to the prompt (e.g., radar velocity for motion, LiDAR depth for shape) and scales efficiently for dense BEV maps.

4. Dynamic Gated Graph Fusion (DGGF)

DGGF unifies BEV sensor features with linguistic context and constructs dynamic graphs to focus computation on semantically salient regions. Its components:

• Text–Point Gating: A channel-wise gate is derived from the prompt token embedding $i_i$9, modulating BEV features by $$R_t = \{p_j^r = (x_j, y_j, z_j, r_j, v_j)\}_{j=1}^{N_r} \subset \mathbb{R}^5$$0.
• Dynamic Graph Construction: BEV cells are graph nodes; adjacency is determined by the condition $$R_t = \{p_j^r = (x_j, y_j, z_j, r_j, v_j)\}_{j=1}^{N_r} \subset \mathbb{R}^5$$1 for features $$R_t = \{p_j^r = (x_j, y_j, z_j, r_j, v_j)\}_{j=1}^{N_r} \subset \mathbb{R}^5$$2, $$R_t = \{p_j^r = (x_j, y_j, z_j, r_j, v_j)\}_{j=1}^{N_r} \subset \mathbb{R}^5$$3, where $$R_t = \{p_j^r = (x_j, y_j, z_j, r_j, v_j)\}_{j=1}^{N_r} \subset \mathbb{R}^5$$4 are estimated across diagonal-flipped quadrant pairs and for variable axial offsets—eschewing fixed KNN neighborhoods.
• Dynamic Graph Convolution: Message passing and feature updates are performed via a dynamic convolution operation followed by aggregation and fusion with the original BEV map.

This architecture limits spurious connections in the graph, enhancing focus on prompt-relevant features.

5. C3D-RECHead: Corner-to-Edge Regression Head

C3D-RECHead anchors box regression at the object edge nearest to the ego-vehicle, differing from standard center-based models:

• Parameterization: The 3D box is described by $$R_t = \{p_j^r = (x_j, y_j, z_j, r_j, v_j)\}_{j=1}^{N_r} \subset \mathbb{R}^5$$5; the eight corners $$R_t = \{p_j^r = (x_j, y_j, z_j, r_j, v_j)\}_{j=1}^{N_r} \subset \mathbb{R}^5$$6 and all possible edges $$R_t = \{p_j^r = (x_j, y_j, z_j, r_j, v_j)\}_{j=1}^{N_r} \subset \mathbb{R}^5$$7 are computed.
• Edge Selection: The nearest edge $$R_t = \{p_j^r = (x_j, y_j, z_j, r_j, v_j)\}_{j=1}^{N_r} \subset \mathbb{R}^5$$8 is designated as the anchor.
• Prediction: Outputs include a heatmap $$R_t = \{p_j^r = (x_j, y_j, z_j, r_j, v_j)\}_{j=1}^{N_r} \subset \mathbb{R}^5$$9 centered at the nearest-edge corner, sub-voxel and height offsets, and residuals for dimensions and orientation.
• Loss: The total loss $r_j$0, with a focal loss $r_j$1 on the heatmap and $r_j$2.

Anchoring on the nearest edge improves regression fidelity for depth-sensitive prompts.

6. Implementation, Datasets, and Evaluation

TPCNet is evaluated on Talk2Radar and Talk2Car, which include over 27,000 natural language prompts referring to cars, pedestrians, and cyclists with comprehensive 3D bounding box annotations. Preprocessing employs voxelization (32 pillars for LiDAR, 10 for radar, BEV resolution 0.1\,m), multi-frame radar accumulation, and data augmentations (BEV random flip, global rotation ±45°, scaling [0.95, 1.05]).

Table: Excerpt of Core Quantitative Results (Guan et al., 11 Mar 2025)

Model	Sensors	mAP (EAA)	mAOS (EAA)	3D APₐ	3D AP_b
T-RadarNet	Radar₅	16.71	14.88	47.2	30.5
TPCNet	Radar₅+LiDAR	23.95	22.01	52.3	33.6

Ablation studies demonstrate each component's criticality: LiDAR+Radar fusion (mAP: 16.71→23.95), BACA cross-attention, DGGF dynamic graphs, and C3D-RECHead all yield measurable gains. Prompt-type results indicate that fusion substantially improves performance on velocity and motion-based expressions: for “velocity” prompts, fusion yields 40.8 mAP, outperforming single-modality alternatives.

7. Significance, Limitations, and Future Directions

Fusing 4D radar motion (velocity, reflection) with LiDAR’s high-fidelity 3D geometry demonstrably reduces false positives in motion-dependent grounding and enhances detection for distant objects and ambiguous linguistic prompts. BACA adaptively selects between modalities according to the prompt content, while DGGF’s dynamic spatial graphs yield finer focus and reduced clutter compared to static KNN.

A plausible implication is that this paradigm can be naturally extended to incorporate additional 4D sensor modalities (e.g., stereo-vision depth, event cameras), and applied to decentralized perception settings (robot navigation or V2X) by graph-based aggregation across vantage points.

TPCNet establishes a state-of-the-art benchmark for prompt-guided 4D point cloud grounding, integrating efficient fusion architectures and dynamic graph networks (Guan et al., 11 Mar 2025).