HyperPointFormer: 3D Multimodal Segmentation

• The paper introduces a dual-branch Transformer architecture that fuses geometric and spectral modalities through bidirectional cross-attention.
• It employs a multi-scale design with local k-NN based Vector Self-Attention and Farther Point Sampling to enhance per-point classification.
• Quantitative evaluations on remote sensing datasets demonstrate improved mean F1, overall accuracy, and mIoU over previous methods.

HyperPointFormer is an end-to-end neural architecture for multimodal 3D semantic segmentation, designed to directly fuse geometric and spectral information from raw point clouds using a dual-branch Transformer encoder and bidirectional cross-attention. Addressing the limitations of 2D rasterization-based fusion, HyperPointFormer enables joint learning on the native 3D spatial-spectral domain, offering flexible per-point classification and the ability to generate both 3D and projected 2D predictions. The system is specifically tailored for remote sensing and geospatial applications where rich spectral (e.g., hyperspectral, RGB imagery) and geometric (lidar/photogrammetry) modalities are available (Rizaldy et al., 29 May 2025).

1. Dual-Branch Transformer Architecture

The core structure of HyperPointFormer comprises two parallel branches: one processes geometric inputs (e.g., lidar XYZ coordinates), while the other handles spectral features (e.g., hyperspectral bands, image-derived values). Each branch embeds its respective input into a shared $D$-dimensional latent space through a point-wise MLP, followed by a hierarchical encoder with four successive Vector Self-Attention (VSA) and downsampling blocks. Farthest Point Sampling (FPS) hierarchically reduces the number of points (from 4096 to 512 across scales), with each attention layer operating over local $k$-nearest neighborhood graphs ($k=16$). A mirrored decoder progressively upsamples via nearest-neighbor interpolation, reconstructing per-point predictions at the original resolution. Critically, after each encoder scale, a bidirectional CrossPointAttention (CPA) module fuses spectral and geometric representations, enabling information flow across modalities prior to deeper abstraction or upsampling (Rizaldy et al., 29 May 2025).

2. Cross-Attention Fusion Mechanism

Standard scaled dot-product attention, defining attention weights between queries $Q \in \mathbb{R}^{N \times D}$, keys $K \in \mathbb{R}^{M \times D}$, and values $V \in \mathbb{R}^{M \times D}$, computes $A = \mathrm{softmax}(Q K^\top/\sqrt{D})$ and $\mathrm{AttentionOutput} = AV$. In HyperPointFormer’s CPA, cross-modal fusion propagates information from branch $L$ (lidar) to branch $S$ (spectral) as:

$$Q_L = F_L W_Q^L, \quad K_S = F_S W_K^S, \quad V_S = F_S W_V^S$$

$$A^{CPA}_L = \text{softmax}( Q_L K_S^\top / \sqrt{D_e} ), \quad F^{CPA}_L = A^{CPA}_L V_S$$

The branch state is updated via a residual pathway and a learned scalar $\gamma$:

$\mathrm{CPA}(F_L, F_S) = F_L + \gamma F^{CPA}_L$

The same process is applied in the reverse direction; the bidirectional fused output at each scale is $$F_{\mathrm{fused}} = \mathrm{CPA}(F_L, F_S) + \mathrm{CPA}(F_S, F_L)$$. This procedure can be multi-headed, as in canonical Transformer architectures, with head-specific projections and final linear recombination. Unlike early or late fusion, this design interleaves cross-attention at each hierarchy, allowing mutual relevance estimation at multiple spatial supports (Rizaldy et al., 29 May 2025).

3. Feature Extraction and Hierarchical Processing

Each branch begins with a point-wise MLP mapping raw geometric ($X_L \in \mathbb{R}^{N\times 3}$) or spectral inputs ($X_S \in \mathbb{R}^{N \times b}$, $b$ bands) into $D$-dimensional activations:

$$F^{(0)}_L = \sigma(X_L W_0^L + b_0^L), \quad F^{(0)}_S = \sigma(X_S W_0^S + b_0^S)$$

Instead of explicit positional encodings, the VSA block models relative point relationships through $\beta(Q, K) = Q - K$, yielding noncommutative, translation-invariant attention. This approach captures local geometric structure and spectral associations by computing:

$$A_{VSA} = \mathrm{softmax}(\beta(Q, K)/\sqrt{D}), \quad F_{VSA} = A_{VSA} \odot V$$

where $\odot$ denotes channel-wise multiplication. Within each attention layer, FPS-based downsampling and $k$-NN grouping define neighborhoods over which self- and cross-attention are performed. Decoder upsampling is effected via nearest-neighbor interpolation; skip connections merge multi-scale fused features (Rizaldy et al., 29 May 2025).

4. Multi-Scale Fusion and Network Workflow

The network architecture comprises four encoding scales—S$_1$ (4096 points), S$_2$ (2048), S$_3$ (1024), S$_4$ (512)—with spectral and geometric modalities processed independently at each scale before fusing via CPA. This multi-scale hierarchical design ensures both locality (early layers sensitive to fine features like vegetation or small structures) and global context (deep layers integrating broad spatial regimes). The decoding path reverses this progression, restoring the original point count and merging hierarchical context through skip connections. The design enables fine-grained per-point classification and supports the projection of 3D predictions onto 2D maps (Rizaldy et al., 29 May 2025).

5. Learning Objective and Optimization

Semantic segmentation is supervised with a weighted cross-entropy loss over $N$ points and $C$ classes:

$$L = -\frac{1}{N} \sum_{i=1}^N \sum_{c=1}^C w_c y_{i,c} \log p_{i,c}$$

where $y_{i,c}$ is the ground-truth indicator, $p_{i,c}$ is the softmax-predicted class probability, and $w_c$ optionally addresses class imbalance. Optimization employs Adam ($\mathrm{lr}=1\mathrm{e}{-3}$), batch size 16, for 100–200 epochs contingent on dataset, selecting the checkpoint with maximal mean intersection-over-union (mIoU) for validation. No additional regularization beyond standard weight decay is stated (Rizaldy et al., 29 May 2025).

6. Datasets and Evaluation Benchmarks

HyperPointFormer was evaluated on several multimodal 3D remote sensing datasets:

Dataset	Modalities	Classes/Labels	Evaluation Block
DFC2018	Airborne lidar, hyperspectral, RGB	20 urban classes	75×75 m; 4096 pts
ISPRS Vaihingen	Lidar + NIR-Red-Green ortho-imagery	9 classes	30×30 m; 4096 pts
DFC2019	Airborne lidar only	5 classes	500×500 m tiles

On DFC2018, spectra and geometry are fused per 3D block via nearest-neighbor projection. ISPRS Vaihingen uses NN-based band assignment; DFC2019 employs geometry alone (Rizaldy et al., 29 May 2025).

7. Quantitative Results and Discussion

Performance metrics follow standard conventions: per-class Precision, Recall, F1, Overall Accuracy (OA), mean IoU, and Kappa statistic. Principal results include:

• DFC2018 (3D, 2-fold cross-validation): mean F1 = 57.4%, OA = 79.9%, mIoU = 46.9%. Outperforms Point Transformer (F1 = 54.8%) and KPConv+SE fusion (F1 ≈ 42.5%).
• DFC2018 (2D projection, test split): OA = 64.4%, Kappa = 62.2%, surpassing MFT (OA = 64.2%), Cross-HL (OA ≈ 63.3%), and top 2D contest solutions (OA ≈ 63%).
• ISPRS Vaihingen3D: OA = 83.0%, mean F1 = 71.0%; stronger than Point Transformer (OA = 81.3%, F1 = 68.7%) and spatiotemporal ConvNets (OA ≈ 83.4%, F1 ≈ 65.5%).
• DFC2019: OA = 98.2%, mIoU = 91.5%, marginally exceeding Point Transformer (OA = 98.1%) (Rizaldy et al., 29 May 2025).

8. Strengths, Limitations, and Application Domains

Key advantages include:

• Fully 3D fusion avoids rasterization artifacts and leverages both geometric and spectral features at all scales.
• Bidirectional cross-attention enables mutual relevance estimation of modalities at every stage, enhancing inter-modal synergies.
• Multi-scale hierarchy efficiently captures both fine detail and large-scale context through attention and FPS sampling.
• Prediction flexibility: 3D outputs can be projected onto 2D maps, but not vice versa, providing broader utility.

Notable limitations:

• Spectral-to-3D projection can misalign under dense canopy or occlusion; learned inpainting could ameliorate this issue.
• Local-only attention (via $k$-NN) requires downsampling to cover global context. Incorporating hybrid local/global attention may improve large-scale patterns.
• Class imbalance, especially for rare categories (e.g., water, crosswalks), remains challenging despite class weighting.

Primary application areas include urban land-use mapping, infrastructure monitoring, vegetation and forestry analysis, mineral exploration in 3D outcrop datasets, and any context involving the joint segmentation of 3D point clouds with multi-spectral or hyperspectral imagery (Rizaldy et al., 29 May 2025).