Multi-Headed Cloud Transform (MHCT)

• Multi-Headed Cloud Transform (MHCT) is a versatile architectural block for deep point cloud processing, supporting both discriminative and generative tasks.
• It employs multiple parallel heads to project point features onto low-dimensional grids using bilinear/trilinear splatting and convolution, ensuring efficient long-range feature mixing.
• MHCT achieves state-of-the-art performance by integrating spatial transformer concepts with optimized 2D/3D convolutions, offering scalability and accuracy.

The Multi-Headed Cloud Transform (MHCT) is a versatile architectural block for deep point cloud processing, designed to serve both discriminative tasks such as segmentation and classification, and generative tasks including point cloud inpainting and image-based point cloud reconstruction. MHCT combines spatial transformer concepts and multi-view convolutional strategies with the computational efficiency typical of standard 2D and 3D convolutions. Its defining feature is the use of multiple parallel heads that project point features into low-dimensional grids, convolving these grids independently before recombining them to update the per-point feature representation. This paradigm offers dense long-range feature mixing with controllable computational complexity, supporting state-of-the-art accuracy on a variety of point cloud tasks (Mazur et al., 2020).

1. Block Architecture and Processing Pipeline

Each MHCT block operates on point clouds $X = \{x_i \in \mathbb{R}^f\}_{i=1}^N$ with corresponding spatial positions $P = \{p_i \in \mathbb{R}^3\}_{i=1}^N$. The pipeline adopted by each of the $H$ heads consists of the following key stages:

1. Per-head Key and Value Prediction:
   • For each head $h$ and point $i$, a "key" $k^h_i \in [0,1]^d$ ($d \in \{2,3\}$), and a "value" $v^h_i \in \mathbb{R}^{c_{\rm in}}$ are predicted.
   • The key is formed by learned per-head residual transformations in SE(3), followed by an affine map and sigmoid, specifically:

   $$p_i' = T^h\bigl(p_i + d^h(p_i)\bigr), \quad k^h_i = \sigma(\pi^h(p_i')) \in [0,1]^d,$$

   where $d^h$ is a small MLP, $P = \{p_i \in \mathbb{R}^3\}_{i=1}^N$0 is rigid, and $P = \{p_i \in \mathbb{R}^3\}_{i=1}^N$1 is affine. - The value is normalized as:

   $P = \{p_i \in \mathbb{R}^3\}_{i=1}^N$2

2. Rasterization (Splatting):

   • Each head maintains a zero-initialized grid $P = \{p_i \in \mathbb{R}^3\}_{i=1}^N$3.
   • For each point, the key is interpreted as continuous $P = \{p_i \in \mathbb{R}^3\}_{i=1}^N$4 coordinates and mapped to grid indices. A $P = \{p_i \in \mathbb{R}^3\}_{i=1}^N$5-linear interpolation (bilinear/trilinear) is used to splat $P = \{p_i \in \mathbb{R}^3\}_{i=1}^N$6 onto grid corners using weights $P = \{p_i \in \mathbb{R}^3\}_{i=1}^N$7, aggregating by $P = \{p_i \in \mathbb{R}^3\}_{i=1}^N$8 operation.
3. Convolutional Propagation:
   • A small standard $P = \{p_i \in \mathbb{R}^3\}_{i=1}^N$9-dimensional convolutional network (e.g., $H$0 kernel) processes $H$1, producing $H$2 with potentially intermediate normalization and activation.
4. De-Rasterization (Slicing):
   • Points are reassigned features by interpolating from their grid neighborhoods:

   $H$3

• The result is mapped back to the target feature dimension via normalization, activation, and affine mapping.

1. Multi-Head Fusion and Residual Update:
   • At each point, outputs from all heads are summed and normalized. A final affine transformation and residual connection produce the updated feature:

   $H$4

2. Rasterization and Splatting Schemes

Rasterization is a core component of MHCT, realized per head via $H$5-linear (bilinear or trilinear) splatting of per-point values onto dense grids. Each key $H$6 is scaled to grid coordinates and "splatted" with weight $H$7 to $H$8 neighboring grid cells. The update to grid $H$9 is performed via $h$0 instead of mean or sum:

$h$1

This choice of $h$2 aggregation enhances gradient flow and empirical accuracy relative to alternative aggregations.

3. Convolutional Propagation and De-Rasterization

After assembly on grids, each head applies a local dense convolutional network, typically with $h$3 (2D) or $h$4 (3D) kernels, to propagate features. These convolutions are spatially local, parameter-shared, and leverage highly optimized routines, maintaining grid resolution throughout.

Slicing interpolates convolved grid features back to the irregular points using the same bilinear/trilinear weights as splatting. The sliced feature undergoes normalization (BatchNorm, InstanceNorm, or AdaIN), nonlinearity (ReLU), and an affine projection to the desired output dimension.

4. Hyperparameters, Architectural Choices, and Ablation

Table: Head Types and Typical Hyperparameters

Head Type	Dimensionality ($h$5)	Grid Resolution ($h$6)	Channels ($h$7)
Planar (2D)	2	64–128	16 → 16
Volumetric (3D)	3	16–32	32 → 32

Significant hyperparameters include:

• Number of heads $h$8: Higher $h$9 permits diverse projections and receptive fields; values such as $i$0 (16 planar, 16 volumetric) are typical.
• Grid resolution $i$1: Higher $i$2 yields finer resolution at increased computational/memory cost.
• Kernel size: Standard $i$3 (2D) or $i$4 (3D); deeper stack increases representational power.
• Channel dimensions: Larger channels increase block capacity and cost.

Ablation shows that $i$5 aggregation, learnable per-point keys, and mixing of planar/volumetric heads (surface and volumetric contexts) improve performance, while increased numbers of heads and network depth provide gains up to a threshold (Mazur et al., 2020).

5. Computational Complexity and Scalability

Let $i$6 denote the number of points, $i$7 the number of heads, $i$8 the grid size per head, and $i$9 the channel dimension. The asymptotic computational cost per head is:

• $k^h_i \in [0,1]^d$0 for key/value computation.
• $k^h_i \in [0,1]^d$1 for splatting/slicing.
• $k^h_i \in [0,1]^d$2 for the dense convolutions.

Total cost for MHCT is roughly $k^h_i \in [0,1]^d$3. For practical configurations (e.g., $k^h_i \in [0,1]^d$4, $k^h_i \in [0,1]^d$5 for 2D heads), $k^h_i \in [0,1]^d$6, making MHCT considerably more efficient than full $k^h_i \in [0,1]^d$7 self-attention yet more expensive than $k^h_i \in [0,1]^d$8 local graph convolutions. Dense grid convolutional operations are highly optimized for GPU execution, further benefiting practical scalability.

6. Comparison to Classical Attention and Spatial Transformer Mechanisms

MHCT diverges from classical multi-head dot-product attention, which computes dense pairwise scores $k^h_i \in [0,1]^d$9 at $d \in \{2,3\}$0 cost and aggregates as $d \in \{2,3\}$1. MHCT replaces the soft attention matrix with sparse, grid-based "attention" through splatting and convolutions, achieving linear scaling in $d \in \{2,3\}$2 with parallel convolutional heads and grid resolution $d \in \{2,3\}$3.

Contrasted with Spatial Transformer Networks, which use (global or local) warping and grid sampling, MHCT independently learns multiple per-point projections onto small grids in parallel. There is no single global transformation; instead, numerous local differentiable projections through individual heads are deployed simultaneously.

Points of distinction and advantages for point cloud applications include:

• Efficient global and long-range feature mixing via inexpensive convolutions on small grids.
• Full differentiability with respect to both key and value prediction pathways.
• Suitability for both recognition and generative tasks, in contrast to some attention-based point architectures that lack straightforward generative capabilities.
• Linear scaling with input size $d \in \{2,3\}$4 and optimal use of high-throughput convolutional primitives.

7. Practical Implementation and Application Scope

MHCT is implemented by coding differentiable splatting/slicing (potentially via CUDA-optimized routines for batching), one dense convolution per head, followed by summation and residual fusion. The architecture as described provides all necessary algorithmic elements for reimplementation and adaptation.

Architectures built with MHCT blocks achieve state-of-the-art performance across diverse point cloud tasks, confirming its generalist capabilities on recognition and generative benchmarks (Mazur et al., 2020). Empirical evaluation demonstrates competitive efficiency and accuracy relative to both attention-based and graph-based methods, particularly for large-scale point sets and in scenarios demanding both local and global context aggregation.