---
title: 'CTF-Net: Convolution & Transformer for Point Clouds'
url: https://www.emergentmind.com/topics/convolutional-transform-feature-network-ctf-net
type: topic
---

# CTF-Net: Convolution & Transformer for Point Clouds

A Convolutional Transform Feature Network (CTF-Net) is a deep neural architecture for learning expressive representations from unordered point clouds by jointly extracting and fusing fine-grained local and global geometric features. Its central module, the CT-block, couples a convolution-style local branch and a transformer-style global branch through bidirectional feature transmission, enabling mutual guidance between locality and context. The resulting network backbones—when stacked—demonstrate state-of-the-art effectiveness in point cloud classification and segmentation with compact computation [2111.15400].

## 1. Motivation and Architectural Rationale

Point clouds encode 3D geometry without spatial regularity. Local feature extraction algorithms (e.g., PointNet++, graph-CNNs) excel at capturing local shape but cannot encode global contextual dependencies. Conversely, transformer-based attention networks can learn holistic global relationships but struggle to retain detailed local priors. CTF-Net's innovation is to simultaneously and reciprocally learn both local and global features, mediated by lightweight feature transmission bridges within each CT-block, thereby ensuring maximally informative per-point descriptors.

## 2. CT-block: Joint Local and Global Feature Extraction

The CT-block is a dual-branch module designed for mutual enhancement of local and global point cloud features.

### 2.1 Convolution-branch (Local Feature Extractor)

- **Input:** $F_\ell^{(i-1)} \in \mathbb{R}^{N_\ell \times C_\ell}$, representing neighbor-grouped local features.
- **Operations:**
    1. **Sampling & Grouping:** Farthest Point Sampling (FPS) selects $N_\ell^\text{out}$ points. For each, the $S$ nearest neighbors form $F_1 \in \mathbb{R}^{N_\ell^\text{out} \times S \times C_\ell}$.
    2. **Point-wise MLP ("conv₁"):** $F_2 = \mathrm{LBR}_1(F_1)$.
    3. **Feature Transmission from Global Branch ("ft₂"):** $F_2' = F_2 + \mathrm{ft}_2(F_g^{(i)})$.
    4. **Second MLP ("conv₂"):** $F_3 = \mathrm{LBR}_2(F_2')$.
    5. **Max-pooling:** $F_\ell^{(i)} = \max_{j=1..S} F_3[:, j, :]$.
- **Key formula:**
    $$F_\ell^{(i)} = \text{conv}_2(\text{conv}_1(F_\ell^{(i-1)}) + \text{ft}_2(F_g^{(i)}))$$

### 2.2 Transformer-branch (Global Feature Extractor with Offset-Attention)

- **Input:** $F_g^{\text{in}} \in \mathbb{R}^{N_g \times d_e}$.
- **Operations:**
    1. **Linear projection:** $[Q, K, V] = F_g^{\text{in}} [W_q, W_k, W_v]$; $Q, K \in \mathbb{R}^{N_g \times d_a}, V \in \mathbb{R}^{N_g \times d_e}$.
    2. **Attention scores:** $\bar{A} = Q K^\top$.
    3. **Offset-Attention (Laplacian normalization):** Double channel softmax normalization.
    4. **Context aggregation:** $F_a = A V$.
    5. **Residual block:** $F_g^{\text{out}} = \mathrm{LBR}(F_a - F_g^{\text{in}}) + F_g^{\text{in}}$.
- **Key formula:**
    $$F_g^{(i)} = \mathrm{trans}( \mathrm{ft}_1(\mathrm{conv}_1(F_\ell^{(i-1)})) + F_g^{(i-1)} )$$

### 2.3 Feature Transmission Elements

Bidirectional connections ensure mutual guidance:

- $\mathrm{ft}_1$: "local → global" up-samples local features using distance-weighted interpolation, MLP, BN.
- $\mathrm{ft}_2$: "global → local" down-samples global features, matches sample indices and channels, MLP, BN.

## 3. Network Assembly for Classification and Segmentation

CT-blocks are assembled hierarchically to form CTF-Net backbones for classification and segmentation.

### 3.1 Classification Backbone

- **Input:** $P \in \mathbb{R}^{N \times 3}$.
- **Embedding:** Local (FPS and MLP) and global (MLP) feature streams.
- **CT-block stack:** $L=3$ blocks; convolution-branch downsamples by 2 ($N \rightarrow N/2$), channels double per block.
- **Heads:**
    - Local head: $\max$-pool $F_\ell^{L}$, FC, produce scores.
    - Global head: concatenate $F_g^{1}, F_g^{2}, F_g^{3}$, $\max$-pool, FC for scores.
    - Final output: Average between heads.

### 3.2 Segmentation Backbone (U-Net Style)

- **Encoder:** Identical to the classification stack.
- **Decoder:** Up-sample local features per level to $N$ points, add skip connections.
- **Global features:** Concatenate $F_g^{1..3}$ at final stage.
- **Heads:** FC layers for per-point local and global segmentation scores. Final output sums these scores.

## 4. Algorithmic Workflow and Pseudocode

### CT-block Forward Pass

```
Inputs: F_l_in (N_l×C_l_in), F_g_in (N_g×C_g_in)
Outputs: F_l_out (N_l′×C_l_out), F_g_out (N_g×C_g_out)
1. sample_idx ← FPS(N_l→N_l′)
2. F_grp ← group_points(F_l_in, sample_idx, S)
3. F2 ← LBR1(F_grp)
4. F_g_proj ← ft2(F_g_in)
5. F2′ ← F2 + expand(F_g_proj to S-dim)
6. F3 ← LBR2(F2′)
7. F_l_out ← max_pool(F3, dim=neighbors)
8. F_loc_proj ← ft1(F2)
9. F_qkv ← linear_qkv(F_g_in + F_loc_proj)
10. A ← normalize_offset_attention(Q,K)
11. F_a ← A·V
12. F_g_out ← LBR(F_a − F_g_in) + F_g_in
13. return F_l_out, F_g_out
```

### CTFNet Forward Pass

```
Inputs: P ∈ℝ^(N×3)
1. F_l0 ← LocalEmbedMLP(P)
2. F_g0 ← GlobalEmbedMLP(P)
3. for i in 1…L do
4.    F_li, F_gi ← CT_block_forward(F_l^{i−1}, F_g^{i−1})
5. end for
6. s_l ← fc_loc( max_pool(F_l^L) )
7. s_g ← fc_glob( max_pool(cat(F_g^1,F_g^2,F_g^3)) )
8. return (s_l + s_g) / 2
```

## 5. Hyper-parameters and Implementation Details

The network's design is characterized by:

- Number of CT-blocks: $L=3$
- Neighborhood size: $S=32$ in convolution-branch (optimal trade-off w.r.t. FLOPs and accuracy)
- Transformer embedding dimension: $d_e=256$, attention latent: $d_a=64$
- MLP expansion: Local channel doubles per block (32→64→128)
- Loss function: Dual cross-entropy (classification or per-point segmentation)
- Optimizer: SGD, momentum=0.9, initial lr=1e-3, cosine annealing
- Data augmentation: Classification uses random z-rotation and jitter ($\sigma=0.02$); segmentation uses anisotropic scaling in $[0.8, 1.25]$ and multi-scale testing

## 6. Empirical Results and Ablation Studies

CTF-Net achieves competitive state-of-the-art performance on ModelNet40 classification and ShapeNetPart segmentation benchmarks:

### ModelNet40 Classification

| Method        | OA    | mAcc   |
|---------------|-------|--------|
| PointNet++    | 91.9% |   –    |
| PCT           | 93.2% |   –    |
| CT-block      | 93.5% | 90.8%  |

### ShapeNetPart Segmentation

| Method        | pIoU (single) | pIoU (multi)  |
|---------------|--------------|--------------|
| PointNet++    |  –           | 85.1%        |
| PCT           |  –           | 86.4%        |
| CT-block      | 86.3%        | 86.5%        |

### Ablation Study (ModelNet40 OA / ShapeNetPart pIoU)

| Variant                       | OA     | pIoU    |
|-------------------------------|--------|---------|
| Convolution-branch only       | 91.82% | 85.23%  |
| Transformer-branch only       | 91.75% | 85.51%  |
| Both, no feature transmission | 92.59% | 85.70%  |
| Full CT-block                 | 93.52% | 86.29%  |

### Hyper-parameter Trade-offs

- Varying $S \in \{8,16,32,64,128\}$: $S \approx 32$ achieves the best accuracy/cost.
- Transformer dimension $d_e \in \{64,128,256,512,1024\}$: $d_e=256$ optimal before overfitting and computational expense rise.

## 7. Comparative Analysis with Related Architectures

The CTF-Net’s simultaneous integration of convolution and attention parallels developments in other joint local-global point cloud processing networks. 3DCTN [2203.00828] utilizes hierarchical sampling, multi-scale graph convolution for local feature aggregation (LFA), and transformer-based global feature learning (GFL), with offset-attention and vector subtraction operators yielding optimal trade-offs. Both CTF-Net and 3DCTN demonstrate that coupling local grouping and global attention surpasses single-paradigm baselines in accuracy and efficiency.

A plausible implication is that mutual guidance between local and global branches, when supported by effective inter-branch feature transmission, is the architectural driver for the observed performance gains on point cloud benchmarks. Key distinctions include CTF-Net’s explicit dual-branch bridge approach at every block—contrasting with 3DCTN’s module-level alternation—and the demonstration that ablations removing either joint extraction or bidirectional transmission result in substantial accuracy degradation.

## Summary

The Convolutional Transform Feature Network (CTF-Net), constructed from multiple CT-blocks, enables the joint learning and fusion of convolution-local and transformer-global features for unordered point cloud data. Bidirectional transmission mechanisms interleave local and global semantic information, resulting in dense, discriminative descriptors. Empirical evidence substantiates the superiority of this dual-branch architecture in classification and segmentation tasks compared to single-method networks, establishing CTF-Net as an efficient, high-performing backbone for geometric deep learning [2111.15400].

Source: https://www.emergentmind.com/topics/convolutional-transform-feature-network-ctf-net