---
title: 'FTCFormer: Fuzzy Token Clustering Transformer'
url: https://www.emergentmind.com/topics/fuzzy-token-clustering-transformer-ftcformer
type: topic
---

# FTCFormer: Fuzzy Token Clustering Transformer

Searching arXiv for the FTCFormer paper and closely related backbone references.
FTCFormer, short for **Fuzzy Token Clustering Transformer**, is a transformer-based backbone for image classification that replaces fixed grid-downsampling with a semantically adaptive token reduction mechanism. The central premise is that most transformer architectures embed images into uniform, grid-based vision tokens and therefore neglect the underlying semantic meanings of image regions, which can lead to suboptimal feature representations. FTCFormer addresses this by introducing a **Fuzzy Token Clustering and Merging (FTCM) module** that dynamically generates vision tokens based on semantic meanings rather than spatial positions, allocating fewer tokens to less informative regions and more to semantically important regions, regardless of spatial adjacency or shape irregularity [2507.10283].

## 1. Architectural definition and backbone organization

FTCFormer follows a **four-stage hierarchical Transformer backbone** similar to PVT or TCFormer, but replaces the fixed grid-downsampling between stages with FTCM. In each downsampling transition, specifically between **Stage 1→2, 2→3, and 3→4**, the module performs four operations in sequence: **clustering center determination via DPC-FKNN**, **token assignment via Spatial Connectivity Score (SCS)**, **channel-wise token merging (Cmerge)**, and **token interaction via Cross-Attention** [2507.10283].

The stage-wise organization is:

**Input Tokens → [L Transformer Layers + Stride-Conv SR] → FTCM ↓ → Next-Stage Tokens**

where the downward arrow denotes clustering-based downsampling. The final Stage 4 tokens are average-pooled and fed to a linear classification head.

This backbone definition places FTCFormer within the family of hierarchical vision transformers, but its downsampling operator is no longer a purely spatial reduction. Instead, the reduction step is conditioned on token similarity structure and local semantic organization. A plausible implication is that FTCFormer treats token count as a representational budget to be redistributed according to semantic salience rather than fixed image lattice geometry.

## 2. Fuzzy clustering center determination

The clustering-based downsampling module begins with **DPC-FKNN**, which combines density peak clustering with fuzzy $K$-nearest-neighbor structure. Given $N$ input tokens $X=\{x_i\}_{i=1}^N$ in $\mathbb{R}^C$, FTCFormer first computes pairwise Euclidean distances
$$
d_{ij}=\|x_i-x_j\|_2
$$
and the $K_{\!Fuzzy}$-nearest neighbors of each token $i$, denoted $\mathrm{KNN}(i)$.

A **fuzzy distance kernel** $\mu(i,j)$ is then defined as
$$
\mu(i,j)=
\begin{cases}
\exp\!\left(-d_{ij}^2/(d_{ij}+1)^2\right), & \text{if } j\in \mathrm{KNN}(i) \\
\exp\!\left(-(\phi\cdot d_{ij})^2/(d_{ij}+1)^2\right), & \text{otherwise}
\end{cases}
$$
where $\phi=\mathrm{std}(\{d_{ij}\})$ attenuates contributions of non-neighbors.

The **local density** of token $i$ combines KNN locality with global context:
$$
\rho_i=
\frac{1}{K_{\!Fuzzy}}\sum_{j\in \mathrm{KNN}(i)} \mu(i,j)
+
\frac{1}{N}\sum_{j=1}^N \mu(i,j).
$$

Let
$$
S=\arg\max_k \rho_k.
$$
For each token $i$, the **distance score** is
$$
\delta_i=
\begin{cases}
\min_{j:\rho_j>\rho_i}\|x_i-x_j\|_2, & \text{if } \exists j \text{ with } \rho_j>\rho_i \\
\max_j \|x_i-x_j\|_2, & \text{otherwise.}
\end{cases}
$$

FTCFormer then defines
$$
\gamma_i=\rho_i\cdot \delta_i
$$
and sorts tokens by descending $\gamma_i$, selecting the top $M$ centers. The number of centers is chosen to produce a **2× resolution reduction**, with the example given as $M\approx N/4$ to match typical strided Transformer downsampling.

Once the centers $C=\{c_1,\dots,c_M\}$ are fixed, **fuzzy memberships** $u_{ij}\in[0,1]$ are assigned by normalized fuzzy affinity:
$$
u_{ij}=\frac{\mu(i,c_j)}{\sum_{\ell=1}^M \mu(i,c_\ell)}.
$$
The paper also states that one may optionally iterate a standard fuzzy KNN update with fuzzifier $m>1$:
$$
u_{ij}^{(t+1)}=
\left[
\sum_{\ell=1}^M
\left(
\frac{\mu(i,c_j)}{\mu(i,c_\ell)}
\right)^{1/(m-1)}
\right]^{-1}.
$$

In methodological terms, DPC-FKNN is the component that establishes semantically meaningful centers before actual downsampling occurs. The combination of a local density term and a higher-density distance term makes center selection depend jointly on neighborhood support and feature-space separation, rather than on spatial subsampling heuristics alone [2507.10283].

## 3. Spatial Connectivity Score and channel-wise token merging

After center selection, FTCFormer performs **hard assignment** of remaining tokens to centers using the **Spatial Connectivity Score (SCS)**. For a token $i$ and center $c_j$, the method defines:

- **Shared Nearest Neighbors**
  $$
  |\mathrm{SNN}(i,j)| = |\mathrm{KNN}(i)\cap \mathrm{KNN}(c_j)|
  $$
- **Closeness to Neighbors**
  $$
  \mathrm{CN}(i,j)=\sum_{u\in \mathrm{KNN}(i)} \frac{1}{d_{iu}+1}
  +
  \sum_{v\in \mathrm{KNN}(c_j)} \frac{1}{d_{jv}+1}
  $$
- **Spatial Connectivity Score**
  $$
  \mathrm{SCS}(i,j)=|\mathrm{SNN}(i,j)|\cdot \mathrm{CN}(i,j)
  $$

The assignment rule is explicit. If $\max_j \mathrm{SCS}(i,j)>0$, the token is assigned to
$$
\arg\max_j \mathrm{SCS}(i,j).
$$
Otherwise, it is assigned to the nearest center under Euclidean distance:
$$
\arg\min_j \|x_i-c_j\|_2.
$$

The stated purpose of SCS is to ensure that tokens join centers to which they are both close and share local high-density neighborhoods, thereby alleviating high-dimensional noise. This is a more structured criterion than pure nearest-center assignment. A plausible misconception is that FTCFormer clusters only by Euclidean proximity; the formulation shows that its assignment stage is explicitly topology-aware through shared-neighbor structure.

Within each cluster $p$, FTCFormer then applies **channel-wise merging (Cmerge)**. For each feature channel $c=1,\dots,C$, let $P_j^c$ be the learned importance score for token $j$ in channel $c$, predicted by a small linear layer. The merged token $y_p\in\mathbb{R}^C$ is defined channel by channel as
$$
y_p^c=
\frac{\sum_{j\in \mathrm{Cluster}_p} e^{P_j^c}\cdot x_j^c}
{\sum_{j\in \mathrm{Cluster}_p} e^{P_j^c}}.
$$

The stated rationale is that, by regressing $P_j^c$ per channel rather than per token, Cmerge preserves fine-grained semantic details spread across different channels. This indicates that token coarsening in FTCFormer is not a simple averaging or pooling operation; it is a channel-selective aggregation scheme designed to retain heterogeneous semantic content across feature dimensions [2507.10283].

## 4. End-to-end forward pass and token interaction

The end-to-end forward pass is described as a nine-step pipeline:

1. Patch-embed the input image via initial strided convolution to produce tokens $X^0$ of size $N^0$.
2. Apply Stage 1 Transformer blocks to obtain $X^1$.
3. Apply FTCM downsampling, consisting of DPC-FKNN, SCS assignment, Cmerge, and Cross-Attention, producing $X^{1'}$ with $N^1\approx N^0/2$.
4. Apply Stage 2 Transformer to obtain $X^2$.
5. Apply FTCM again to obtain $X^{2'}$ with $N^2\approx N^1/2$.
6. Apply Stage 3 Transformer to obtain $X^3$.
7. Apply FTCM again to obtain $X^{3'}$ with $N^3\approx N^2/2$.
8. Apply Stage 4 Transformer to obtain $X^4$.
9. Apply global average pooling to $X^4$, followed by a linear head for class scores.

The **Cross-Attention** inside each FTCM uses queries from the merged tokens and keys/values from the original tokens. The formulation also integrates the **average channel importance** $\mathrm{avg}_c(P)$ as an additive bias in the softmax denominator.

This ordering is significant because it means clustering does not terminate token interaction with the original representation. Instead, merged tokens subsequently attend to the original tokens. This suggests that FTCFormer treats clustering-based downsampling as a lossy compression step that is immediately compensated by a token interaction mechanism, rather than as irreversible pooling. The paper characterizes the resulting module as maintaining **end-to-end differentiability** while only marginally increasing computational cost [2507.10283].

## 5. Empirical results across domains

FTCFormer is evaluated on **32 datasets across 7 domains**, including fine-grained, natural, remote sensing, medical, MNIST-like, CIFAR-like, and other datasets such as DTD, Sketch, and FER2013. The reported training setup uses **2 Transformer blocks per stage**, **$K_{\!Fuzzy}=5$**, **$K_{\!SCS}=5$**, image resizing to **$224\times224$** for large images and **$64\times64$** for small images, and optimization with **AdamW**, **LR = 0.001**, **cosine decay**, and **5-epoch warm-up** [2507.10283].

Average accuracy improvement over the **TCFormer baseline** is reported as follows:

| Domain | Average improvement |
|---|---:|
| Fine-grained | +1.43% |
| Natural | +1.09% |
| Medical | +0.97% |
| Remote sensing | +0.55% |
| MNIST-like | +0.21% |
| CIFAR-like | +0.06% |

Representative Top-1 accuracy results are given explicitly. On **Flowers102**, performance improves from **77.83** to **79.12**. On **Stanford Cars**, it improves from **81.83** to **84.55**. On **RESISC45**, it improves from **96.43** to **96.62**. On **PCAM**, it improves from **86.47** to **87.49**. On **ImageNet-1k**, the paper reports **77.5 → 77.9**, with parameters increasing from **14.1M → 14.6M** and GFLOPs from **3.8 → 4.1**.

These results are described as **consistent improvements over the TCFormer baseline** across diverse domains. The distribution of gains is not uniform: the largest reported average improvement is on fine-grained datasets, whereas MNIST-like and CIFAR-like gains are smaller. This suggests, though does not by itself prove, that semantically adaptive tokenization may be especially useful when class discrimination depends on localized or irregular semantic structures rather than on simpler global statistics.

## 6. Ablation findings, hyperparameters, and scope

The ablation study is reported on **Flowers102 / DTD / Stanford Cars / BloodCell**. The progression is:

- **Baseline (DPC-KNN only)**: 77.83 / 52.66 / 81.83 / 88.73
- **+ DPC-FKNN**: 78.63 / 53.62 / 84.72 / 89.10
- **+ SCS (assignment)**: 78.79 / 53.99 / 85.33 / 89.18
- **+ Cmerge (full FTCM)**: 79.12 / 54.20 / 84.55 / 89.91

The paper reports **marginal overhead: +0.26 GFLOPs, +0.38 M params**. It also states that FTCFormer adds only **approximately +0.26 GFLOPs (+6.8%) and +0.38 M params (+2.7%)** over TCFormer’s grid-downsampling while achieving **up to +1.43% absolute accuracy**. Relative to standard **MaxPool/AvgPool downsampling**, FTCM yields **+1.23–1.56% gain on hard datasets**, with Flowers102 and FER2013 cited as examples [2507.10283].

The reported critical hyperparameters are:

- $K_{\!Fuzzy}=5$
- $K_{\!SCS}=5$
- Number of centers per FTCM stage $\approx N/4$

The hyperparameter study states that performance is **optimal near $(K_{\!Fuzzy},K_{\!SCS})=(5,5)$** and **robust for $K_{\!Fuzzy}\in[3,7]$**, whereas **small $K_{\!SCS}=1$ degrades sharply** because it resembles pure Euclidean assignment.

These findings delimit the method’s scope. FTCFormer is presented specifically for **image classification**, and the empirical claims are restricted to that setting. A plausible implication is that the method’s main contribution is not merely clustering as a preprocessing operation, but a tightly integrated downsampling operator whose center selection, assignment, merging, and post-merge interaction are jointly configured to favor semantically adaptive tokenization over rigid grid-based strategies.

Source: https://www.emergentmind.com/topics/fuzzy-token-clustering-transformer-ftcformer