FTCFormer: Fuzzy Token Clustering Transformer

• FTCFormer is a hierarchical vision transformer that replaces fixed grid-downsampling with semantically adaptive fuzzy token clustering.
• Its FTCM module uses DPC-FKNN for center selection, Spatial Connectivity Score for token assignment, and channel-wise merging to preserve semantic details.
• Empirical results show consistent gains in classification accuracy across multiple datasets with marginal increases in computational cost.

Searching arXiv for the FTCFormer paper and closely related backbone references. FTCFormer, short for Fuzzy Token Clustering Transformer, is a transformer-based backbone for image classification that replaces fixed grid-downsampling with a semantically adaptive token reduction mechanism. The central premise is that most transformer architectures embed images into uniform, grid-based vision tokens and therefore neglect the underlying semantic meanings of image regions, which can lead to suboptimal feature representations. FTCFormer addresses this by introducing a Fuzzy Token Clustering and Merging (FTCM) module that dynamically generates vision tokens based on semantic meanings rather than spatial positions, allocating fewer tokens to less informative regions and more to semantically important regions, regardless of spatial adjacency or shape irregularity (Bao et al., 14 Jul 2025).

1. Architectural definition and backbone organization

FTCFormer follows a four-stage hierarchical Transformer backbone similar to PVT or TCFormer, but replaces the fixed grid-downsampling between stages with FTCM. In each downsampling transition, specifically between Stage 1→2, 2→3, and 3→4, the module performs four operations in sequence: clustering center determination via DPC-FKNN, token assignment via Spatial Connectivity Score (SCS), channel-wise token merging (Cmerge), and token interaction via Cross-Attention (Bao et al., 14 Jul 2025).

The stage-wise organization is:

Input Tokens → [L Transformer Layers + Stride-Conv SR] → FTCM ↓ → Next-Stage Tokens

where the downward arrow denotes clustering-based downsampling. The final Stage 4 tokens are average-pooled and fed to a linear classification head.

This backbone definition places FTCFormer within the family of hierarchical vision transformers, but its downsampling operator is no longer a purely spatial reduction. Instead, the reduction step is conditioned on token similarity structure and local semantic organization. A plausible implication is that FTCFormer treats token count as a representational budget to be redistributed according to semantic salience rather than fixed image lattice geometry.

2. Fuzzy clustering center determination

The clustering-based downsampling module begins with DPC-FKNN, which combines density peak clustering with fuzzy $K$-nearest-neighbor structure. Given $N$ input tokens $X=\{x_i\}_{i=1}^N$ in $\mathbb{R}^C$, FTCFormer first computes pairwise Euclidean distances

$d_{ij}=\|x_i-x_j\|_2$

and the $K_{\!Fuzzy}$-nearest neighbors of each token $i$, denoted $\mathrm{KNN}(i)$.

A fuzzy distance kernel $\mu(i,j)$ is then defined as

$$\mu(i,j)= \begin{cases} \exp\!\left(-d_{ij}^2/(d_{ij}+1)^2\right), & \text{if } j\in \mathrm{KNN}(i) \ \exp\!\left(-(\phi\cdot d_{ij})^2/(d_{ij}+1)^2\right), & \text{otherwise} \end{cases}$$

where $N$0 attenuates contributions of non-neighbors.

The local density of token $N$1 combines KNN locality with global context:

$N$2

Let

$N$3

For each token $N$4, the distance score is

$N$5

FTCFormer then defines

$N$6

and sorts tokens by descending $N$7, selecting the top $N$8 centers. The number of centers is chosen to produce a 2× resolution reduction, with the example given as $N$9 to match typical strided Transformer downsampling.

Once the centers $X=\{x_i\}_{i=1}^N$0 are fixed, fuzzy memberships $X=\{x_i\}_{i=1}^N$1 are assigned by normalized fuzzy affinity:

$X=\{x_i\}_{i=1}^N$2

The paper also states that one may optionally iterate a standard fuzzy KNN update with fuzzifier $X=\{x_i\}_{i=1}^N$3:

$X=\{x_i\}_{i=1}^N$4

In methodological terms, DPC-FKNN is the component that establishes semantically meaningful centers before actual downsampling occurs. The combination of a local density term and a higher-density distance term makes center selection depend jointly on neighborhood support and feature-space separation, rather than on spatial subsampling heuristics alone (Bao et al., 14 Jul 2025).

3. Spatial Connectivity Score and channel-wise token merging

After center selection, FTCFormer performs hard assignment of remaining tokens to centers using the Spatial Connectivity Score (SCS). For a token $X=\{x_i\}_{i=1}^N$5 and center $X=\{x_i\}_{i=1}^N$6, the method defines:

• Shared Nearest Neighbors

$X=\{x_i\}_{i=1}^N$7

• Closeness to Neighbors

$X=\{x_i\}_{i=1}^N$8

• Spatial Connectivity Score

$X=\{x_i\}_{i=1}^N$9

The assignment rule is explicit. If $\mathbb{R}^C$0, the token is assigned to

$\mathbb{R}^C$1

Otherwise, it is assigned to the nearest center under Euclidean distance:

$\mathbb{R}^C$2

The stated purpose of SCS is to ensure that tokens join centers to which they are both close and share local high-density neighborhoods, thereby alleviating high-dimensional noise. This is a more structured criterion than pure nearest-center assignment. A plausible misconception is that FTCFormer clusters only by Euclidean proximity; the formulation shows that its assignment stage is explicitly topology-aware through shared-neighbor structure.

Within each cluster $\mathbb{R}^C$3, FTCFormer then applies channel-wise merging (Cmerge). For each feature channel $\mathbb{R}^C$4, let $\mathbb{R}^C$5 be the learned importance score for token $\mathbb{R}^C$6 in channel $\mathbb{R}^C$7, predicted by a small linear layer. The merged token $\mathbb{R}^C$8 is defined channel by channel as

$\mathbb{R}^C$9

The stated rationale is that, by regressing $d_{ij}=\|x_i-x_j\|_2$0 per channel rather than per token, Cmerge preserves fine-grained semantic details spread across different channels. This indicates that token coarsening in FTCFormer is not a simple averaging or pooling operation; it is a channel-selective aggregation scheme designed to retain heterogeneous semantic content across feature dimensions (Bao et al., 14 Jul 2025).

4. End-to-end forward pass and token interaction

The end-to-end forward pass is described as a nine-step pipeline:

1. Patch-embed the input image via initial strided convolution to produce tokens $d_{ij}=\|x_i-x_j\|_2$1 of size $d_{ij}=\|x_i-x_j\|_2$2.
2. Apply Stage 1 Transformer blocks to obtain $d_{ij}=\|x_i-x_j\|_2$3.
3. Apply FTCM downsampling, consisting of DPC-FKNN, SCS assignment, Cmerge, and Cross-Attention, producing $d_{ij}=\|x_i-x_j\|_2$4 with $d_{ij}=\|x_i-x_j\|_2$5.
4. Apply Stage 2 Transformer to obtain $d_{ij}=\|x_i-x_j\|_2$6.
5. Apply FTCM again to obtain $d_{ij}=\|x_i-x_j\|_2$7 with $d_{ij}=\|x_i-x_j\|_2$8.
6. Apply Stage 3 Transformer to obtain $d_{ij}=\|x_i-x_j\|_2$9.
7. Apply FTCM again to obtain $K_{\!Fuzzy}$0 with $K_{\!Fuzzy}$1.
8. Apply Stage 4 Transformer to obtain $K_{\!Fuzzy}$2.
9. Apply global average pooling to $K_{\!Fuzzy}$3, followed by a linear head for class scores.

The Cross-Attention inside each FTCM uses queries from the merged tokens and keys/values from the original tokens. The formulation also integrates the average channel importance $K_{\!Fuzzy}$4 as an additive bias in the softmax denominator.

This ordering is significant because it means clustering does not terminate token interaction with the original representation. Instead, merged tokens subsequently attend to the original tokens. This suggests that FTCFormer treats clustering-based downsampling as a lossy compression step that is immediately compensated by a token interaction mechanism, rather than as irreversible pooling. The paper characterizes the resulting module as maintaining end-to-end differentiability while only marginally increasing computational cost (Bao et al., 14 Jul 2025).

5. Empirical results across domains

FTCFormer is evaluated on 32 datasets across 7 domains, including fine-grained, natural, remote sensing, medical, MNIST-like, CIFAR-like, and other datasets such as DTD, Sketch, and FER2013. The reported training setup uses 2 Transformer blocks per stage, $K_{\!Fuzzy}$5, $K_{\!Fuzzy}$6, image resizing to $K_{\!Fuzzy}$7 for large images and $K_{\!Fuzzy}$8 for small images, and optimization with AdamW, LR = 0.001, cosine decay, and 5-epoch warm-up (Bao et al., 14 Jul 2025).

Average accuracy improvement over the TCFormer baseline is reported as follows:

Domain	Average improvement
Fine-grained	+1.43%
Natural	+1.09%
Medical	+0.97%
Remote sensing	+0.55%
MNIST-like	+0.21%
CIFAR-like	+0.06%

Representative Top-1 accuracy results are given explicitly. On Flowers102, performance improves from 77.83 to 79.12. On Stanford Cars, it improves from 81.83 to 84.55. On RESISC45, it improves from 96.43 to 96.62. On PCAM, it improves from 86.47 to 87.49. On ImageNet-1k, the paper reports 77.5 → 77.9, with parameters increasing from 14.1M → 14.6M and GFLOPs from 3.8 → 4.1.

These results are described as consistent improvements over the TCFormer baseline across diverse domains. The distribution of gains is not uniform: the largest reported average improvement is on fine-grained datasets, whereas MNIST-like and CIFAR-like gains are smaller. This suggests, though does not by itself prove, that semantically adaptive tokenization may be especially useful when class discrimination depends on localized or irregular semantic structures rather than on simpler global statistics.

6. Ablation findings, hyperparameters, and scope

The ablation study is reported on Flowers102 / DTD / Stanford Cars / BloodCell. The progression is:

• Baseline (DPC-KNN only): 77.83 / 52.66 / 81.83 / 88.73
• + DPC-FKNN: 78.63 / 53.62 / 84.72 / 89.10
• + SCS (assignment): 78.79 / 53.99 / 85.33 / 89.18
• + Cmerge (full FTCM): 79.12 / 54.20 / 84.55 / 89.91

The paper reports marginal overhead: +0.26 GFLOPs, +0.38 M params. It also states that FTCFormer adds only approximately +0.26 GFLOPs (+6.8%) and +0.38 M params (+2.7%) over TCFormer’s grid-downsampling while achieving up to +1.43% absolute accuracy. Relative to standard MaxPool/AvgPool downsampling, FTCM yields +1.23–1.56% gain on hard datasets, with Flowers102 and FER2013 cited as examples (Bao et al., 14 Jul 2025).

The reported critical hyperparameters are:

• $K_{\!Fuzzy}$9
• $i$0
• Number of centers per FTCM stage $i$1

The hyperparameter study states that performance is optimal near $i$2 and robust for $i$3, whereas small $i$4 degrades sharply because it resembles pure Euclidean assignment.

These findings delimit the method’s scope. FTCFormer is presented specifically for image classification, and the empirical claims are restricted to that setting. A plausible implication is that the method’s main contribution is not merely clustering as a preprocessing operation, but a tightly integrated downsampling operator whose center selection, assignment, merging, and post-merge interaction are jointly configured to favor semantically adaptive tokenization over rigid grid-based strategies.