Density Peak Clustering-Fuzzy KNN

• The paper demonstrates a novel fuzzy integration in density estimation, enabling robust token center selection without iterative optimization.
• DPC-FKNN combines local and global density terms via fuzzy KNN weighting, enhancing semantic tokenization in transformer architectures.
• Empirical studies reveal improved performance and scalability in various image recognition tasks with minimal additional computational overhead.

Density Peak Clustering–Fuzzy K-Nearest Neighbor (DPC-FKNN) is a hybrid clustering mechanism that combines the center-selection logic of Density Peak Clustering (DPC) with fuzzy K-nearest-neighbor weighting. In FTCFormer, DPC-FKNN is the center-determination stage of a clustering-based token downsampling module: it computes a fuzzy neighborhood kernel over token embeddings, derives a density score for each token, evaluates its distance to higher-density tokens, and selects the highest-scoring density peaks as cluster centers (Bao et al., 14 Jul 2025). More broadly, related work on density-based clustering indicates that DPC-FKNN should be understood as part of a wider design space in which DPC’s sensitivity to local density estimation is mitigated by adaptive neighborhood modeling, graph structure, or diffusion-based density surrogates (Zheng et al., 2021).

1. Definition and conceptual position

Within FTCFormer, DPC-FKNN replaces the DPC-KNN mechanism used in TCFormer and is explicitly intended to be robust to the choice of $K$, robust to uneven densities and noisy feature maps, and amenable to matrix-based, non-iterative implementation inside a transformer block (Bao et al., 14 Jul 2025). Its immediate purpose is not generic offline clustering, but semantic token center selection during hierarchical downsampling. The selected centers determine which spatial locations survive as downsampled tokens and therefore control later token assignment and token merging.

The mechanism retains the two canonical DPC assumptions. First, cluster centers should have high local density relative to surrounding points. Second, cluster centers should be far from any point of still higher density. DPC-FKNN preserves these assumptions but modifies how density is estimated: instead of a hard KNN count or a cutoff-ball count, it uses a fuzzy kernel that assigns stronger weights to points inside the $K_{Fuzzy}$-nearest-neighbor set and weaker, globally discounted weights to all remaining points (Bao et al., 14 Jul 2025).

A common misconception is that DPC-FKNN necessarily implies a standalone iterative fuzzy clustering algorithm. In FTCFormer that is not the case. The fuzzy component is embedded in the density estimator itself, and center selection remains a deterministic DPC-style ranking step. There is no explicit fuzzy c-means-style alternating optimization and no separate iterative membership-update phase (Bao et al., 14 Jul 2025).

2. Mathematical mechanism

At a given stage, FTCFormer operates on tokens $\{x_i\}_{i=1}^N$, with $x_i \in \mathbb{R}^C$, obtained from a feature map of size $H \times W$, so $N = H \times W$. The basic metric is Euclidean distance,

$d_{ij} = \|x_i - x_j\|_2.$

Two hyperparameters appear in the surrounding module: $K_{Fuzzy}$ for DPC-FKNN and $K_{SCS}$ for the later Spatial Connectivity Score assignment stage (Bao et al., 14 Jul 2025).

The fuzzy kernel $\mu(i,j)$ is defined casewise. Tokens inside $K_{Fuzzy}$0 receive a stronger exponential weighting, while tokens outside $K_{Fuzzy}$1 are attenuated by a global sparsity factor $K_{Fuzzy}$2, the standard deviation of pairwise distances. The denominator $K_{Fuzzy}$3 further downweights distant points. The intended properties are distance monotonicity and neighborhood emphasis: closer pairs yield larger $K_{Fuzzy}$4, and points inside $K_{Fuzzy}$5 contribute much more strongly than non-KNN points (Bao et al., 14 Jul 2025).

Local density $K_{Fuzzy}$6 is then formed by combining two terms: a KNN-based term over $K_{Fuzzy}$7 and a global term over all $K_{Fuzzy}$8 tokens. The paper’s interpretation is that the first term captures local concentration, whereas the second term supplies global embedding context. This mixed local-plus-global construction is the core FKNN contribution in FTCFormer: it softens the brittleness of hard KNN density estimates and reduces dependence on a single neighborhood scale (Bao et al., 14 Jul 2025).

Distance to higher-density points follows standard DPC: $K_{Fuzzy}$9 The scalar center score is

$\{x_i\}_{i=1}^N$0

Cluster centers are selected by maximizing $\{x_i\}_{i=1}^N$1, and the top $\{x_i\}_{i=1}^N$2 scoring tokens are retained, corresponding to $\{x_i\}_{i=1}^N$3 downsampling and preserving hierarchical consistency with standard vision transformers (Bao et al., 14 Jul 2025).

3. Role inside the FTCFormer downsampling pipeline

DPC-FKNN is the first step of the Fuzzy Token Clustering and Merging (FTCM) module. Each FTCM block has four stages: clustering center determination, token assignment, token merging, and token interaction. DPC-FKNN is the mechanism used in the first stage and therefore anchors the rest of the pipeline (Bao et al., 14 Jul 2025).

After centers are selected, assignment is not performed by DPC-FKNN itself. FTCFormer introduces a Spatial Connectivity Score (SCS) for token-to-center assignment. SCS uses shared nearest neighbors and a closeness term derived from inverse distances over local neighborhoods. Each non-center token is assigned to the center with maximal SCS; if all SCS values are zero, assignment falls back to Euclidean distance. This design separates the notion of “what is a center” from “how are non-centers attached,” with DPC-FKNN providing the former and SCS the latter (Bao et al., 14 Jul 2025).

The next stage is channel-wise merging (Cmerge), in which tokens inside each cluster are merged by a channel-dependent weighted average using learned importance scores $\{x_i\}_{i=1}^N$4. The merged tokens then interact with the original tokens via cross-attention. DPC-FKNN therefore influences not only center selection but the entire geometry of the downsampled representation, because better centers induce more semantically coherent clusters before Cmerge and attention are applied (Bao et al., 14 Jul 2025).

In the transformer setting, the method’s stated motivation is semantic tokenization. The density peaks found in feature space are meant to correspond to semantically meaningful regions rather than fixed spatial grids. The reported visualizations associate finer token allocation with foreground objects, lesions, digits, faces, and structural parts, while allocating fewer tokens to background regions. This suggests that DPC-FKNN is functioning as a nonuniform feature-space sampler rather than a conventional image-space partitioner (Bao et al., 14 Jul 2025).

4. Relation to density estimation, scalability, and broader DPC variants

Research on density design for DPC is directly relevant to DPC-FKNN because the center score $\{x_i\}_{i=1}^N$5 is only as informative as the density estimator $\{x_i\}_{i=1}^N$6. Kernel-diffusion work replaces the naive $\{x_i\}_{i=1}^N$7-ball density used in DBSCAN and classical DPC with the stationary distribution of a Markov chain built from a kernel $\{x_i\}_{i=1}^N$8. Its fast surrogate, the Fast Kernel Diffusion (FKD) density,

$\{x_i\}_{i=1}^N$9

is the column average of the transition matrix and is computable in linear time and space for sparse graphs. Under an $x_i \in \mathbb{R}^C$0-separated-cluster assumption with uniform $x_i \in \mathbb{R}^C$1-ball kernel, the cluster-average FKD density is equal across clusters, and Theorem 2 establishes asymptotic equivalence between FKD and the full kernel diffusion density. For DPC-FKNN-style pipelines, this suggests that fuzzy assignment and center selection can benefit from replacing naive densities with adaptive diffusion densities, especially when clusters are small or uneven (Zheng et al., 2021).

Large-scale DPC research has also clarified how the DPC backbone can be accelerated independently of the fuzzy layer. PECANN abstracts DPC into index construction, neighbor search, density computation, dependent-point computation, center/noise selection, and label propagation, then implements predicate-based search for nearest higher-density neighbors using graph-based approximate nearest neighbor search with a doubling strategy. On datasets up to 1.28 million points and 1024 dimensions, PECANN reports $x_i \in \mathbb{R}^C$2–$x_i \in \mathbb{R}^C$3 speedups over FASTDP while maintaining competitive ARI, and two orders of magnitude speedup over a parallel low-dimensional DPC baseline (Yu et al., 2023). A plausible implication is that a large-scale DPC-FKNN can reuse this decomposition: approximate or exact DPC for center discovery, followed by fuzzy membership estimation on the same neighbor graph.

Exact parallel DPC work provides a complementary result for low- to moderate-dimensional regimes. Priority-search kd-trees and Fenwick-tree-based methods achieve exact density and dependent-point computation with polylogarithmic span, and the best algorithm reports $x_i \in \mathbb{R}^C$4–$x_i \in \mathbb{R}^C$5 speedups over the previous best parallel exact DPC algorithm while remaining exact (Huang et al., 2023). This suggests that DPC-FKNN does not inherently require approximation: when geometry is favorable, exact DPC center extraction can serve as the front end of a fuzzy KNN hybrid.

For mixed-type data, component-wise peak-finding methods build mutual KNN graphs, compute KNN-based local density inside connected components, and use conductance to determine the number of centers with overall $x_i \in \mathbb{R}^C$6 complexity. The associated technical analysis explicitly interprets this as a natural substrate for DPC-FKNN-style assignment on mixed attributes (Tobin et al., 2020). This suggests that DPC-FKNN should be regarded not as a single formula, but as a compositional scheme whose DPC, density, graph, and fuzzy-assignment pieces can be adapted to different data regimes.

5. Empirical behavior

In FTCFormer’s component ablation, adding DPC-FKNN alone to the TCFormer baseline improves Flowers102 from $x_i \in \mathbb{R}^C$7 to $x_i \in \mathbb{R}^C$8, DTD from $x_i \in \mathbb{R}^C$9 to $H \times W$0, Stanford Cars from $H \times W$1 to $H \times W$2, and Blood Cell from $H \times W$3 to $H \times W$4. The same ablation reports a compute increase of only $H \times W$5 GFLOPs and no extra parameters, isolating center determination as the dominant single contributor among the FTCM components (Bao et al., 14 Jul 2025).

At the full-model level, FTCFormer reports consistent gains over TCFormer on 32 datasets: $H \times W$6 on five fine-grained datasets, $H \times W$7 on six natural image datasets, $H \times W$8 on three medical datasets, and $H \times W$9 on four remote sensing datasets. The interpretation given in the paper is that semantically adaptive tokenization, with DPC-FKNN controlling where tokens are allocated, is responsible for these gains (Bao et al., 14 Jul 2025).

Independent density-based clustering evidence supports the same general conclusion: better density estimation materially improves DPC-style center recovery. On 13 UCI benchmarks, kernel-diffusion densities used inside DPC outperform naive and local-contrast densities; on emore_200k, asymmetric FKD reaches a BCubed F-score of $N = H \times W$0; and on MS1M, asymmetric KD and FKD report $N = H \times W$1 and $N = H \times W$2 for Pairwise F-score / BCubed F-score, respectively (Zheng et al., 2021). Although these results are not evaluations of FTCFormer’s DPC-FKNN, they reinforce the broader point that DPC hybrids are highly sensitive to the underlying density surrogate.

6. Limitations, misconceptions, and open directions

The main practical limitation in FTCFormer is computational. DPC-FKNN requires pairwise distances or KNN structure over current-stage tokens, and the paper explicitly identifies acceleration for high-resolution images as a future direction. The method is also non-parametric and not directly supervised: its effect on the end-to-end objective is indirect, mediated through better tokenization rather than a dedicated clustering loss (Bao et al., 14 Jul 2025).

Hyperparameter sensitivity is reduced but not eliminated. FTCFormer sets $N = H \times W$3 and $N = H \times W$4 across all datasets, and the reported ablation indicates robustness for a range of $N = H \times W$5 values except degenerate ones. By contrast, very small $N = H \times W$6 values can cause SCS to degenerate toward Euclidean assignment and degrade performance (Bao et al., 14 Jul 2025). More generally, related DPC research shows that graph connectivity, kernel bandwidth, and neighborhood size remain structurally important even when adaptive or diffusion-based densities are used (Zheng et al., 2021).

Another misconception is to equate DPC-FKNN with soft final labeling alone. In FTCFormer, fuzzy weighting primarily modifies density estimation before center selection; the later assignment stage is SCS-based and ultimately hard. Conversely, in other DPC-plus-fuzzy-KNN designs discussed in related technical summaries, the fuzzy part can appear after DPC as a soft assignment stage. This suggests that “DPC-FKNN” names an architectural pattern rather than a universally fixed algorithm.

The most immediate extensions stated for FTCFormer are transfer of its clustering principles to CNN architectures, acceleration for high-resolution images, and adaptation to other modalities such as text or point clouds (Bao et al., 14 Jul 2025). A plausible implication, when read alongside scalable DPC and adaptive-density work, is that future DPC-FKNN systems will separate three concerns: robust density construction, scalable dependent-point search, and fuzzy assignment or weighting tuned to the downstream task.