---
title: Dynamic Patch Merging in Vision Transformers
url: https://www.emergentmind.com/topics/dynamic-patch-merging-dcts
type: topic
---

# Dynamic Patch Merging in Vision Transformers

Dynamic Patch Merging (dCTS) represents a family of algorithms and architectural modules for adaptively reducing the number of tokens in vision transformers and related architectures by dynamically grouping, pooling, or merging input patches (or tokens) based on sample-dependent heuristics, learned scoring functions, or content-aware predictors. Unlike static downsampling or fixed pooling strategies, dCTS mechanisms enable efficient computation by concentrating representation capacity on semantically or structurally complex regions while aggressively merging homogeneous or low-importance areas, yielding significant computational, memory, and throughput gains across domains including high-resolution semantic segmentation, vision-language pretraining, and generative diffusion modeling [2509.14165, 2510.25067, 2412.10181, 2602.16968].

## 1. Core Principles and Variants of Dynamic Patch Merging

The unifying tenet of dCTS approaches is the dynamic, input-dependent adaptation of tokenization granularity within transformer-based vision architectures. Rather than partitioning an input image into a uniformly spaced, fixed grid of small patches (e.g., $16\times 16$ in standard ViT), dCTS modules determine—at runtime or via learned policies—which image regions can be efficiently represented by merged "superpatches" or segment-wise aggregations. Key implementation strategies include:

- **Learned Policy Networks:** As in STEP, a lightweight CNN-based policy network (EfficientNet-Lite0) computes homogeneity or similarity scores over recursive patch groups. Windows exceeding an adaptive threshold are merged, ensuring uniform spatial size after raster concatenation and resizing [2509.14165].
- **Feature Clustering and Importance Scoring:** In Patch-Merging Transformer (PMT), tokens are clustered according to a DPC-KNN density and distance product metric; importance-weighted merging and residual feature updates via cross-attention preserve information from dense or intricate regions [2412.10181].
- **Boundary-Predictor and Differentiable Pooling:** DRIP and similar frameworks use a shallow MLP with Gumbel-Sigmoid gating to assign binary boundaries between tokens, segmenting and pooling embeddings dynamically. This boundary rate aligns with semantic or object structure, facilitating interpretable reductions [2510.25067].
- **Dynamic Patch Scheduling in Generative Models:** DDiT varies the patch size per diffusion timestep according to latent 'acceleration' (third-order finite differences of denoising latents), adapting computation and granularity throughout the generation process [2602.16968].

A summary of leading dCTS system characteristics is presented below:

| Framework      | Granularity Choice         | Merge Trigger                 | Pooling Method      |
|----------------|---------------------------|-------------------------------|---------------------|
| STEP/dCTS      | Hierarchical patch windows | Similarity score (CNN) > $\tau$ | Raster concat + resize |
| PMT/dCTS       | Token clusters (variable)  | DPC-KNN importance           | Weighted softmax + cross-attn |
| DRIP/dCTS      | Sequential segments        | Boundary MLP (Gumbel-Sigmoid)| Average pooling     |
| DDiT/dCTS      | Patch size per timestep    | Per-step latent statistics    | Patch-embed + LoRA |


## 2. Mathematical Formulations and Algorithmic Workflows

dCTS algorithms are characterized by content-dependent, granularity-adaptive processing, typically implemented by a two-stage procedure: (1) importance or boundary prediction followed by (2) segment formation and pooling.

### 2.1 STEP/dCTS Formulation

- **Similarity score** for a window $\mathcal{W}$:
  $$
  S = \sigma(W_p^T \mathrm{embedding}(\mathcal{W})) \in [0,1]
  $$
- **Merge decision:** If $S > \tau_\text{window}$, all $n$ patches in $\mathcal{W}$ are merged.
- **Superpatch construction:** Raster-concatenate input patches, bilinear resize to $16 \times 16$, linearly embed to token space.

No merge-specific loss is used; dCTS is trained end-to-end with standard segmentation losses [2509.14165].

### 2.2 PMT/dCTS Cluster-Based Merging

- **Local density:** ${\rho_i} = \exp\big[ -\frac{1}{k} \sum_{x_j \in \text{KNN}(x_i)} \|x_i - x_j\|^2 \big]$
- **Distance to higher-density point:** $\delta_i$, defined piecewise based on neighborhood density.
- **Importance:** $s_i = \rho_i \cdot \delta_i$
- **Selection:** Top $M$ tokens (by $s_i$) are cluster centers.
- **Merging:** Within clusters, importance-weighted sum via softmax scores.
- **Residual update:** Cross-attention combines merged tokens with the full set of original tokens [2412.10181].

### 2.3 DRIP/dCTS Segmentwise Pooling

- **Boundary prediction:** For $n$ input tokens $\{x_1, \ldots, x_n\}$, a 2-layer MLP outputs logits $\ell_t$, translated to probabilities $p_t = \sigma(\ell_t)$.
- **Gumbel-Sigmoid Sampling:**
  $$
  \hat{b}_t = \sigma\left(\frac{\mathrm{logit}(p_t) + \mathrm{logit}(u_t)}{\tau}\right)
  $$
  with $u_t \sim \text{Uniform}(0,1)$, used to produce binary boundaries $b_t$.
- **Pooling:** Tokens between adjacent boundaries are average-pooled to form merged tokens [2510.25067].

### 2.4 DDiT/dCTS Adaptive Patch Scheduling

- For each diffusion inference step $t$, compute the third-order finite-difference:
  $$
  \Delta^3 z_{t-1} = (z_{t-1} - z_t) - (z_t - z_{t+1})
  $$
- Partition into patches of candidate sizes $p_i$; for each, compute the ρ-th percentile of standard deviations $\sigma^{p_i,(\rho)}$.
- **Choice:** Select largest $p_t$ with $\sigma^{p_i,(\rho)} < \tau$; otherwise use finest patching.
- Patch-embed, apply transformer step, and proceed to next timestep [2602.16968].

## 3. Integration into Transformer Architectures

Integration strategies for dCTS modules depend on architectural and task-specific requirements:

- **ViT/Semantic Segmentation:** Merged patches or superpatches are resized to a canonical size (e.g., $16 \times 16$ or $32 \times 32$), linearly embedded, and fed to downstream transformer encoder blocks. Learned position embeddings are transferred by bilinear interpolation, maintaining spatial geometry [2509.14165, 2412.10181].
- **Dual- or Multi-Scale Backbones:** dCTS augments or replaces hierarchical downsampling with dynamic, scene-dependent granularity selection, supporting both single-branch and multi-branch architectures for segmentation [2412.10181].
- **Vision-Language Pretraining:** DRIP's dynamically pooled token set is padded and masked for continuity in transformer operations, enabling flexible batch processing despite variable token count across examples [2510.25067].
- **Diffusion Models:** DDiT introduces separate linear patch embeddings per patch size, interpolated positional encodings, and LoRA adapters for minimal fine-tuning. At inference, patch-embed modules are switched dynamically, while attention and feed-forward layers remain unchanged [2602.16968].

## 4. Computational Complexity and Efficiency Gains

dCTS modules universally target substantial reductions in token count, FLOPs, and memory usage without significant performance loss.

- **STEP/dCTS (ViT-Large on 1024×1024 images):**
  - Tokens: $4096 \rightarrow 1624$ ($\approx$2.5× reduction)
  - FLOPs: $2086 \rightarrow 802$ GFLOPs ($\approx$2.6× reduction)
  - Throughput: $12\rightarrow41$ FPS ($\approx$3.4× boost)
  - $<1.5\%$ drop in mIoU for segmentation [2509.14165].

- **PMT/dCTS (Cityscapes):**
  - Memory usage: $2.1\rm\,GB$ (single-branch) vs $2.4\rm\,GB$ (dual-branch) with higher mIoU [2412.10181].

- **DRIP/dCTS (ViT-B-16):**
  - 4× pooling: $11.29 \rightarrow 8.46$ GFLOPs ($\approx$1.33× speedup)
  - Top-1 ImageNet drop $\approx0.6\%$ at 4× reduction [2510.25067].

- **DDiT/dCTS (Diffusion):**
  - Up to $3.52\times$ speedup in image generation, with $0.6$ FID increase at most on FLUX-1.Dev [2602.16968].

A table summarizing measured reductions:

| Method & Domain       | Token Reduction | FLOP Reduction | Throughput Speedup | Principal Performance Impact |
|----------------------|----------------:|---------------:|-------------------:|-----------------------------|
| STEP/dCTS [2509.14165] | 2.5×            | 2.6×           | 3.4×               | $<$1.5% mIoU drop           |
| PMT/dCTS [2412.10181]  | Adaptive        | $\sim$2.1$\rm\,GB$| -                 | $-$2.7% mIoU if dCTS removed|
| DRIP/dCTS [2510.25067] | 4–10×           | 1.33–1.77×      | up to 1.8×         | $<$1% top-1 drop (ImageNet) |
| DDiT/dCTS [2602.16968] | Adaptive        | up to 3.52×     | up to 3.52×        | $\le0.6$ FID (FLUX-1)       |


## 5. Empirical Results and Ablation Studies

Experiments across segmentation, vision-language pretraining, and diffusion synthesis tasks consistently report substantial savings in computational cost with minimal to moderate baseline metric degradation:

- **STEP/dCTS:** $<1.5\%$ absolute mIoU drop across Cityscapes, COCOStuff10k, ADE20K. Most merges occur at the 2×2 patch level; larger merges are rare, especially in visually complex scenes [2509.14165].
- **PMT/dCTS:** On DeepGlobe, dCTS outperforms static convolutional downsampling by 2.7% mIoU; memory use is consistently lower than dual-branch comparators with state-of-the-art performance [2412.10181].
- **DRIP/dCTS:** 4× token reduction yields only 0.6% ImageNet top-1 loss; CLIP zero-shot retrieval is also retained. Replacing dynamic pooling with static pooling reduces accuracy, highlighting the merit of adaptive merging [2510.25067].
- **DDiT/dCTS:** Best results with order-3 finite-difference "acceleration" signal for patch scheduling. Threshold tuning trades off speed for quality, but even at 3.5× acceleration, CLIP similarity drops by only $<0.006$ [2602.16968].

A plausible implication is that the informatics of adaptive pooling or merging inherently preserves salient structures in deep representations, limiting the performance penalty even at aggressive compression levels.

## 6. Applications and Generalization

Dynamic patch merging has proven effective in a range of domains:

- **Semantic Segmentation:** Enables practical high-resolution training and inference on images up to $1024\times1024$, with adaptive focus on object boundaries and fine-grained details [2509.14165, 2412.10181].
- **Vision-Language Models:** DRIP demonstrates efficient pretraining and continual domain adaptation (e.g., BioCLIP) with substantial memory and flops savings, agnostic to backbone architecture [2510.25067].
- **Diffusion-Based Generative Models:** DDiT achieves significant test-time speedup for both image and video synthesis, maintaining generation quality and consistency with textual or visual prompts [2602.16968].
- **General Vision Tasks:** dCTS modules can be readily integrated into detection (small object emphasis), depth estimation (discontinuity-aware pooling), and video processing (motion boundary preservation) [2412.10181].

The evidence suggests that dCTS is a general, extensible paradigm for dynamic token adaptation in large-scale vision transformers, complementing or surpassing static and two-branch granularity strategies.

## 7. Limitations, Challenges, and Future Directions

While dCTS modules achieve favorable trade-offs between computational efficiency and representation integrity, some important considerations include:

- **Trade-off Tuning:** Thresholds and boundary rates introduce task- and data-dependent hyperparameters; careful calibration is required for optimal balance [2509.14165, 2510.25067].
- **Complexity of Dynamic Scheduling:** For generative diffusion models, the scheduling of patch sizes introduces new layers of inference-time complexity and potential instability, though LoRA adapters and residual skips help regularize training [2602.16968].
- **Information Loss in Extreme Compression:** Aggressive token reduction (e.g., 10× in DRIP) yields non-negligible accuracy drops, highlighting a limit to achievable compression without substantial information loss [2510.25067].
- **Scalability of Pooling Functions:** For large candidate patch grids or deep transformer stacks, clustering and importance metric computation may introduce additional overhead, and the scalability of assignment algorithms can become a bottleneck [2412.10181].

Future research is expected to focus on learned, end-to-end differentiable token repartitioning, improved interpretability of merging decisions, and extension to multimodal and temporally dynamic vision tasks leveraging the flexibility of dCTS paradigms.

Source: https://www.emergentmind.com/topics/dynamic-patch-merging-dcts