Patch Merging: Techniques and Applications

• Patch merging is a computational strategy that aggregates spatially or semantically proximate data patches into abstract representations, promoting efficient hierarchical processing.
• It is widely used in Vision Transformers, segmentation, and retrieval systems through methods like fixed spatial pooling and dynamic, feature-aware clustering.
• Advanced approaches dynamically adapt merging using local density measures and cross-attention to optimize trade-offs between context preservation and computational efficiency.

Patch merging refers to a family of computational and algorithmic strategies that combine or aggregate information from multiple spatially or semantically proximate "patches"—subregions or tokens of data—into fewer, typically more abstract, representations. It originated within image processing (both in physical sciences and computer vision) and has become an essential component in hierarchical Vision Transformers (ViTs), large-scale document analysis pipelines, nonparametric segmentation algorithms, and even astrophysical feature tracking. Modern research addresses both deterministic and data-adaptive (feature-aware) patch merging, covering mechanisms from simple spatial pooling to clustering-based selection, optimizing for both computational efficiency and preservation of fine spatial details.

1. Patch Merging in Hierarchical Vision Transformers

Hierarchical Vision Transformers employ patch merging to build multi-scale feature hierarchies, reminiscent of spatial resolution decreases in CNNs. The canonical design, as found in models like PVT and Swin Transformer, reduces the spatial grid size while (typically) increasing embedding dimension at stage boundaries. The standard implementation groups non-overlapping $2\times2$ windows and applies a linear projection: $$Y(i,j) = [X(2i,2j), X(2i+1,2j), X(2i,2j+1), X(2i+1,2j+1)]\,W_{PM} + b_{PM}$$ where $X \in \mathbb{R}^{H \times W \times C}$, $Y \in \mathbb{R}^{H/2 \times W/2 \times 2C}$ and $W_{PM} \in \mathbb{R}^{4C \times 2C}$ (Yu et al., 2024).

More advanced paradigms, such as Stepwise Patch Merging (SPM) and the Patch-Merging Transformer (PMT), move beyond fixed-grid downsampling. SPM includes Multi-Scale Aggregation (MSA) using channel-split depthwise convolutions of varied kernel size, followed by Guided Local Enhancement (GLE) with self-attention over local windows and guide token extraction, yielding improved trade-offs between context integration and fine-grained details (Yu et al., 2024). PMT, in contrast, dynamically adapts region granularity by data-driven clustering, allocating more representational capacity to informative or boundary-rich regions (Sun et al., 2024). This enables a single transformer to capture both global and local structure efficiently.

2. Data-Driven and Adaptive Patch Merging Strategies

Modern patch merging frequently leverages feature similarity for dynamic allocation, notably in ultra-high-resolution segmentation and memory-constrained retrieval. The PMT framework computes, for each patch token $x_i$:

• Local density $\rho_i$ using $k$-nearest neighbors:

$$\rho_i = \exp\Bigl(-\frac{1}{k}\sum_{x_j \in KNN(x_i)} \|x_i - x_j\|^2_2\Bigr)$$

• A density-peak distance score $s_i = \rho_i \cdot \delta_i$ determines cluster centers.
• Tokens are assigned to nearest cluster centers, and a softmax-weighted sum produces merged features $y_i$.
• Cross-attention between merged tokens and original patches reinjects fine-grained details (Sun et al., 2024).

Similarly, in visual document retrieval, token/patch merging seeks maximal information retention under fixed memory cost. Feature-wise hierarchical clustering (using cosine similarity) in low-dimensional projected space achieves significant memory compression (down to 2.8% of the original) with only a 5.4% drop in retrieval performance. In contrast, simple spatial pooling rapidly degrades beyond small compression factors (Ma et al., 5 Jun 2025).

The general step-wise structure of these adaptive approaches is as follows:

1. Compute feature similarities or statistical densities.
2. Cluster or partition patches into meaningful groups.
3. Aggregate features (e.g., via mean or weighted mean) by group.
4. Optionally, further refine merged representations via attention, convolution, or non-linear projection.

3. Patch Merging in Nonparametric Segmentation and Scientific Imaging

Outside deep learning, patch merging is a central operation in nonparametric segmentation and physical-data analysis. In latent source models for image segmentation, local label patch estimates $\xi_i$ are first inferred for each spatial location by maximizing a log posterior that balances kernel-density likelihood (of intensity patch $Y[i]$ given proposed label patch) and global consistency with the current dense label map $L$. Subsequently, a merging step—formulated as the global maximization of an augmented Lagrangian over $L$—fuses overlapping patch estimates to produce a globally consistent label configuration: $$L^{(t+1)} = \arg\max_L \Bigl(F(L;\alpha) - \frac{\beta}{2}\sum_{i} \|L[i] - \xi_i^{(t+1)} + \tfrac{1}{\beta}\lambda_i^{(t)}\|^2\Bigr)$$ This soft-averaging or "patch-mixing" is essential to propagate local consensus and suppress local inconsistencies, especially for fine structural boundaries (Chen et al., 2015).

In solar physics, merging describes the combination of spatially overlapping, contiguous magnetic flux patches into single entities in time-resolved magnetograms. Automatic algorithms detect cases where multiple patches in frame $t$ become a single patch in $t+1$, and statistical models relate merging frequency to flux content, with merging showing a weak power-law dependence:

$$\frac{\partial P_{\rm mrg}^{\rm APP}(\phi)}{\partial t} = p_{0,\rm mrg}\left(\frac{\phi}{\phi_0}\right)^{\beta_{\rm mrg}}$$

with $\beta_{\rm mrg} \approx 0.28$ (Iida et al., 2012). Merging (alongside splitting) dominates flux redistribution in the supergranular network, rapidly maintaining scale-free flux distributions.

4. Patch Merging for Memory-Efficient Retrieval and Representation

Storage-efficient visual document retrieval systems must aggregate high-dimensional patch-level embeddings into compact, semantically meaningful summaries. The principal strategies are:

• Spatial pooling (1D/2D): Group fixed windows and average features. Computationally trivial, but at high compression factors, loses semantic co-occurrence and deteriorates retrieval accuracy.
• Feature-space clustering (semantic clustering): Offline hierarchical clustering (e.g., Ward linkage) in low-dimensional embedding space, replacing all patch embeddings in a cluster by their mean. While clustering is $O(N_p^2)$, it yields substantial performance retention even at high merging factors.
• Train-free vs. fine-tuned merging: Post-hoc merging on frozen models is fast but less accurate at strong compression. End-to-end fine-tuning of the retriever with merging integrated recovers much of the performance loss, especially when the number of output patches is small (Ma et al., 5 Jun 2025).

The "Light-ColPali/ColQwen2" pipeline exemplifies this approach, achieving 98.2% retrieval accuracy with only 11.8% of original memory cost when merging patches via semantic clustering after the dimensionality projector. Fine-tuned variants outperform all spatial pooling or pruning baselines in the high-compression regime.

5. Patch Merging in Software Engineering: Codebase Patch Integration

In software engineering, "patch merging" refers to the integration of source code changes ("patches") across divergent codebases, most notably in long-lived software “forks” that have been structurally refactored. Challenges arise as textual context for a patch may no longer align due to renaming, relocation, and functional reorganization, rendering naive textual cherry-pick merges (e.g., git cherry-pick) likely to fail.

Refactoring-aware patch merging systems, such as RePatch (Ogenrwot et al., 8 Aug 2025), address this by:

1. Detecting refactorings (e.g., renames, moves, extractions) in both source and target via AST-level tools (RefactoringMiner).
2. Inverting those refactorings to temporarily realign both codebases to a common pre-refactoring structure.
3. Applying the patch in this realigned context.
4. Replaying the refactorings to restore the codebases to their independent structures.

Empirically, this procedure enables successful patch integration in 52.8% of cases where git cherry-pick fails due to refactoring-driven divergence, demonstrating its critical role in cross-variant software maintenance.

6. Computational Complexity and Empirical Impacts

The computational savings and accuracy effects of patch merging depend on context and algorithm:

• Hierarchical ViT patch merging reduces attention complexity from $O(P^2 D)$ to $O(P M D + k P D)$ in PMT, where $M\ll P$ in homogeneous regions and $k$ is the number of kNNs for density estimation (Sun et al., 2024). SPM incurs minor additional overhead (5–8%) but enables consistent gains in dense prediction (Yu et al., 2024).
• In visual document retrieval, memory cost scales with the number of patch-level embeddings retained ($N_p'$ vs. $N_p$). Semantic clustering is optimal for high merging factor.
• Scientific and segmentation contexts, where overlap is essential, pay a cost linear in the number of pixels times patch size per iteration but benefit from robust global consistency.

Ablation studies consistently show that dynamic, data-driven merging (clustering, attention-based updating) significantly outperforms static spatial pooling in both accuracy and flexibility, especially at aggressive compression or downsampling levels (Sun et al., 2024, Yu et al., 2024, Ma et al., 5 Jun 2025).

7. Broader Implications and Future Directions

Patch merging is a unifying abstraction across domains: controlling computational resource allocation, maintaining global consistency, achieving compression, and enabling robust cross-system transfer. The recent trends are toward (1) data-adaptive grouping via feature similarity, (2) integration with cross-attention to preserve details, and (3) algorithmic paradigms that blend global and local structure elegantly. Open challenges include making clustering or adaptive merging fast enough for real-time or extremely large-scale deployment, devising learnable merging strategies for non-vision data, and extending refactoring-aware merging to multiple languages and arbitrary code transformations.

Key papers providing foundational and state-of-the-art accounts of patch merging include (Sun et al., 2024, Yu et al., 2024, Ma et al., 5 Jun 2025, Chen et al., 2015, Iida et al., 2012), and (Ogenrwot et al., 8 Aug 2025).