Dynamic Patching & Adaptive Scale Selection

• Dynamic patching dynamically adjusts the granularity of computational partitions to focus resources on complex regions while saving on simpler ones.
• Adaptive scale selection uses content-aware metrics like entropy and finite-difference acceleration to determine optimal patch sizes without retraining.
• Together, these techniques reduce computational costs and improve model fidelity in tasks such as image generation, time series forecasting, and PDE modeling.

Dynamic patching and adaptive scale selection are families of methods that dynamically adjust the granularity of computational partitions—patches or token blocks—according to content complexity, information density, or resource constraints. These techniques appear across autoregressive visual generation, diffusion transformers, patch-based PDE surrogates, time series forecasting, high-resolution remote sensing, streaming volumetric data, and adversarial robustness. By operating at variable resolution, dynamic patching enables more efficient resource allocation, mitigating the quadratic (or higher) scaling of self-attention or tensor contraction costs while preserving (or even improving) model quality and fidelity.

1. Foundational Principles and Motivations

The inefficiency of fixed-size patching or tokenization is most acute in regimes where the complexity of the input signal is highly non-uniform. In visual transformers, uniform patches fail to allocate sufficient capacity to rich spatial details and waste computation on smooth regions. In time series or PDE modeling, uniform patches conflate periods of high and low signal activity, driving unnecessary computation in redundant segments. Dynamic patching addresses this by locally varying patch sizes ("granularity")—allocating finer patches to complex regions and coarser patches to smooth areas—often guided by content-aware metrics such as next-token entropy (Srivastava et al., 26 Dec 2025), spectral entropy (Feng et al., 30 Sep 2025), finite-difference acceleration (Kim et al., 19 Feb 2026), or data-driven difficulty measures (Liu et al., 2024).

Adaptive scale selection refers to the ability to choose, often at inference time and without retraining, the optimal patch size or scale to match a target computational budget, desired fidelity, or fluctuating signal complexity. The synergy between dynamic patching and adaptive scale selection forms the basis for compute-adaptive, robust, and high-fidelity modeling in large-scale, heterogeneous data domains.

2. Information-Theoretic and Data-Driven Patch Granularity Criteria

A central challenge in dynamic patching is establishing a criterion to locally select or schedule patch size. Several rigorous approaches have emerged:

• Next-token prediction entropy for image generation (DPAR): Shannon entropy computed from a lightweight autoregressive model, with low-entropy tokens merged into coarser patches and high-entropy ("detail-rich") tokens isolated into fine patches (Srivastava et al., 26 Dec 2025).
• Third-order finite-difference ("acceleration") for diffusion transformers (DDiT): The spatial variance of latent "acceleration" signals the emergence of details, triggering fine patches during late denoising or coarse patches in early structure-building phases (Kim et al., 19 Feb 2026).
• Spectral entropy and affinity-based gating for time series: Local FFT-based spectral entropy signals high information density, prompting selection of finer patches. Patch selection uses mixture-of-size token routers (MoS-DP) with affinity-based gating for instance-level adaptation (Feng et al., 30 Sep 2025).
• Patch confidence or reconstruction difficulty using learned or semantic confidence maps in diffusion SR (PatchScaler): Patches are grouped as "simple," "medium," or "hard" according to averaged reconstruction difficulty, and each group receives a different sample schedule (Liu et al., 2024).
• Relative local variation for time series (TimeSqueeze): A normalized difference threshold guides adaptive segmentation, allocating short patches to high-variation regions and long patches elsewhere (Ankireddy et al., 11 Mar 2026).
• Actor-critic (RL) scale policy for semantic segmentation in remote sensing (GeoAgent): A learned scale control agent observes local and global context to select the most appropriate zoom factor for each patch in a large image (Liu et al., 2023).
• Bond dimension/rank explosion in tensor decompositions: Patch splitting in QTT is triggered by exceeding a local bond-dimension cap or by pivot-cost/rank increase criteria, recursively refining only where the tensor is locally uncompressible (Grosso et al., 25 Feb 2026).

3. Algorithmic Frameworks and Architectural Integrations

Dynamic patching is implemented as a modular transformation in the model's data pipeline or backbone, often with minimal requirement for retraining or architectural disruption.

• Patch merging/splitting (DPAR, TimeSqueeze): Token streams are scanned left-to-right; tokens are appended to the current patch so long as the local content criterion and a patch-length cap are met. Otherwise, a new patch starts. Row boundary resets and hard caps stabilize spatial contiguity (Srivastava et al., 26 Dec 2025, Ankireddy et al., 11 Mar 2026).
• Patch schedule per denoising step (DDiT): At each timestep, the content-complexity metric is evaluated for all candidate patch sizes, and the largest admissible patch size is adopted for the next step. Seamless swapping of embedding and de-embedding kernels/positional encodings is handled via learned linear projections and LoRA branches (Kim et al., 19 Feb 2026).
• Mixture-of-size patch routing and fusion (Kairos): Each coarse patch is simultaneously assigned to multiple scale branches with softmax gating, and all activated granularities contribute to each fine token via weighted ancestry-MLP fusion (Feng et al., 30 Sep 2025).
• Divide-and-conquer adaptive partitioning (QTT): Patches are recursively subdivided along bit axes with the highest pivot cost, forming a block-sparse, divide-and-conquer representation that avoids the global rank explosion typical of monolithic TTs (Grosso et al., 25 Feb 2026).
• RL scale-control and dual-branch fusion (GeoAgent): Policy networks select scales for each windowed patch, and segmentation backbones fuse local and global context, with per-scale rewards and asynchronous optimization (Liu et al., 2023).
• Cyclic or budget-driven patch scheduling (CKM/CSM for PDEs): Plug-and-play kernel/stride modulators allow dynamic patch size scheduling (including cyclic modulation or explicit cost/error trade-off selection) in ViT-style PDE surrogates (Mukhopadhyay et al., 12 Jul 2025).
• Adversarial patch optimization via differentiable superpixel clustering (SPAP): SLIC-based clustering is differentiated via the Implicit Function Theorem within an EOT pipeline, enabling piecewise-smooth, scale-resilient adversarial patch geometries (Bagley et al., 23 Nov 2025).

4. Efficiency, Scaling, and Theoretical Properties

Dynamic patching consistently reduces computational load—e.g., quadratic attention or contraction costs—by a factor proportional to the square of the average patch length. This is achieved while maintaining, or sometimes improving, model quality metrics.

• Image generation (DPAR): Token count reduced by 1.81x–2.06x, training FLOPs by up to 40%, FID improved by up to 27% over static patching; dynamic scaling at inference achieves further 10–15% FLOPs trade-off with minimal FID degradation (Srivastava et al., 26 Dec 2025).
• Diffusion transformers (DDiT): Inference speedups of 2.18x–3.52x at <1% FID/CLIP loss, with speed/fidelity controlled by the patch-variance threshold (Kim et al., 19 Feb 2026).
• Time series forecasting: TimeSqueeze attains up to 20x faster convergence, 8x higher data efficiency, and 3.4x lower memory versus point-wise tokenization, with near-identical error (Ankireddy et al., 11 Mar 2026); Kairos achieves state-of-the-art zero-shot MASE/CRPS in compact models (Feng et al., 30 Sep 2025).
• Tensor train computations: For localizable targets, adaptive patching cuts both memory and contraction cost by factors of 5–10x or more, with global error budgets never exceeded (Grosso et al., 25 Feb 2026).
• PatchSelector-based object detection and PatchScaler super-resolution: In DPR, patch-wise selection and selective refinement yield 77.2% compute savings at nearly 9-fold mAP improvement for small objects (Zhang et al., 2023). In PatchScaler, groupwise shortcut sampling achieves ~4x speedup at minor (1 dB) PSNR loss, with improved perceptual metrics (Liu et al., 2024).

Analytically, per-layer attention cost reduces from $O(T^2)$ (point tokens) to $O((T/m)^2)$ for average patch size $m$; similar quadratic (or higher) reductions hold for blocked PDE surrogates and tensor trains.

5. Adaptive Scale Selection Policies and Practical Guidelines

Adaptive scale selection—dynamic determination of patch size—is realized in several modes:

• Content and cost-driven tuning at inference: In DPAR and DDiT, entropy/variance thresholds are dialed up or down at inference without retraining, directly trading compute for fidelity (Srivastava et al., 26 Dec 2025, Kim et al., 19 Feb 2026).
• Budget or precision-constrained selection: In CKM/CSM-based frameworks, target budgets or error tolerances are met by consulting precomputed cost/error tables over allowed patch sizes (Mukhopadhyay et al., 12 Jul 2025).
• Online reinforcement or variational selection: In RL-based approaches, the scale policy is explicitly optimized for task reward under current context, with reward design tailored to both local and global objectives (Liu et al., 2023).
• Confidence/difficulty-based branching: In PatchScaler and DPR, difficulty/confidence maps segment the input into groups or classes receiving tailored sample/super-resolution schedules, maximizing efficiency for "easy" and accuracy for "hard" patches (Liu et al., 2024, Zhang et al., 2023).

Typical hyperparameter setting in practice is governed by validation curves of average patch length, downstream quality (FID, mAP, MASE), and the explicit cost limits of the underlying hardware or application environment.

6. Limitations, Open Problems, and Application Domains

While dynamic patching and adaptive scale selection yield substantial efficiency and fidelity benefits across diverse domains, remaining limitations include:

• Uniformly high-complexity signals: When information density is high everywhere (e.g., textures), little patch merging is possible and computational gains diminish (e.g., DPAR degenerates to full token cost) (Srivastava et al., 26 Dec 2025).
• Complexity of online optimization: Some methods (e.g., QTT patching) can suffer from combinatorial overhead if a proliferation of non-mergeable small patches occurs (Grosso et al., 25 Feb 2026).
• Discrete patch size grids: In most vision, physics, or time-series models, only a finite (often small) menu of patch sizes is supported, potentially leaving gains on the table relative to fully continuous or mask-based adaptive tokenization (Mukhopadhyay et al., 12 Jul 2025, Kim et al., 19 Feb 2026).
• Cross-domain patch-size fusion: Many frameworks treat spatial, temporal, and frequency scales separately; simultaneous adaptation in all axes remains a technical challenge, motivating further research on multi-dimensional adaptive tokenization (Feng et al., 30 Sep 2025).

Current and emerging applications include high-resolution image synthesis, video and volumetric streaming, sparse and dense PDE surrogates, gridless time series forecasting, robust real-world adversarial attacks, remote sensing semantic segmentation, super-resolution, early object detection in resource-constrained settings, and large-scale quantum/computational physics.

7. Representative Results and Empirical Benchmarks

Domain or Model Family	Dynamic Patching Result	Adaptive Scale Policy
DPAR (Visual AR generation)	27% FID improvement, 2x token reduction (Srivastava et al., 26 Dec 2025)	Entropy threshold adjusted at inference
DDiT (Diffusion Transformers)	2–3.5x speedup, <1% FID/CLIP degradation (Kim et al., 19 Feb 2026)	Patch-variance threshold and per-step sizing
PatchScaler (Diffusion SR)	4x speedup, perceptual ManIQA +0.11 (Liu et al., 2024)	Patch grouping by confidence and step schedule
PDPDEs (CKM/CSM ViT surrogates)	5–10x speedup, 50% rollout improvement (Mukhopadhyay et al., 12 Jul 2025)	Budget/error-constrained lookup
TimeSqueeze (Time Series)	20x faster convergence, 8x data saving (Ankireddy et al., 11 Mar 2026)	Difference threshold, per-sample
Kairos (Time Series)	SOTA zero-shot MASE/CRPS with small models (Feng et al., 30 Sep 2025)	Spectral entropy + router+fusion
QTT (Tensor Trains)	5–10x contraction speedup, error bounded (Grosso et al., 25 Feb 2026)	Pivot-cost splitting per patch
DPR (Patch-based OD)	77% compute savings, mAP ↑ 1.03→8.93 (Zhang et al., 2023)	Patch-scores & scale-fusion, upsample/max

These results collectively establish dynamic patching and adaptive scale selection as central pillars for next-generation compute-adaptive, robust, and high-fidelity foundation models across diverse domains.