Grid Partitioned Attention (GPA)

• The paper’s main contribution is reformulating attention in image domains by partitioning the query grid and adaptively pruning keys to drastically reduce memory complexity.
• It employs a coarse-to-fine strategy by downsampling queries and keys to predict fine-scale support, making high-resolution conditional image generation computationally feasible.
• Empirical results demonstrate GPA’s superior performance on benchmarks, with improved FID scores and detail preservation compared to traditional dense attention methods.

Searching arXiv for the cited GPA-related papers and terminology to ground the article in current records. Grid Partitioned Attention (GPA) is an approximate attention algorithm for image domains that makes attention practical at high spatial resolutions by combining structured sparsity with an image-domain inductive bias. In its original formulation, GPA does not approximate the attention values everywhere; instead, it predicts which keys are relevant for groups of nearby queries at a coarse resolution and then computes exact attention only over those selected keys at the original resolution. The defining assumption is that spatially close queries tend to have similar relevant keys, and that coarse-scale attention structure roughly predicts fine-scale attention structure (Jetchev et al., 2021).

1. Definition and conceptual basis

GPA was introduced for high-resolution detail generation, especially conditional image generation tasks such as pose morphing, where dense all-to-all attention is prohibitive in memory and compute (Jetchev et al., 2021). The method starts from the observation that for an image of spatial size $h \times w$, the number of positions is $n = hw$, so standard attention requires an $n \times n$ affinity matrix. The paper gives the concrete example that at $128 \times 128$, $n = 2^{14}$, so the attention matrix has $2^{28}$ entries; at larger resolutions this becomes impractical in GPU memory (Jetchev et al., 2021).

The baseline full attention used in the GPA paper is

$$X=\mathcal{A}(Q,K,V)=V\cdot \mathcal{S}(Q,K)=V\cdot \mathrm{softmax}(K^\top Q),$$

with flattened tensors

$$Q,K\in\mathbb{R}^{c_k\times n},\qquad V\in\mathbb{R}^{c_v\times n},\qquad n=hw,$$

and dense affinity matrix

$A=\mathcal{S}(Q,K)\in\mathbb{R}^{n\times n}.$

GPA replaces this dense pattern with a two-part structure: queries are partitioned into regular spatial cells, and each query cell is assigned only a small selected set of keys (Jetchev et al., 2021).

The “grid partitioned” component refers to this spatial partitioning of the query grid. The “attention” component remains exact within the selected support: queries in the same partition cell share the same candidate key set, but they do not share the same attention weights. This is a central distinction from centroid-based clustering schemes, because GPA groups queries only for support selection, not for collapsing their individual attention distributions (Jetchev et al., 2021).

The method is therefore sparse in two senses. First, nearby queries are grouped by a fixed grid-aligned partition. Second, keys are pruned adaptively by a coarse relevance estimate. The sparsity is neither random nor purely low-rank; it is a structured support-selection mechanism derived from image geometry and coarse-to-fine attention transfer (Jetchev et al., 2021).

2. Algorithmic formulation

The GPA paper defines three foundational operators: downsampling, upsampling of index sets, and spatial partitioning (Jetchev et al., 2021).

For $X\in\mathbb{R}^{c\times h\times w}$, with $n = hw$0 divisible by $n = hw$1, downsampling is

$n = hw$2

where

$n = hw$3

For an index structure $n = hw$4, upsampling of index sets is

$n = hw$5

A partitioning operator

$n = hw$6

splits a tensor $n = hw$7 into $n = hw$8 disjoint cells $n = hw$9 whose union is the full spatial index set. In practice, the paper describes this as regular square-cell partitioning of the query grid (Jetchev et al., 2021).

The relevance operator $n \times n$0 selects the $n \times n$1 most relevant key indices for a query set $n \times n$2: $n \times n$3 where $n \times n$4 and selected indices satisfy

$n \times n$5

This means that within one query cell, GPA selects keys with the highest average activation across the entire local query group (Jetchev et al., 2021).

The algorithm proceeds in two phases.

At low resolution, queries and keys are downsampled: $n \times n$6 Dense attention is then computed on the coarse grid: $n \times n$7 The coarse query grid is partitioned into $n \times n$8 cells, and for each cell the top-$n \times n$9 relevant coarse keys are selected (Jetchev et al., 2021).

At full resolution, the query partition is made consistent with the coarse partition,

$128 \times 128$0

and the selected key sets are upsampled,

$128 \times 128$1

For each partition cell $128 \times 128$2, GPA gathers only the selected keys and values and computes

$128 \times 128$3

The outputs are then recomposed over the grid (Jetchev et al., 2021).

Operationally, a query at full resolution belongs to some partition cell $128 \times 128$4 and can attend only to keys in $128 \times 128$5. All queries in the same cell share this candidate support, but each query still computes its own attention distribution over that support. This design implements the paper’s main inductive bias: nearby queries should have similar relevant-key sets (Jetchev et al., 2021).

3. Complexity, tunable parameters, and inductive bias

The GPA paper frames the method as a tunable trade-off between memory usage and model power (Jetchev et al., 2021). Full attention over $128 \times 128$6 positions scales as $128 \times 128$7. GPA replaces this with two terms: a coarse dense stage controlled by the downsampling factor $128 \times 128$8, and a sparse fine stage controlled by the number of selected keys $128 \times 128$9.

For the low-resolution phase, the paper states a complexity of

$n = 2^{14}$0

For the sparse fine-resolution phase, if there are $n = 2^{14}$1 partitions and each partition has $n = 2^{14}$2 queries attending to $n = 2^{14}$3 selected keys, the total memory cost is

$n = 2^{14}$4

The factor $n = 2^{14}$5 cancels because the work is distributed across the $n = 2^{14}$6 cells (Jetchev et al., 2021).

Three hyperparameters control this trade-off:

Parameter	Role	Effect stated in the paper
$n = 2^{14}$7	Downsampling factor for Phase 1	Larger $n = 2^{14}$8 makes the coarse stage cheaper but rougher
$n = 2^{14}$9	Number of query partition cells	Larger $2^{28}$0 gives smaller query cells and weaker smoothness assumption
$2^{28}$1	Relevant coarse keys per query cell	Larger $2^{28}$2 increases capacity and memory/computation

The paper discusses two limiting cases explicitly. If

$2^{28}$3

then after upsampling all $2^{28}$4 keys are considered for all queries, so GPA becomes equivalent to full attention. If

$2^{28}$5

then all queries share a single global relevant-key dictionary, and $2^{28}$6 controls a single image-wide support set (Jetchev et al., 2021).

The method’s inductive bias is stronger than generic sparse attention. It assumes not only sparsity, but also local smoothness of relevance: spatially close queries should attend to similar regions. It also assumes that coarse attention structure is predictive of fine attention structure. The paper identifies white noise as a counterexample to this assumption, which clarifies that GPA is not meant as a universal approximation strategy independent of image statistics (Jetchev et al., 2021).

4. Use in high-resolution conditional image generation

In the original work, GPA is embedded in a deep attention-copying architecture for pose morphing (Jetchev et al., 2021). The architecture consists of two U-Nets: a source branch that receives the source appearance image $2^{28}$7 and source pose $2^{28}$8, and a target branch that receives the target pose $2^{28}$9 and generates the output image $$X=\mathcal{A}(Q,K,V)=V\cdot \mathcal{S}(Q,K)=V\cdot \mathrm{softmax}(K^\top Q),$$0. The copying blocks are placed in decoder layers.

Within each copying block, the source decoder produces keys and values, and the target decoder produces queries. Each block contains three trainable $$X=\mathcal{A}(Q,K,V)=V\cdot \mathcal{S}(Q,K)=V\cdot \mathrm{softmax}(K^\top Q),$$1 convolutions to project features into query, key, and value tensors. The attention output is concatenated to the target feature map; the paper notes that residual addition or concatenation are both possible, and that concatenation was used in experiments (Jetchev et al., 2021).

This design targets a specific limitation of latent-only conditional generation. Pose morphing requires preservation of high-frequency source details while moving them to new spatial configurations. The paper contrasts GPA-based copying with optical-flow-style warping, arguing that flow is extremely local and difficult to optimize under strong deformations, whereas attention copying can transfer information from semantically corresponding but spatially displaced regions and can combine multiple source locations (Jetchev et al., 2021).

The architecture is “deep” because attention-copying modules are inserted at multiple decoder resolutions. On DeepFashion at $$X=\mathcal{A}(Q,K,V)=V\cdot \mathcal{S}(Q,K)=V\cdot \mathrm{softmax}(K^\top Q),$$2, the appendix reports the following decoder schedule (Jetchev et al., 2021):

Resolution	Channels	Attention type
$$X=\mathcal{A}(Q,K,V)=V\cdot \mathcal{S}(Q,K)=V\cdot \mathrm{softmax}(K^\top Q),$$3	640	full attention
$$X=\mathcal{A}(Q,K,V)=V\cdot \mathcal{S}(Q,K)=V\cdot \mathrm{softmax}(K^\top Q),$$4	640	full attention
$$X=\mathcal{A}(Q,K,V)=V\cdot \mathcal{S}(Q,K)=V\cdot \mathrm{softmax}(K^\top Q),$$5	640	full attention
$$X=\mathcal{A}(Q,K,V)=V\cdot \mathcal{S}(Q,K)=V\cdot \mathrm{softmax}(K^\top Q),$$6	640	full attention
$$X=\mathcal{A}(Q,K,V)=V\cdot \mathcal{S}(Q,K)=V\cdot \mathrm{softmax}(K^\top Q),$$7	320	GPA, $$X=\mathcal{A}(Q,K,V)=V\cdot \mathcal{S}(Q,K)=V\cdot \mathrm{softmax}(K^\top Q),$$8
$$X=\mathcal{A}(Q,K,V)=V\cdot \mathcal{S}(Q,K)=V\cdot \mathrm{softmax}(K^\top Q),$$9	160	GPA, $$Q,K\in\mathbb{R}^{c_k\times n},\qquad V\in\mathbb{R}^{c_v\times n},\qquad n=hw,$$0

All GPA layers are downsampled to $$Q,K\in\mathbb{R}^{c_k\times n},\qquad V\in\mathbb{R}^{c_v\times n},\qquad n=hw,$$1 in Phase 1; at that size, the query grid is partitioned into $$Q,K\in\mathbb{R}^{c_k\times n},\qquad V\in\mathbb{R}^{c_v\times n},\qquad n=hw,$$2 cells, each cell has spatial size $$Q,K\in\mathbb{R}^{c_k\times n},\qquad V\in\mathbb{R}^{c_v\times n},\qquad n=hw,$$3, and $$Q,K\in\mathbb{R}^{c_k\times n},\qquad V\in\mathbb{R}^{c_v\times n},\qquad n=hw,$$4 relevant keys are selected per cell. This is the “pivot configuration,” denoted GPA1024_12_64 in the experiments (Jetchev et al., 2021).

For $$Q,K\in\mathbb{R}^{c_k\times n},\qquad V\in\mathbb{R}^{c_v\times n},\qquad n=hw,$$5 generation, full attention is used up to $$Q,K\in\mathbb{R}^{c_k\times n},\qquad V\in\mathbb{R}^{c_v\times n},\qquad n=hw,$$6, and GPA is used at $$Q,K\in\mathbb{R}^{c_k\times n},\qquad V\in\mathbb{R}^{c_v\times n},\qquad n=hw,$$7, $$Q,K\in\mathbb{R}^{c_k\times n},\qquad V\in\mathbb{R}^{c_v\times n},\qquad n=hw,$$8, and $$Q,K\in\mathbb{R}^{c_k\times n},\qquad V\in\mathbb{R}^{c_v\times n},\qquad n=hw,$$9, with Phase 1 always at $A=\mathcal{S}(Q,K)\in\mathbb{R}^{n\times n}.$0. In that setting, the query is partitioned into

$A=\mathcal{S}(Q,K)\in\mathbb{R}^{n\times n}.$1

single-pixel cells at the coarse scale (Jetchev et al., 2021).

The training objective combines perceptual paired loss, self-supervised copying loss, and GAN loss: $A=\mathcal{S}(Q,K)\in\mathbb{R}^{n\times n}.$2 The perceptual paired loss is

$A=\mathcal{S}(Q,K)\in\mathbb{R}^{n\times n}.$3

and the self-supervised copying loss is

$A=\mathcal{S}(Q,K)\in\mathbb{R}^{n\times n}.$4

where affine transforms are sampled by adding

$A=\mathcal{S}(Q,K)\in\mathbb{R}^{n\times n}.$5

to the identity transform (Jetchev et al., 2021).

5. Empirical performance and observed behavior

The GPA paper reports results on pose morphing benchmarks including DeepFashion, a ChaL / CLOTH3D-related benchmark, and a high-resolution fashion dataset from Yildirim et al. (Jetchev et al., 2021). The central empirical claim is that GPA improves the memory/quality trade-off relative to low-rank approximation baselines while preserving high-frequency appearance detail.

On the DeepFashion benchmark, the reported table is (Jetchev et al., 2021):

Method	FID	SSIM
Attention Copying + GPA	8.86	0.767
Attention Copying + NO	13.05	0.752
BFT	14.5	0.766
DSC	19.9	0.761
XING	42.8	0.756

The same table also reports R2G and G2R scores, with Attention Copying + GPA at $A=\mathcal{S}(Q,K)\in\mathbb{R}^{n\times n}.$6 and $A=\mathcal{S}(Q,K)\in\mathbb{R}^{n\times n}.$7, respectively (Jetchev et al., 2021). The paper characterizes these results as new state of the art on DeepFashion pose morphing, emphasizing FID and human perceptual scores over SSIM (Jetchev et al., 2021).

In a self-supervised affine copying benchmark on DeepFashion, GPA around the pivot configuration is reported as giving the best copying performance among compared variants, including no attention above $A=\mathcal{S}(Q,K)\in\mathbb{R}^{n\times n}.$8, Nyströmformer, and Performer. The paper also states that GPA used less memory than Performer and Nyströmformer in the compared setup, and that moving the Phase-1 resolution away from the $A=\mathcal{S}(Q,K)\in\mathbb{R}^{n\times n}.$9 pivot made GPA more memory costly and slower (Jetchev et al., 2021).

At higher resolution on ChaL at $X\in\mathbb{R}^{c\times h\times w}$0, the paper ablates

$X\in\mathbb{R}^{c\times h\times w}$1

The reported finding is that increasing $X\in\mathbb{R}^{c\times h\times w}$2 increases memory usage but also improves FID and visual quality, with richer and sharper qualitative outputs (Jetchev et al., 2021). This directly supports the intended role of $X\in\mathbb{R}^{c\times h\times w}$3 as the efficiency/expressiveness control.

Qualitative observations in the paper further describe GPA as effective for transferring stripes, pockets, ripped jeans, logos, and cherries, with less blur than low-rank approximations and with the capacity to copy rare or unique details, including out-of-sample tests such as copying from a painting-domain source into a target pose (Jetchev et al., 2021). The visualized key contributions show heterogeneous behavior: some queries are nearly local and dominated by one key, while others aggregate broader regions; for occluded target parts, GPA may copy from semantically different but visually similar regions, such as using leg skin-color features to fill a hand region (Jetchev et al., 2021).

6. Relation to neighboring “grid” and grouped-attention ideas

The term “Grid Partitioned Attention” is also useful as a comparative lens for methods that use grid-like organization without implementing the original GPA operator. Three papers are especially relevant in this respect: Qffusion’s Quadrant-grid Arrangement, the QKVA grid perspective in Stacked-DETR, and Grouped Differential Attention (Li et al., 11 Jan 2025, Sheng, 2022, Lim et al., 8 Oct 2025).

Qffusion’s “Quadrant-grid Arrangement” (QGA) is GPA-relevant but not an explicit GPA operator (Li et al., 11 Jan 2025). The method partitions latent input into a $X\in\mathbb{R}^{c\times h\times w}$4 spatial grid and arranges two reference-image latents and four condition latents into quadrant-aligned positions. For the appearance branch, the arranged quadrants are

$X\in\mathbb{R}^{c\times h\times w}$5

and for the condition branch they are

$X\in\mathbb{R}^{c\times h\times w}$6

The fused tensor is

$X\in\mathbb{R}^{c\times h\times w}$7

where $X\in\mathbb{R}^{c\times h\times w}$8 denotes channel-wise concatenation, and if each latent has size $X\in\mathbb{R}^{c\times h\times w}$9, then

$n = hw$00

Here the partitioning is spatial and explicit, but the attention mechanism is inherited unchanged from Stable Diffusion’s denoising U-Net. The method therefore uses partitioned input with global self-attention, not a partition-aware attention operator. The paper explicitly emphasizes that “only the input format of Stable Diffusion is modified” and that no additional networks or complex training stages are introduced (Li et al., 11 Jan 2025).

This distinction matters because GPA, in the original sense, restricts candidate key support per query cell. QGA does not modify the attention graph in that way. Its quadrants bias semantics and temporal structure, but there is no masking, no local-only windowing, and no modified attention formula. A plausible implication is that QGA is best understood as a grid-partitioned latent packing scheme rather than a true GPA layer (Li et al., 11 Jan 2025).

The “QKVA grid” in Stacked-DETR is more distant still (Sheng, 2022). It decomposes attention algebraically through

$n = hw$01

and summarizes the output as

$n = hw$02

Its “grid” is a conceptual geometric visualization of standard attention over image-derived features, not a partitioning operator. The paper explicitly connects this view to window attention in the appendix by noting that selecting certain parts of pixels on the $n = hw$03 dimension can correspond to image squares analogous to Swin Transformer windows, but it does not define block masks, local attention, or sparse inter-partition routing (Sheng, 2022). Its architectural consequence is instead a single-head encoder at one high-resolution feature level inside Stacked-DETR, motivated by memory analysis rather than by grid-restricted attention (Sheng, 2022).

Grouped Differential Attention is related to GPA only in a broad sense of partitioning attention computation, but along a different axis (Lim et al., 8 Oct 2025). It partitions attention heads into signal-preserving and noise-control groups with an asymmetric ratio

$n = hw$04

and reuses the smaller noise-control group in a way described as akin to Grouped Query Attention. Its head output is formed by subtracting a group-level noise map from a per-head signal map. This is head-space partitioning rather than token-grid partitioning, and the paper does not claim a change in the $n = hw$05 token-token attention asymptotics (Lim et al., 8 Oct 2025). From a GPA perspective, it is therefore a neighboring grouped-attention design, not a spatial GPA method.

A concise comparison is useful:

Method	What is partitioned	Attention pattern
GPA (Jetchev et al., 2021)	Query grid into spatial cells; keys pruned per cell	Sparse support, exact attention within selected keys
QGA in Qffusion (Li et al., 11 Jan 2025)	Latent input packed into a $n = hw$06 spatial grid	Standard Stable Diffusion self-attention over packed grid
QKVA grid (Sheng, 2022)	No operator-level partition; geometric interpretation only	Standard dense attention
GDA (Lim et al., 8 Oct 2025)	Heads into signal/noise groups	Standard sequence-length scaling, grouped head reuse

7. Limitations, misconceptions, and significance

A common misconception is to treat any method using a grid layout as “GPA.” The literature represented here shows that this is too broad. In the original GPA paper, the essential mechanism is not merely grid organization; it is the coupling of grid-partitioned queries with adaptive key pruning derived from coarse attention (Jetchev et al., 2021). Methods such as Qffusion’s QGA use grid partitioning as an input-layout strategy, while QKVA grid uses “grid” as a conceptual picture of dense attention, and GDA partitions heads rather than tokens (Li et al., 11 Jan 2025, Sheng, 2022, Lim et al., 8 Oct 2025).

The original GPA formulation also has clear limitations. Its top-$n = hw$07 key selection is non-differentiable because Phase 1 selects key indices by sorting, and the paper identifies fully differentiable GPA as future work (Jetchev et al., 2021). The method relies on the assumption that low-resolution attention predicts high-resolution structure and that local groups of queries share similar relevant keys. This can fail for abrupt correspondence changes within a partition cell or for data that violate local spatial correlation assumptions (Jetchev et al., 2021). GPA is also not “free”: it still contains a dense coarse-attention stage, even if this stage is much cheaper than full-resolution dense attention (Jetchev et al., 2021).

Within those constraints, GPA occupies a distinctive place among efficient attention methods. Compared with random sparse patterns such as BigBird, it is explicitly image-structured. Compared with LSH-based approaches such as Reformer, it does not require keys to be partitioned into non-overlapping buckets of similar size and naturally accommodates cross-attention between different pathways. Compared with clustered attention, it preserves query-specific attention weights inside each shared support set. Compared with Nyströmformer and Performer, it is support-selection based rather than low-rank (Jetchev et al., 2021).

Its lasting significance lies in the combination of three elements. First, it makes high-resolution attention feasible enough to insert into decoder stages where fine detail matters most. Second, it introduces an interpretable inductive bias tied to spatial structure rather than relying on generic approximation alone. Third, it demonstrates that coarse-to-fine support prediction can be especially effective in conditional generation settings where copying from one pathway to another is more important than abstract global summarization (Jetchev et al., 2021). This suggests a broader principle: for image generation, attention efficiency can be improved not only by reducing arithmetic complexity, but also by imposing structured assumptions about which correspondences are likely to matter.