Patch-Level Neighboring Relations

• Patch-level neighboring relation is a formalism that defines how image or data patches relate via spatial overlap, feature similarity, or graph connectivity.
• It serves as a structural prior to enforce local continuity, context transfer, and regularization in tasks like self-supervised learning, segmentation, and anomaly detection.
• Applications across vision, graphics, and privacy demonstrate improved performance metrics through techniques such as patch-based synthesis, K-NN regularization, and differential privacy amplification.

Searching arXiv for recent and foundational papers on patch-level neighboring relations across vision, graphics, SSL, medical imaging, and privacy. Patch-level neighboring relation denotes a family of formalisms in which an image patch, tokenized patch, point-cloud patch, or spatio-temporal patch is related to other patches by spatial adjacency, overlap, feature similarity, graph connectivity, or localized substitution. Across the literature, it is not a single fixed construct but a recurring device for imposing local continuity, transferring context, regularizing representations, or tightening statistical guarantees. It appears in patch-based synthesis, self-supervised ViT training, CNN interpretability, weakly supervised change detection, anomaly detection, whole-slide segmentation, 3D pose voting, cross-view retrieval, and differential privacy (Yun et al., 2022, Sun et al., 2019, Chen et al., 2015, Durmaz et al., 25 Mar 2026).

1. Conceptual scope

The surveyed work suggests that patch-level neighboring relation is best understood as a structural prior: a patch is not treated in isolation, but as an element of a local or semi-local system whose neighboring elements constrain its representation, label, synthesis, or retrieval score. Different papers instantiate this prior differently.

Formulation	Representative definition	Representative use
Overlap relation	Two patches are neighbors if they share a non-empty overlap (Fish et al., 2020)	Seamless patch synthesis
Spatial local neighborhood	An 8-connected $3\times 3$ window around a patch (Yun et al., 2022)	Positive selection in SSL
Complete pairwise relation	Every ordered pair of patches is related on a complete directed graph (Camporese et al., 2022)	Spatial/geometric pretext tasks
Cross-sample K-NN	A patch is linked to K nearest patches from peer images (Sun et al., 2019)	Graph-based regularization
Offset-preserving smoothness	Neighboring query patches tend to match neighboring training patches (Fragoso et al., 2017)	PatchMatch-style correspondence
Grid graph adjacency	Non-overlapping WSI patches are nodes connected by spatial adjacency (Su, 13 May 2025)	Multi-hop context aggregation
Localized substitution	Two datasets differ only inside a rectangular image patch (Durmaz et al., 25 Mar 2026)	DP amplification under cropping

A central consequence is that “neighboring” is domain-dependent. In some works it means literal spatial adjacency on a lattice; in others it means feature-space affinity inside a restricted window, reciprocal neighborhood in a database, or even a privacy adjacency relation defined over datasets rather than over tokens. This suggests that the term is methodological rather than taxonomic: it names a way of constraining computation around patch-local structure rather than a single algorithmic template.

2. Formalizations of neighborhood structure

In patch-based sketch stylization, neighboring relation is defined by overlap. At inference, the sketch is partitioned into overlapping patches of size $p=64$ with overlap $o=16$ and stride $p-o=48$. A graph $G=(V,E)$ is formed over non-empty patches, with edges between patches whose overlap region is non-empty. A Breadth First Search traversal then ensures that, except for a random root patch, each patch inherits styled border content from already translated neighbors (Fish et al., 2020). The relation is therefore geometric and asymmetric in execution order, even though overlap itself is symmetric.

In SelfPatch for ViTs, the relation is first restricted spatially and then refined in feature space. For a query patch $x^{(i)}$, candidate neighbors $N^{(i)}$ are the 8-connected spatial neighbors in a $3\times 3$ neighborhood. Positives are the top-$k$ candidates ranked by cosine similarity on current student features,

$$s(i,j)=\frac{f_\theta^{(i)}(x)^\top f_\theta^{(j)}(x)}{\|f_\theta^{(i)}(x)\|_2\|f_\theta^{(j)}(x)\|_2}.$$

The selected set $p=64$0 is dynamic, view-specific, and asymmetric: $p=64$1 does not imply $p=64$2 (Yun et al., 2022). The neighboring relation is thus neither purely geometric nor purely semantic; it is a two-stage local-semantic construction.

RelViT generalizes the notion further by defining relations on the complete directed graph of all patch pairs. For patch coordinates $p=64$3 and $p=64$4, the relative spatial relation $p=64$5 uses the coarse categories $p=64$6 and $p=64$7, producing 9 spatial classes. RelViT additionally defines pairwise Euclidean distance $p=64$8 and angle $p=64$9, both normalized to $o=16$0, and learns these for all pairs rather than only immediate neighbors (Camporese et al., 2022). In this formulation, “neighboring” expands to a global pairwise relational geometry.

CompNN uses a different formalization grounded in smoothness of CNN embeddings. If a query patch centered at $o=16$1 matches a training patch centered at $o=16$2, then a spatially adjacent query patch $o=16$3 is expected to match near $o=16$4, namely $o=16$5. This local offset preservation underlies HyperPatch-Match propagation and makes neighbor relations operational in approximate nearest-neighbor search (Fragoso et al., 2017).

For video anomaly detection, PSTRP decouples neighboring relation into spatial and temporal parts. Spatial neighbors are adjacent patches in a four-neighborhood, quantified through Canberra-distance-based edge differences, while temporal neighbors are patches along the same ROI track whose relation is measured by cosine distance. The method then predicts both order matrices and pairwise distance matrices, so neighborhood is encoded jointly as adjacency and recoverable relational structure (Shen et al., 2024).

3. Learning objectives and architectural mechanisms

A major line of work uses neighboring relations as self-supervision. SelfPatch augments DINO with a patch-level teacher-student objective,

$o=16$6

where $o=16$7 is an attention-based aggregation of teacher features over the selected positives $o=16$8. The total loss is $o=16$9 with $p-o=48$0 by default (Yun et al., 2022). The relation is not merely observed; it is distilled into patch embeddings.

RelViT instead supervises all patch tokens with explicit relational heads. Spatial relation classification, distance regression, angle regression, and absolute position classification are all aggregated into

$p-o=48$1

with the $p-o=48$2 values set to $p-o=48$3 in practice. Because all $p-o=48$4 patch pairs contribute targets, the model turns pairwise neighborhood structure into dense supervision on the transformer output tokens (Camporese et al., 2022).

Patch-level Neighborhood Interpolation constructs K-NN graphs over patches clipped from inputs or hidden feature maps and interpolates each patch with its neighbors:

$p-o=48$5

This same neighboring relation underlies both Pani VAT, where the interpolation coefficients are adversarially optimized under a quadratic constraint, and Pani MixUp, where they are sampled and normalized against a Beta-distributed mixing ratio (Sun et al., 2019). Here neighborhood is a regularization substrate rather than a target.

MS-Former uses patch-level annotation in a different way. It crops bi-temporal images into non-overlapping paired patches, assigns each pair a binary patch label, and supervises a change map through adaptive max pooling:

$p-o=48$6

Its Bi-Directional Attention Block updates memory prototypes from patch-derived local and pyramid-global contexts and then injects the refined prototypes back into pixel tokens. The paper is explicit that it does not define an adjacency graph or CRF-like local regularizer; neighboring relations are handled implicitly through pooling hierarchies, global attention, and patch-level supervision (Li et al., 2023).

GNCAF uses an explicit graph. WSI patches become nodes, self-loops are added, and the normalized adjacency is

$p-o=48$7

A multi-layer GCN propagates information from one-hop to $p-o=48$8-hop neighbors, and the hop-wise target-node embeddings are concatenated and compressed into a single context token,

$p-o=48$9

This token is fused with local tokens by multi-head self-attention before decoding (Su, 13 May 2025). The neighboring relation is therefore both explicit, through graph structure, and target-centric, through the context token.

4. Inference-time exploitation and composition

Patch-level neighboring relation is often most visible at inference. In SketchPatch, training uses hybrid patches with styled borders and plain interiors so that the generator learns to continue patterns inward from border cues. At test time, already translated overlaps from neighboring patches are pasted into the border regions of the current plain patch, reproducing the conditioning signal seen during training. No explicit seam loss, border continuity penalty, feathering, or Poisson blending is introduced; continuity emerges from overlap, traversal order, and border-conditioned synthesis (Fish et al., 2020).

HyHDRNet uses local patch neighborhoods inside shifted windows. Reference patch queries and non-reference patch keys are matched by scaled dot-product attention with relative position bias,

$G=(V,E)$0

This patch aggregation branch is paired with a pixel-level ghost attention branch and a learned gating module that cross-modulates both streams. The method uses patch-level neighbors to retrieve off-position content in saturated or misaligned regions and pixel-level attention to suppress ghosted pixels (Yan et al., 2023).

In 3DPVNet, neighboring relation is defined over local 3D patches centered at sampled points. For a center patch $G=(V,E)$1, the neighbor set $G=(V,E)$2 is given by Euclidean $G=(V,E)$3-NN on patch centers. Neighbor features are weighted by

$G=(V,E)$4

stochastically gated, and max-pooled into a semi-global descriptor. That descriptor is then mapped to a local transform, producing a patch-level 6D pose vote (Liu et al., 2020). Neighboring relation here stabilizes voting under noise and occlusion by moving from strictly local evidence to semi-global context.

CompNN operationalizes neighboring relation through propagation in a nearest-neighbor field. Using cosine distance between hyperpatch embeddings, it reconstructs query inputs and outputs by copying training patches from matched locations, while HyperPatch-Match proposes new candidates from already matched left and upper neighbors. The local smoothness of CNN feature embeddings is what makes this propagation effective rather than brittle (Fragoso et al., 2017).

In aerial-ground LiDAR retrieval, neighboring relation reappears after representation learning. Expanded Reciprocal re-ranking first defines reciprocal neighbors from a top-$G=(V,E)$5 retrieval list, then expands one hop:

$G=(V,E)$6

Features are refined by uniform averaging over expanded reciprocal sets, and final query-database scores aggregate bilateral neighbor evidence. The relation is therefore database-level and post-hoc, rather than token-level or train-time (Yang et al., 17 Jun 2026).

5. Applications and empirical behavior

In self-supervised dense prediction, local patch relations improve downstream ViT features. SelfPatch increases DINO performance from $G=(V,E)$7 to $G=(V,E)$8 in $G=(V,E)$9 on COCO object detection, from $x^{(i)}$0 to $x^{(i)}$1 in $x^{(i)}$2 on instance segmentation, and from $x^{(i)}$3 to $x^{(i)}$4 in ADE20K mIoU on semantic segmentation (Yun et al., 2022). RelViT likewise reports consistent improvements on small datasets, including gains of $x^{(i)}$5 on CIFAR-10, $x^{(i)}$6 on CIFAR-100, and $x^{(i)}$7 on ImageNet-100 in the SSL pre-train plus fine-tune setting (Camporese et al., 2022).

In weak or structured supervision, the effect is similarly pronounced. MS-Former reports that performance improves as patch size decreases on BCDD, with F1 values of $x^{(i)}$8, $x^{(i)}$9, $N^{(i)}$0, and $N^{(i)}$1 at patch sizes $N^{(i)}$2, $N^{(i)}$3, $N^{(i)}$4, and $N^{(i)}$5, respectively, and surpasses TransWCD-DL by $N^{(i)}$6 IoU and $N^{(i)}$7 F1 at $N^{(i)}$8 patch size (Li et al., 2023). GNCAF reports improvements up to $N^{(i)}$9 in mF1 and $3\times 3$0 in mIoU for TLS semantic segmentation, while also transferring to lymph node metastasis segmentation with mF1 $3\times 3$1 and mIoU $3\times 3$2 (Su, 13 May 2025).

For anomaly detection and retrieval, neighborhood modeling affects both ranking quality and robustness. PSTRP achieves AUROC values of $3\times 3$3 on UCSD Ped2, $3\times 3$4 on Avenue, and $3\times 3$5 on ShanghaiTech, with ablations showing additive gains from distance-constraint supervision and object optimization (Shen et al., 2024). In aerial-ground LiDAR place recognition, patch-level self-supervision improves average Recall@1 by $3\times 3$6 and average Recall@1\% by $3\times 3$7 on CS-Urban-Scenes, while Expanded Reciprocal re-ranking adds a further $3\times 3$8 Recall@1 on CS-Campus3D and $3\times 3$9 on CS-Urban-Scenes without additional training (Yang et al., 17 Jun 2026).

In image synthesis and restoration, patch-level relations govern continuity and detail transfer. SketchPatch reports qualitative seam suppression through overlap-conditioned BFS inference rather than quantitative seam metrics (Fish et al., 2020). HyHDRNet reports on the Kalantari test set $k$0, $k$1, and HDR-VDP-2 $k$2, and on the Hu test set $k$3, $k$4, $k$5, $k$6, and HDR-VDP-2 $k$7 (Yan et al., 2023). In 3D perception, 3DPVNet reports average ADD $k$8 on LineMOD, compared with $k$9 for DenseFusion and $$s(i,j)=\frac{f_\theta^{(i)}(x)^\top f_\theta^{(j)}(x)}{\|f_\theta^{(i)}(x)\|_2\|f_\theta^{(j)}(x)\|_2}.$$0 for DeepIM, and its ablation isolates the neighboring fusion effect: ADD rises from $$s(i,j)=\frac{f_\theta^{(i)}(x)^\top f_\theta^{(j)}(x)}{\|f_\theta^{(i)}(x)\|_2\|f_\theta^{(j)}(x)\|_2}.$$1 without neighbors to $$s(i,j)=\frac{f_\theta^{(i)}(x)^\top f_\theta^{(j)}(x)}{\|f_\theta^{(i)}(x)\|_2\|f_\theta^{(j)}(x)\|_2}.$$2 with neighbors but no WNFF and to $$s(i,j)=\frac{f_\theta^{(i)}(x)^\top f_\theta^{(j)}(x)}{\|f_\theta^{(i)}(x)\|_2\|f_\theta^{(j)}(x)\|_2}.$$3 with WNFF (Liu et al., 2020).

6. Theory, limitations, and recurrent points of confusion

A theoretical formulation appears in the latent source model for patch-based segmentation. There, the central local property is the jigsaw condition: for every mixture component at a pixel $$s(i,j)=\frac{f_\theta^{(i)}(x)^\top f_\theta^{(j)}(x)}{\|f_\theta^{(i)}(x)\|_2\|f_\theta^{(j)}(x)\|_2}.$$4, there exists a neighbor $$s(i,j)=\frac{f_\theta^{(i)}(x)^\top f_\theta^{(j)}(x)}{\|f_\theta^{(i)}(x)\|_2\|f_\theta^{(j)}(x)\|_2}.$$5 whose mixture model contains a component with the same mean and label. Under this condition, and with user neighborhoods $$s(i,j)=\frac{f_\theta^{(i)}(x)^\top f_\theta^{(j)}(x)}{\|f_\theta^{(i)}(x)\|_2\|f_\theta^{(j)}(x)\|_2}.$$6, the expected pixel labeling error of pointwise nearest neighbor and weighted majority voting is bounded by

$$s(i,j)=\frac{f_\theta^{(i)}(x)^\top f_\theta^{(j)}(x)}{\|f_\theta^{(i)}(x)\|_2\|f_\theta^{(j)}(x)\|_2}.$$7

where $$s(i,j)=\frac{f_\theta^{(i)}(x)^\top f_\theta^{(j)}(x)}{\|f_\theta^{(i)}(x)\|_2\|f_\theta^{(j)}(x)\|_2}.$$8 is the cross-label separation gap in patch space (Chen et al., 2015). In this setting, neighboring relation is not a heuristic convenience but the condition that makes nonparametric patch voting analytically tractable.

A different theoretical use appears in differential privacy. Under patch-level substitution adjacency, two datasets differ only inside a rectangular patch $$s(i,j)=\frac{f_\theta^{(i)}(x)^\top f_\theta^{(j)}(x)}{\|f_\theta^{(i)}(x)\|_2\|f_\theta^{(j)}(x)\|_2}.$$9 of one image. Random cropping then induces a worst-case patch inclusion probability $p=64$00, and composition with minibatch subsampling yields an effective sampling rate

$p=64$01

The privacy amplification theorem states that the per-step privacy profile is controlled by $p=64$02 rather than the record-level sampling rate $p=64$03, so random cropping amplifies privacy “for free” whenever $p=64$04 (Durmaz et al., 25 Mar 2026). The neighboring relation here is neither geometric adjacency nor semantic similarity; it is a localized neighboring relation on datasets.

Several recurring misconceptions are contradicted by the literature. First, neighboring relation need not mean immediate 4- or 8-connected spatial adjacency: RelViT uses a complete graph over all patch pairs, and Pani uses K-NN graphs across peer images rather than within-image neighborhoods (Camporese et al., 2022, Sun et al., 2019). Second, seamlessness or consistency need not be enforced by explicit seam or boundary losses: SketchPatch obtains continuity through hybrid border conditioning and BFS ordering without explicit continuity penalties (Fish et al., 2020). Third, explicit graphs are not mandatory: MS-Former handles neighboring relations implicitly through adaptive pooling, global memory prototypes, and attention, while GNCAF makes the graph explicit and multi-hop (Li et al., 2023, Su, 13 May 2025).

The same literature also shows that locality is beneficial but not automatically optimal. SelfPatch reports that widening the candidate set from a $p=64$05 neighborhood to all patches severely degrades performance, with DAVIS $p=64$06 dropping from $p=64$07 to $p=64$08 in the relevant ablation, whereas GNCAF finds that gains plateau or vary slightly beyond $p=64$09 to $p=64$10 hops (Yun et al., 2022, Su, 13 May 2025). A plausible implication is that patch-level neighboring relation functions best when it captures semantically coherent local structure without collapsing into indiscriminate global mixing.