Patch-Grid Point Sampling Methods & Applications

• Patch-Grid point sampling is a structured approach that organizes points or pixels via regular grids or patch decompositions for efficient, localized representation.
• It enables local reconstruction and inverse alignment by mapping sparse samples back to detailed spatial structures in images, 3D surfaces, and point clouds.
• Applications include image replay, dense Vision Transformer inference, neural implicit surface reconstruction, and LiDAR semantic segmentation for improved efficiency.

Patch-Grid (PG) point sampling denotes a family of structured sampling, localization, and query schemes in which points, pixels, or dense prediction coordinates are organized by a regular grid, a local patch decomposition, or both. The literature does not use the label in a single canonical way. In image replay, “Grid-based Patch Sampling (GPS)” partitions an image into a uniform grid of non-overlapping square patches and samples one pixel from each patch (Ma et al., 14 Apr 2025). In neural implicit surface modeling, “Patch-Grid” localizes representation and supervision to patch volumes $V_p$ and merge-grid cells $C_i$ (Lin et al., 2023). In dense Vision Transformer inference, Phase Marginalization samples discrete patch-grid phases $\phi=(d_x,d_y)$ of the patch lattice rather than arbitrary image points (Ercan, 6 Jun 2026). In point-cloud segmentation, by contrast, CPGNet uses “Point-Grid (PG),” not Patch-Grid, and its closest sampling operations are point-to-grid pooling and grid-to-point bilinear interpolation (Li et al., 2022).

1. Terminology and conceptual scope

The term “Patch-Grid (PG) point sampling” sits across several adjacent but non-identical literatures. In the most literal image-space sense, GPS is a grid-wise point sampling method over patches: one representative pixel per patch, arranged in original spatial order as a low-resolution surrogate image (Ma et al., 14 Apr 2025). In the most literal 3D shape sense, Patch-Grid organizes point sampling and point querying through two nested spatial structures: patch feature volumes for local signed-distance representation and a merge-grid octree for local composition of multiple patches (Lin et al., 2023). In ViT dense prediction, the sampled object is not a point in the continuous image plane but a structured patch-grid phase, namely the origin or offset of the tokenization lattice (Ercan, 6 Jun 2026).

This terminological variation matters because superficially similar acronyms are not equivalent. CPGNet explicitly defines PG as a Point-Grid fusion mechanism, not as a standalone sampling algorithm (Li et al., 2022). HybridPoint is partially analogous to PG sampling because it mixes salient points and approximately uniform coverage points, but it does not define an explicit patch-grid lattice (Li et al., 2023). GMS-3DQA is analogous in mechanism but samples 2D projection mini-patches from a regular image grid, not 3D points from native geometry (Zhang et al., 2023). One document sometimes associated with sampler comparison contains no proposed sampler, no sampling equations, and no experiments; its relevance is limited to the bibliography it carries (Ouyang et al., 2023).

A plausible implication is that PG point sampling is better treated as a structured design pattern than as a single algorithm. Across the cited work, the recurring pattern is local regularization of otherwise irregular data: patch-local support, cell-local assignment, or finite discrete offsets of a patch lattice.

2. Image-space patch-grid sampling

The clearest explicit formulation of patch-grid point sampling in image space appears in GPS. For an input image $x \in \mathbb{R}^{r \times r \times C}$ and sampling factor $f \in \mathbb{N}$, the image is partitioned into an $r' \times r'$ grid of patches with

$r' = \left\lfloor \frac{r}{f} \right\rfloor .$

From each patch $P_{i,j}$, a single pixel is sampled uniformly,

$$x_{i,j} = P_{i,j}(u,v), \qquad u,v \sim \mathcal{U}(0,f-1),$$

and the sampled outputs are assembled into a compact image $$x^{\mathrm{GPS}} \in \mathbb{R}^{r' \times r' \times C}$$ that preserves patch-grid arrangement (Ma et al., 14 Apr 2025). Under a fixed pixel budget, a memory that stores $C_i$0 full-resolution images can store $C_i$1 GPS-compressed samples. During training replay, same-class GPS samples are grouped in chunks of size $C_i$2 and tiled into an $C_i$3 grid to form a synthetic full-resolution image. During inference in the SCR setup, GPS samples are upsampled by pixel repetition and used with a Nearest Class Mean classifier (Ma et al., 14 Apr 2025).

Phase Marginalization uses a different image-space PG object: the phase of a ViT patch grid. For patch size $C_i$4, the phase is

$C_i$5

A frozen encoder $C_i$6 and dense head $C_i$7 are run on a phase-shifted input $C_i$8, producing

$C_i$9

Outputs are then inverse-aligned and aggregated by

$\phi=(d_x,d_y)$0

The central variant is Uniform Phase Marginalization with $\phi=(d_x,d_y)$1, using

$\phi=(d_x,d_y)$2

for even patch size $\phi=(d_x,d_y)$3 (Ercan, 6 Jun 2026). This is explicitly training-free and post-hoc. The paper frames patch-grid phase as a nuisance variable because changing the patch partition changes the token evidence available to a pixel, especially near boundaries (Ercan, 6 Jun 2026).

GMS-3DQA provides a related projection-space formulation. Each rendered projection $\phi=(d_x,d_y)$4 is divided into a uniform $\phi=(d_x,d_y)$5 grid,

$\phi=(d_x,d_y)$6

with $\phi=(d_x,d_y)$7 in experiments. Candidate mini-patches $\phi=(d_x,d_y)$8 are derived from each grid cell, and a fixed number

$\phi=(d_x,d_y)$9

is randomly selected from each of six projections. For $x \in \mathbb{R}^{r \times r \times C}$0, $x \in \mathbb{R}^{r \times r \times C}$1, so a total of $x \in \mathbb{R}^{r \times r \times C}$2 mini-patches are spliced into a $x \in \mathbb{R}^{r \times r \times C}$3 Quality Mini-Patch Map (Zhang et al., 2023). The method is structurally analogous to PG sampling because it uses regular partition, local patch candidates, sparse balanced selection, and compact aggregation, but it samples 2D projection patches rather than 3D points (Zhang et al., 2023).

3. Point-cloud and point-grid formulations

In point-cloud segmentation, CPGNet defines a Point-Grid fusion block with four steps: Point to Grid (P2G), 2D FCN, Grid to Point (G2P), and Point Fusion (Li et al., 2022). For a 3D point $x \in \mathbb{R}^{r \times r \times C}$4 projected to $x \in \mathbb{R}^{r \times r \times C}$5, the set of points assigned to grid cell $x \in \mathbb{R}^{r \times r \times C}$6 is

$x \in \mathbb{R}^{r \times r \times C}$7

and the grid feature is obtained by channel-wise max pooling,

$x \in \mathbb{R}^{r \times r \times C}$8

After 2D semantic extraction on BEV and RV grids, grid features are transmitted back to points by bilinear interpolation,

$x \in \mathbb{R}^{r \times r \times C}$9

This is the operation in the paper closest to “grid/patch sampling onto points”: a fixed $f \in \mathbb{N}$0 local neighborhood lookup around the projected point location (Li et al., 2022).

GridFormer pushes the point-grid formulation further by making within-cell aggregation learned rather than purely pooled. Points are localized relative to a regular plane or volume grid using

$f \in \mathbb{N}$1

where $f \in \mathbb{N}$2 is the grid resolution (Li et al., 2024). For each grid location $f \in \mathbb{N}$3 and point $f \in \mathbb{N}$4, point-grid attention weights are

$f \in \mathbb{N}$5

and the updated grid feature is

$f \in \mathbb{N}$6

Point features are then updated by combining transformed point features with interpolated grid features (Li et al., 2024). The paper describes the grid as a transfer point connecting space and point cloud, and this makes the grid an active mediator rather than a passive index (Li et al., 2024).

HybridPoint occupies a different position in the design space. Salient points are selected from an intermediate point set $f \in \mathbb{N}$7 using covariance eigenvalue ratios,

$f \in \mathbb{N}$8

with $f \in \mathbb{N}$9, followed by Non-Maximum Suppression (Li et al., 2023). Non-salient points are taken from the downsampled set $r' \times r'$0, which the paper states “can be approximately regarded as uniformly distributed,” and a point $r' \times r'$1 is added if its distance to the salient set satisfies

$r' \times r'$2

The hybrid node set is

$r' \times r'$3

This is not explicit grid partitioning, but it is a coverage-completion mechanism with PG-like intent: repeatable centers where geometry is distinctive, and approximately uniform support where it is not (Li et al., 2023).

4. Patch-Grid for neural implicit surfaces

“Patch-Grid: An Efficient and Feature-Preserving Neural Implicit Surface Representation” provides the most direct 3D shape formulation of PG point sampling (Lin et al., 2023). The method starts from a segmented shape

$r' \times r'$4

where each $r' \times r'$5 is a surface patch. For each patch $r' \times r'$6, a patch volume $r' \times r'$7 is built by computing the axis-aligned bounding box, partitioning it into a regular grid of rectangular or cubic cells, pruning cells that do not enclose any part of $r' \times r'$8, and keeping only nonempty cells (Lin et al., 2023). Grid vertices carry feature vectors

$r' \times r'$9

and a continuous feature field $r' = \left\lfloor \frac{r}{f} \right\rfloor .$0 is defined by trilinear interpolation inside $r' = \left\lfloor \frac{r}{f} \right\rfloor .$1. The patch surface is represented as

$r' = \left\lfloor \frac{r}{f} \right\rfloor .$2

A second spatial structure, the merge grid $r' = \left\lfloor \frac{r}{f} \right\rfloor .$3, is an adaptive octree initialized at the finest patch-volume resolution. Each leaf cell $r' = \left\lfloor \frac{r}{f} \right\rfloor .$4 is intended to contain a simple local patch adjacency configuration, and subdivision stops when the patch adjacency graph in the cell becomes a clique, or the cell is empty, or maximum depth is reached (Lin et al., 2023). This makes point-related operations local in two senses: patch-local for individual SDFs and cell-local for multi-patch merging.

Training-time point sampling is explicitly heterogeneous. Patch losses use surface points $r' = \left\lfloor \frac{r}{f} \right\rfloor .$5, offset near-surface points $r' = \left\lfloor \frac{r}{f} \right\rfloor .$6 with $r' = \left\lfloor \frac{r}{f} \right\rfloor .$7 times the grid-cell size, and off-surface volumetric points $r' = \left\lfloor \frac{r}{f} \right\rfloor .$8 (Lin et al., 2023). The paper states that, in each iteration, it globally samples 10,000 surface points and 10,000 spatial points off the surface and assigns the sampled points to their corresponding cells. Merge supervision is local: $r' = \left\lfloor \frac{r}{f} \right\rfloor .$9 where $P_{i,j}$0 are sampled from the surface patches within the cell $P_{i,j}$1 (Lin et al., 2023). For open surfaces, boundary points $P_{i,j}$2 are sampled from the boundary curve and used with a trimming patch (Lin et al., 2023).

Inference-time point querying is likewise localized. A query point is influenced only by patches whose patch volumes contain it and by patches participating in the same merge-grid leaf cell. The intended logic is: locate $P_{i,j}$3 in a merge-grid leaf cell $P_{i,j}$4, retrieve the set of patches present in $P_{i,j}$5, evaluate $P_{i,j}$6 for relevant patches, and merge the results using a local CSG tree with hard $P_{i,j}$7 operators (Lin et al., 2023). This local support is the central point-query design of Patch-Grid.

5. Algorithmic motifs and computational properties

Across formulations, three algorithmic motifs recur. The first is structured coverage. GPS guarantees one sampled pixel from every $P_{i,j}$8 image patch (Ma et al., 14 Apr 2025). GMS-3DQA chooses a balanced number of grid mini-patches from each of six views and forms a single Quality Mini-Patch Map (Zhang et al., 2023). HybridPoint adds approximately uniform non-salient nodes to salient points so that planar overlap is not discarded (Li et al., 2023).

The second is local reconstruction or inverse alignment after structured sampling. In Phase Marginalization, outputs produced under shifted patch lattices are inverse-aligned by reshaping patch outputs to the raster patch grid, bilinearly upsampling to padded image resolution with align-corners=false, cropping by the inverse phase offset, and recovering the original $P_{i,j}$9 dense field before averaging logits (Ercan, 6 Jun 2026). In CPGNet, grid features are resampled back to points by bilinear interpolation from the four neighboring cells (Li et al., 2022). In GridFormer, bilinear or trilinear interpolation transfers features from learned grids back to points and query coordinates (Li et al., 2024).

The third is locality-driven reduction of search complexity. Patch-Grid localizes point evaluation first to a patch volume and then to a merge-grid leaf cell, replacing global multi-patch reasoning with cell-local composition (Lin et al., 2023). A conceptually related variant appears in particle locating on unstructured meshes: an auxiliary Cartesian grid maps a query point to a vertex patch or edge patch, and the host element is then found by an angular search within that patch domain (Chen et al., 2024). In 2D, after Cartesian indexing in $$x_{i,j} = P_{i,j}(u,v), \qquad u,v \sim \mathcal{U}(0,f-1),$$0, the host element is found by binary search over a sorted fan of incident edges at a vertex, with search cost

$$x_{i,j} = P_{i,j}(u,v), \qquad u,v \sim \mathcal{U}(0,f-1),$$1

and in 3D a same-element radial move is used when necessary before edge-patch angular localization (Chen et al., 2024). The paper emphasizes that only particle coordinates are required at query time (Chen et al., 2024).

The computational profiles reflect these design choices. GPS is intended to be lightweight and model-free, avoiding bi-level optimization and storing $$x_{i,j} = P_{i,j}(u,v), \qquad u,v \sim \mathcal{U}(0,f-1),$$2 compressed samples under the same pixel budget (Ma et al., 14 Apr 2025). Uniform Phase Marginalization with $$x_{i,j} = P_{i,j}(u,v), \qquad u,v \sim \mathcal{U}(0,f-1),$$3 improves over the canonical $$x_{i,j} = P_{i,j}(u,v), \qquad u,v \sim \mathcal{U}(0,f-1),$$4 baseline across measured segmentation, depth, and local matching settings, while $$x_{i,j} = P_{i,j}(u,v), \qquad u,v \sim \mathcal{U}(0,f-1),$$5 is essentially unchanged from $$x_{i,j} = P_{i,j}(u,v), \qquad u,v \sim \mathcal{U}(0,f-1),$$6 and $$x_{i,j} = P_{i,j}(u,v), \qquad u,v \sim \mathcal{U}(0,f-1),$$7 adds little accuracy at much higher latency (Ercan, 6 Jun 2026). CPGNet reports that semantic extraction happens mainly on 2D projected grids; the abstract states that it is comparable with RPVNet while running 4.7 times faster without ensemble models or TTA (Li et al., 2022).

6. Applications, limitations, and common misconceptions

Patch-grid sampling has been used for distinct technical aims. GPS targets online class-incremental learning under memory constraints and reports 3%–4% improvements in average end accuracy under memory-constrained settings (Ma et al., 14 Apr 2025). Phase Marginalization targets dense ViT prediction and reports a compute-matched advantage of $$x_{i,j} = P_{i,j}(u,v), \qquad u,v \sim \mathcal{U}(0,f-1),$$8 mIoU over the strongest tested generic four-forward shift-based TTA row on Cityscapes (Ercan, 6 Jun 2026). Patch-Grid targets neural implicit reconstruction of sharp features, open surfaces, and thin structures through localized patch volumes and merge-grid composition (Lin et al., 2023). CPGNet targets real-time LiDAR semantic segmentation by replacing 3D neighborhood search and sparse 3D convolution with point-grid projection, 2D FCNs, and bilinear back-sampling (Li et al., 2022). GMS-3DQA targets 3D model quality assessment through multi-projection grid mini-patch sampling and compact aggregation into one QMM (Zhang et al., 2023).

Several misconceptions recur. First, PG does not always mean Patch-Grid. In CPGNet it means Point-Grid fusion, and the relevant operations are P2G pooling and G2P interpolation rather than a named PG point sampler (Li et al., 2022). Second, patch-grid sampling does not always mean arbitrary point selection. In Phase Marginalization, the sampled object is the discrete set of patch-grid phases, not free-form image points (Ercan, 6 Jun 2026). Third, patch-grid methods are not always intrinsically 3D. GMS-3DQA samples 2D projection patches, and GPS samples one pixel per image patch (Zhang et al., 2023, Ma et al., 14 Apr 2025). Fourth, not every ostensibly related document contains a sampler; one cited item is an ICCV template with placeholder prose and no proposed method (Ouyang et al., 2023).

Limitations are correspondingly heterogeneous. GPS degrades when compression is too aggressive; the $$x_{i,j} = P_{i,j}(u,v), \qquad u,v \sim \mathcal{U}(0,f-1),$$9-ablation shows $$x^{\mathrm{GPS}} \in \mathbb{R}^{r' \times r' \times C}$$0 is best, while $$x^{\mathrm{GPS}} \in \mathbb{R}^{r' \times r' \times C}$$1 and $$x^{\mathrm{GPS}} \in \mathbb{R}^{r' \times r' \times C}$$2 collapse performance on Mini-ImageNet (Ma et al., 14 Apr 2025). Phase Marginalization is limited to discrete lattice offsets, cost scales approximately linearly with $$x^{\mathrm{GPS}} \in \mathbb{R}^{r' \times r' \times C}$$3, and the benefits are modest rather than dramatic (Ercan, 6 Jun 2026). GridFormer notes that when points are too sparse, learning the weight for a single point is meaningless, so learned within-cell weighting has less benefit (Li et al., 2024). HybridPoint shows that non-salient nodes improve coverage but are intrinsically noisier and therefore require class-aware filtering (Li et al., 2023). Patch-Grid for neural implicits does not define a classic balanced inside/outside sampler; sign information is induced by surface zero supervision, pseudo-SDF offset samples, Eikonal regularization, and local patch support (Lin et al., 2023).

Taken together, these works suggest that PG point sampling is best understood as a family of locality-preserving regularization strategies. The common objective is not merely sparsification, but controlled restriction of sampling or querying to structured local support—grid cells, surface patches, patch phases, or patch-conditioned search domains—so that coverage, locality, and computational efficiency can be improved without abandoning fine-grained spatial structure.