Neighborhood Attention Decoder (NAD)

• Neighborhood Attention Decoder (NAD) is a dense, multistage decoder that employs local sliding-window self-attention to progressively upsample encoder features into full-resolution point maps.
• It integrates transposed convolutions with Neighborhood Attention across five stages to improve local surface geometry and address high-frequency details in point-map predictions.
• Empirical results show NAD outperforms alternative decoders on local and global point metrics, despite modest increases in runtime and memory usage.

Neighborhood Attention Decoder (NAD) most explicitly denotes the decoder introduced in SurGe, where a ViT token grid is progressively upsampled and mixed with Neighborhood Attention to improve local surface geometry in dense monocular point-map prediction (Knaebel et al., 29 May 2026). The phrase is not standardized across the literature. Closely related works use the distinct names “Neighbor-Aware Decoder” in point-of-interest recommendation (Ma et al., 2018) and “Neighborhood decoder” in single-view 3D reconstruction (Lionar et al., 2023), while other papers define neighborhood-based attention operators without introducing a decoder under that name (Hassani et al., 2022, Hassani et al., 2022). This suggests that NAD is best treated as a precise architectural term in SurGe and as a broader descriptive label only with care.

1. Terminological scope and nomenclature

The literature surrounding NAD is heterogeneous. Several papers are relevant because they use neighborhood-restricted aggregation in a decoder or decoder-like role, but they do not refer to the same object.

Work	Exact term	Architectural role
SurGe (Knaebel et al., 29 May 2026)	Neighborhood Attention Decoder	Five-stage progressive decoder for point maps
SAE-NAD (Ma et al., 2018)	Neighbor-Aware Decoder	Additive geographic influence term in an autoencoder decoder
NU-MCC (Lionar et al., 2023)	Neighborhood decoder	Local query-point decoder around predicted anchors
NAT (Hassani et al., 2022)	Neighborhood Attention	Attention primitive and hierarchical backbone
MNA-GT (Li et al., 2022)	Multi-Neighborhood Attention based Graph Transformer	Encoder-style graph Transformer, not a decoder

This distinction matters because a recurrent misconception is to collapse all of these uses into a single decoder family. In SAE-NAD, “NAD” expands to Neighbor-Aware Decoder, not Neighborhood Attention Decoder, and the decoder does not use query-key-value attention; it adds a geographically weighted influence term to reconstruction scores (Ma et al., 2018). In NU-MCC, the exact name is Neighborhood decoder, and its local aggregation is attached to a 3D implicit reconstruction pipeline rather than to a dense image decoder (Lionar et al., 2023). In MNA-GT, neighborhood information is encoded through multiple attention kernels inside a graph Transformer encoder, and the paper explicitly does not define a decoder module in the encoder–decoder sense (Li et al., 2022).

2. Canonical formulation in SurGe

In the most direct use of the term, SurGe places NAD between a DINOv2-initialized ViT-Large encoder and a point-map output head (Knaebel et al., 29 May 2026). For an input image $\mathbf{I}\in\mathbb{R}^{H\times W\times 3}$, the encoder produces

$$\mathbf{Z}\in\mathbb{R}^{\frac{H}{16}\times \frac{W}{16}\times C},$$

and NAD upsamples and decodes these features into a full-resolution point map $\hat{\mathbf{P}}$.

The decoder consists of five stages. Stage $\ell$ operates on

$$\mathbf{X}_\ell \in \mathbb{R}^{\frac{H}{2^{5-\ell}} \times \frac{W}{2^{5-\ell}} \times C_\ell},$$

with channel schedule

$(C_1,\ldots,C_5)=(1024,512,256,128,64),$

and with

$n_1=\cdots=n_5=3.$

Stages $1$ through $4$ upsample by a factor of $2\times$; for $$\mathbf{Z}\in\mathbb{R}^{\frac{H}{16}\times \frac{W}{16}\times C},$$0, the upsampling module is a transposed $$\mathbf{Z}\in\mathbb{R}^{\frac{H}{16}\times \frac{W}{16}\times C},$$1 convolution with stride $$\mathbf{Z}\in\mathbb{R}^{\frac{H}{16}\times \frac{W}{16}\times C},$$2, followed by a $$\mathbf{Z}\in\mathbb{R}^{\frac{H}{16}\times \frac{W}{16}\times C},$$3 convolution. The paper does not describe U-Net-style lateral skip connections from intermediate backbone stages; the decoder is attached to the final encoder token grid rather than to a multi-level skip-fusion pathway (Knaebel et al., 29 May 2026).

A mathematically faithful reconstruction of the stage computation is

$$\mathbf{Z}\in\mathbb{R}^{\frac{H}{16}\times \frac{W}{16}\times C},$$4

$$\mathbf{Z}\in\mathbb{R}^{\frac{H}{16}\times \frac{W}{16}\times C},$$5

$$\mathbf{Z}\in\mathbb{R}^{\frac{H}{16}\times \frac{W}{16}\times C},$$6

followed, for $$\mathbf{Z}\in\mathbb{R}^{\frac{H}{16}\times \frac{W}{16}\times C},$$7, by

$$\mathbf{Z}\in\mathbb{R}^{\frac{H}{16}\times \frac{W}{16}\times C},$$8

At the final stage, the output head predicts $$\mathbf{Z}\in\mathbb{R}^{\frac{H}{16}\times \frac{W}{16}\times C},$$9 and maps them to 3D points as

$\hat{\mathbf{P}}$0

Thus $\hat{\mathbf{P}}$1, and the final point map is obtained by projecting decoder features into a point-map parameterization rather than by direct Cartesian regression (Knaebel et al., 29 May 2026).

Several implementation details are integral to the definition of NAD in this paper. Each NAD block uses Neighborhood Attention with attention window $\hat{\mathbf{P}}$2, head dimension $\hat{\mathbf{P}}$3, and FFN hidden dimension $\hat{\mathbf{P}}$4. Queries and keys use window-matched RoPE, each stage adds a learned UV positional embedding, the blocks use QK normalization, and the architecture removes the usual pre-attention and pre-FFN LayerNorm (Knaebel et al., 29 May 2026).

3. Neighborhood Attention as the operative mechanism

NAD inherits its attention operator from Neighborhood Attention as formalized in NAT (Hassani et al., 2022). For input $\hat{\mathbf{P}}$5 with linear projections $\hat{\mathbf{P}}$6, $\hat{\mathbf{P}}$7, and $\hat{\mathbf{P}}$8, and relative positional biases $\hat{\mathbf{P}}$9, the attention logits for token $\ell$0 over a local neighborhood are defined by the nearest spatial neighbors of that query token:

$\ell$1

with value matrix

$\ell$2

and output

$\ell$3

The crucial structural point is that Neighborhood Attention is a sliding-window attention pattern rather than a blocked window partition. Each query token attends only to its nearest neighbors, but adjacent queries have overlapping neighborhoods. NAT characterizes this as a pixel-wise operation with linear time and space complexity relative to image size, in contrast to the quadratic complexity of full self-attention (Hassani et al., 2022). SurGe uses this operator as the core mixing mechanism of a progressive decoder rather than as a backbone-only primitive (Knaebel et al., 29 May 2026).

The broader neighborhood-attention design space is larger than SurGe’s specific choice. DiNAT introduces Dilated Neighborhood Attention, which expands receptive fields at no additional cost and alternates NA and DiNA inside a hierarchical vision transformer (Hassani et al., 2022). Generalized Neighborhood Attention further unifies sliding-window attention, strided sliding-window attention, and blocked attention through a stride parameter, and it explicitly discusses causal masking as compatible with the NA family (Hassani et al., 23 Apr 2025). These papers do not define an NAD architecture, but they establish the operator lineage from which decoder variants can be built. A plausible implication is that SurGe’s NAD occupies the “plain local self-attention” point in this larger family: progressive, multiscale, and locality-biased rather than globally attentive or dilation-based.

4. Other decoder lineages with NAD-adjacent behavior

The nearest earlier use of the acronym is SAE-NAD, a self-attentive autoencoder for point-of-interest recommendation in which NAD stands for Neighbor-Aware Decoder (Ma et al., 2018). The decoder augments the standard autoencoder reconstruction with a POI-neighborhood term derived from embedding similarity and a geographic RBF kernel. If $\ell$4 is the visited-POI set for user $\ell$5, the neighbor-aware influence matrix is

$\ell$6

which is then summed over visited POIs to form $\ell$7, and the final reconstruction becomes

$\ell$8

The paper explicitly notes that this decoder is not attention in the strict neural-attention sense: there is no query-key-value computation and no softmax over decoder neighbors (Ma et al., 2018). Its relevance to NAD is therefore conceptual rather than terminological: it is a decoder-side neighborhood mechanism, but not a Neighborhood Attention Decoder.

NU-MCC provides a different lineage. Its Neighborhood decoder addresses single-view 3D reconstruction by introducing predicted anchors, described in the abstract as “center points,” as a sparse proxy for input visual features (Lionar et al., 2023). The anchor representation is

$\ell$9

where $$\mathbf{X}_\ell \in \mathbb{R}^{\frac{H}{2^{5-\ell}} \times \frac{W}{2^{5-\ell}} \times C_\ell},$$0 are anchor features and $$\mathbf{X}_\ell \in \mathbb{R}^{\frac{H}{2^{5-\ell}} \times \frac{W}{2^{5-\ell}} \times C_\ell},$$1 are anchor positions. For a query point $$\mathbf{X}_\ell \in \mathbb{R}^{\frac{H}{2^{5-\ell}} \times \frac{W}{2^{5-\ell}} \times C_\ell},$$2, the decoder aggregates only the $$\mathbf{X}_\ell \in \mathbb{R}^{\frac{H}{2^{5-\ell}} \times \frac{W}{2^{5-\ell}} \times C_\ell},$$3-nearest anchors:

$$\mathbf{X}_\ell \in \mathbb{R}^{\frac{H}{2^{5-\ell}} \times \frac{W}{2^{5-\ell}} \times C_\ell},$$4

with weights

$$\mathbf{X}_\ell \in \mathbb{R}^{\frac{H}{2^{5-\ell}} \times \frac{W}{2^{5-\ell}} \times C_\ell},$$5

The per-query predictions are then $$\mathbf{X}_\ell \in \mathbb{R}^{\frac{H}{2^{5-\ell}} \times \frac{W}{2^{5-\ell}} \times C_\ell},$$6. This decoder is neighborhood-restricted and attention-like, but the exact paper term remains “Neighborhood decoder,” not NAD (Lionar et al., 2023).

A third decoder-adjacent route appears in LiDAR geometry compression through Neighborhood Point Attention and NPAFormer (Xue et al., 2022). For sparse tensor inputs with coordinates $$\mathbf{X}_\ell \in \mathbb{R}^{\frac{H}{2^{5-\ell}} \times \frac{W}{2^{5-\ell}} \times C_\ell},$$7 and features $$\mathbf{X}_\ell \in \mathbb{R}^{\frac{H}{2^{5-\ell}} \times \frac{W}{2^{5-\ell}} \times C_\ell},$$8, the local neighborhood is built by coordinate-space $$\mathbf{X}_\ell \in \mathbb{R}^{\frac{H}{2^{5-\ell}} \times \frac{W}{2^{5-\ell}} \times C_\ell},$$9-NN, and relative geometry is encoded as

$(C_1,\ldots,C_5)=(1024,512,256,128,64),$0

Queries, keys, and values are then

$(C_1,\ldots,C_5)=(1024,512,256,128,64),$1

and localized attention is

$(C_1,\ldots,C_5)=(1024,512,256,128,64),$2

This module is embedded in a multiscale occupancy probability approximation used directly in arithmetic decoding. It is not named NAD, but it is decoder-side neighborhood attention in a literal probabilistic decoding loop (Xue et al., 2022).

5. Empirical profile

SurGe motivates NAD by arguing that recent feedforward point-map models achieve strong global geometry while still exhibiting poor local surface geometry, particularly on thin structures and high-frequency detail (Knaebel et al., 29 May 2026). In the decoder ablation, NAD outperforms ViT decoder, DPT head, ConvStack, and ConvStack-L on the reported local point metric across all five listed datasets. On ETH3D, for example, NAD reaches $(C_1,\ldots,C_5)=(1024,512,256,128,64),$3 versus $(C_1,\ldots,C_5)=(1024,512,256,128,64),$4 for ConvStack-L and $(C_1,\ldots,C_5)=(1024,512,256,128,64),$5 for the ViT decoder; on DIODE it reaches $(C_1,\ldots,C_5)=(1024,512,256,128,64),$6 versus $(C_1,\ldots,C_5)=(1024,512,256,128,64),$7 and $(C_1,\ldots,C_5)=(1024,512,256,128,64),$8, respectively. On the point-map normal angular error, NAD is again best on all reported datasets, including $(C_1,\ldots,C_5)=(1024,512,256,128,64),$9 on ETH3D, $n_1=\cdots=n_5=3.$0 on iBims-1, $n_1=\cdots=n_5=3.$1 on GSO, $n_1=\cdots=n_5=3.$2 on Sintel, and $n_1=\cdots=n_5=3.$3 on DIODE. The same study reports that NAD also attains the best average rank, $n_1=\cdots=n_5=3.$4, on the global point-map metric, with first place on $n_1=\cdots=n_5=3.$5 of $n_1=\cdots=n_5=3.$6 datasets. These gains are not free: at $n_1=\cdots=n_5=3.$7, decoder-only runtime is $n_1=\cdots=n_5=3.$8 ms for NAD versus $n_1=\cdots=n_5=3.$9 ms for ConvStack-L, and the full model with DINOv2-Large is $1$0 ms versus $1$1 ms, with peak memory rising from $1$2 GiB to $1$3 GiB (Knaebel et al., 29 May 2026).

NU-MCC offers a complementary efficiency-oriented result for local neighborhood decoding in 3D reconstruction (Lionar et al., 2023). The paper reports that NU-MCC “outperforms MCC by 9.7% in terms of the F1-score on the CO3D-v2 dataset with more than 5x faster running speed.” Table 1 isolates the decoder contribution: MCC with occupancy takes $1$4 s at evaluation with $1$5k query points, whereas the neighborhood decoder without fine features takes $1$6 s and with fine features $1$7 s. This supports the claim that local query-restricted decoding can reduce the cost of large query sets without reverting to global Transformer decoding (Lionar et al., 2023).

SAE-NAD provides evidence from recommendation rather than geometry (Ma et al., 2018). In its component study, the neighbor-aware decoder alone outperforms the weighted autoencoder and also exceeds the self-attentive encoder-only variant on all three datasets and all listed ranking metrics. On Gowalla at P@10 / R@10 / MAP@10, WAE scores $1$8, SAE-WAE scores $1$9, and NAD-WAE scores $4$0. The paper interprets this as evidence that decoder-side neighborhood modeling is especially effective because it directly affects final recommendation scores (Ma et al., 2018).

In LiDAR compression, neighborhood attention also shows a decoder-facing systems benefit (Xue et al., 2022). Relative to OctAttention, the reported average decoding time drops from $4$1 s to $4$2 s, a speedup of about $4$3, while still presenting better compression efficiency. Relative to SparsePCGC, decoding time drops from $4$4 s to $4$5 s, about $4$6 faster. The paper further reports $4$7 bitrate reduction for the lossless scenario and about $4$8 BD-rate gain over G-PCC in lossy coding (Xue et al., 2022).

6. Architectural significance and open distinctions

Across these works, the phrase “Neighborhood Attention Decoder” refers to structurally different objects. In SurGe, NAD is a multistage dense decoder that uses plain Neighborhood Attention for local content-dependent mixing over 2D feature maps (Knaebel et al., 29 May 2026). In NU-MCC, the decoder is neighborhood-restricted because each query point attends only to nearby anchor and fine features in continuous 3D space (Lionar et al., 2023). In SAE-NAD, the decoder is neighborhood-aware because it injects an explicit POI-to-POI influence field based on embedding similarity and geographic proximity, not because it implements self-attention (Ma et al., 2018). In LiDAR compression, neighborhood attention lives inside a conditional probability model used in arithmetic decoding over sparse occupied voxels (Xue et al., 2022).

This diversity exposes three stable axes of variation. The first is how the neighborhood is defined: fixed local image windows in NA-based dense decoders, nearest anchors in implicit 3D reconstruction, coordinate-space $4$9-NN in sparse geometry, or geography-weighted item neighborhoods in recommendation. The second is what role the decoder plays: dense feature upsampling, continuous query evaluation, score reconstruction, or probabilistic entropy decoding. The third is how locality trades against efficiency. SurGe explicitly treats NAD as an accuracy-oriented decoder and states that more efficient variants are a useful future direction (Knaebel et al., 29 May 2026). Generalized Neighborhood Attention and NABLA extend the operator space further through stride-controlled sparse patterns and adaptive block-level masks, but they do not define a decoder architecture called NAD (Hassani et al., 23 Apr 2025, Mikhailov et al., 17 Jul 2025).

A plausible implication is that NAD should not be treated as a single canonical module. The strongest common denominator is decoder-side computation that restricts or reweights interactions according to a neighborhood structure rather than relying on unrestricted global attention or purely fixed-kernel convolution. Under that reading, SurGe provides the clearest explicit NAD, while the surrounding literature supplies adjacent mechanisms, competing nomenclatures, and alternative definitions of what “neighborhood” means in a decoder.