SEAttention3D: Structure-Aware 3D Attention Variants

• SEAttention3D is defined as a naming pattern for structure-aware 3D attention modules that preserve important spatial details across different data representations.
• In SliceSemOcc, the module employs height-aware channel reweighting on voxel features to retain vertical resolution, yielding a modest mIoU improvement.
• Variants in SEFormer and S²-MLLM use quantized offset-based value projections and multimodal intra-/inter-view attention to enhance 3D object detection and visual grounding.

Searching arXiv for the relevant papers and terminology around “SEAttention3D”. [arXiv search unavailable in this interface; proceeding from the supplied arXiv records and IDs.] SEAttention3D is a non-unique name used in recent 3D vision literature for several distinct attention mechanisms rather than for a single canonical operator. In the supplied arXiv record set, the term denotes three different modules: a height-aware squeeze-and-excitation variant for 3D semantic occupancy in SliceSemOcc (Huang et al., 4 Sep 2025), a structure-embedding self-attention mechanism for point-cloud 3D object detection in SEFormer (Feng et al., 2022), and a structure-enhanced multimodal attention block for 3D visual grounding in S$^2$-MLLM (Xu et al., 1 Dec 2025). Across these usages, the common design intent is to retain 3D structural information that would be blurred or discarded by more isotropic attention or pooling operators, but the preserved structure differs substantially across voxel, point, and multi-view RGB-D representations.

1. Nomenclature and conceptual scope

The most immediate point of clarification is terminological. In SliceSemOcc, SEAttention3D is a channel-attention module that preserves the height axis in voxel tensors and assigns dynamic channel attention weights to each height layer (Huang et al., 4 Sep 2025). In SEFormer, SEAttention3D abbreviates “Structure-Embedding Attention” and replaces the single value projection of vanilla self-attention with a dictionary of value projections indexed by quantized 3D offsets (Feng et al., 2022). In S$^2$-MLLM, the same label is used for a “Structure-Enhanced” module that combines intra-view attention, inter-view attention, and multi-level positional encoding for RGB-D views before insertion into an MLLM (Xu et al., 1 Dec 2025).

This naming overlap can create a common misconception that SEAttention3D refers to one standardized block. The supplied literature suggests the opposite: the term functions as a local paper-specific name whose meaning depends on the representation being processed. A plausible implication is that any technical discussion of “SEAttention3D” must specify the paper context, because the tensor layout, attention domain, inductive bias, and reported gains differ materially across the three instances.

2. Height-wise channel attention in SliceSemOcc

In SliceSemOcc, the input is a voxel-feature volume

$$X \in \mathbb{R}^{B \times C \times X \times Y \times Z},$$

with $Z$ denoting the height axis. The module consumes $X$ and returns a re-weighted tensor of the same shape. Its defining modification relative to vanilla SENet is the squeeze step: it collapses the $X$–$Y$ plane but keeps separate statistics for each channel $c$ and height $z$,

$$S_{b,c,z} = \frac{1}{X \cdot Y}\sum_{i=1}^{X}\sum_{j=1}^{Y} X_{b,c,i,j,z}, \qquad S \in \mathbb{R}^{B \times C \times Z}.$$

The excitation stage then applies two $^2$0 1D convolutions over the channel dimension, independently at each height,

$^2$1

$^2$2

followed by broadcast re-weighting:

$^2$3

The principal distinction from standard SENet is explicit in the ablation description: vanilla SENet pools over $^2$4, $^2$5, and $^2$6 and therefore assigns identical channel weights at all heights, whereas SEAttention3D preserves height-wise resolution through average pooling and learns distinct channel weights for each vertical slice. The reported ablation shows that replacing standard SENet yields an increase from $^2$7 mIoU to $^2$8 mIoU, an absolute gain of $^2$9 attributable solely to height-wise differentiated channel attention (Huang et al., 4 Sep 2025).

Implementation notes given for this variant are concrete. The reduction ratio $$X \in \mathbb{R}^{B \times C \times X \times Y \times Z},$$0 controls the bottleneck size in the excitation stage, with common choices $$X \in \mathbb{R}^{B \times C \times X \times Y \times Z},$$1 or $$X \in \mathbb{R}^{B \times C \times X \times Y \times Z},$$2. The two layers can be implemented either as Conv1d or as fully connected layers acting over the $$X \in \mathbb{R}^{B \times C \times X \times Y \times Z},$$3 dimension independently at each height. ReLU follows the first layer, Sigmoid the second, and the module is placed immediately after any 3D-conv or voxel-feature slice so that subsequent layers receive height-aware, channel-reweighted input. In the broader SliceSemOcc framework, this mechanism complements vertical slice extraction and a global local fusion module, reflecting the paper’s claim that most existing occupancy approaches overlook height-axis information when processing voxel features.

3. Structure-embedding attention in SEFormer

SEFormer uses SEAttention3D in a different sense. Its central motivation is that convolution provides structure encoding through distinct kernels at each offset, whereas standard Transformer attention is structure-preserving but isotropic in the value transform. SEAttention3D therefore “marries” the two by using self-attention to select relevant neighbors while applying different value projections depending on each neighbor’s quantized direction and distance to the query (Feng et al., 2022).

For point coordinates $$X \in \mathbb{R}^{B \times C \times X \times Y \times Z},$$4 and per-point features $$X \in \mathbb{R}^{B \times C \times X \times Y \times Z},$$5, the module defines

$$X \in \mathbb{R}^{B \times C \times X \times Y \times Z},$$6

$$X \in \mathbb{R}^{B \times C \times X \times Y \times Z},$$7

The crucial object is $$X \in \mathbb{R}^{B \times C \times X \times Y \times Z},$$8, which is not a single global matrix but a learnable matrix selected from a look-up table indexed by the quantized offset. Relative position is quantized by overlaying a small $$X \in \mathbb{R}^{B \times C \times X \times Y \times Z},$$9 grid, or more generally $Z$0, around each query with grid-step $Z$1, and each neighbor is interpolated onto the nearest virtual grid cell. In practice, multiple radii $Z$2 are used in parallel.

Aggregation is then

$Z$3

$Z$4

The paper also extends this to multi-head attention.

Within the full SEFormer network, a sparse-3D-CNN backbone produces four scales of voxel features. The “Spatial Structure Moduling” stage samples $Z$5 query points on the finest scale; for each of the four scales and each radius, it grid-interpolates approximately $Z$6 keys, applies a block of $Z$7 stacked SEAttention3D units, concatenates outputs across radii, and aggregates across scales via a vanilla Transformer. The ROI head subdivides each first-stage 3D proposal into a $Z$8 grid of sub-regions and again applies SEAttention3D. The reported hyper-parameters are $Z$9 heads, per-head hidden dimensions $X$0, and $X$1 parallel radii with initial $X$2m and then $X$3m (Feng et al., 2022).

The ablation against vanilla Transformer is explicit. In spatial moduling and head configuration, the baseline Transformer reports LEVEL1 mAP $X$4; replacing spatial moduling with SEFormer while keeping the Transformer head gives $X$5; using SEFormer in both places gives $X$6. The main Waymo Open Dataset result is $X$7 mAP and $X$8 mAPH on LEVEL$X$9, $X$0 and $X$1 on LEVEL$X$2, and BEV-mAP LEVEL$X$3 of $X$4 (Feng et al., 2022). The stated intuition is that direction- and distance-aware value projections embed local geometric arrangement into the representation itself, improving shape, orientation, and localization cues that isotropic attention cannot encode.

4. Structure-enhanced multimodal attention in S$X$5-MLLM

In S$X$6-MLLM, SEAttention3D is a multimodal pre-LLM module for 3D visual grounding. It is inserted between a pretrained video encoder and a frozen LLM, and it performs three operations: multi-level positional encoding, intra-view attention, and inter-view attention (Xu et al., 1 Dec 2025). The input is a batch of $X$7 RGB-D views with intrinsics and extrinsics, together with a tokenized natural-language query and $X$8 candidate 3D bounding boxes.

Let

$X$9

denote patch features from the video encoder. Intra-view attention operates separately on each view $Y$0 by projecting

$Y$1

and computing

$Y$2

Inter-view attention instead gathers the same patch index across views, forms

$Y$3

and computes

$Y$4

The final fusion is

$Y$5

optionally followed by a feed-forward network and another LayerNorm.

The positional component is equally central. For patch-center pixel $Y$6 and depth $Y$7, the world coordinate is computed by

$Y$8

and the camera-ray direction is

$Y$9

A NeRF-style sinusoidal coordinate encoding $c$0 and an MLP-based ray encoding $c$1 are added to the raw patch features, together with spatial neighborhood averaging, before attention is applied. During training, a Fast3R-style reconstruction branch imposes 3D-structure supervision; at inference, the reconstruction branch is disabled and only the encoder, SEAttention3D, and LLM path are used (Xu et al., 1 Dec 2025).

The reported results directly connect the module to grounding performance. On ScanRefer, Video-3D-LLM (LoRA) reports [email protected] $c$2 and [email protected] $c$3; SeeGround zero-shot reports $c$4; S$c$5-MLLM reports $c$6. On ReferIt3D with predicted boxes, MCLN reports Sr3D $c$7, Nr3D $c$8, while S$c$9-MLLM reports Sr3D $z$0, Nr3D $z$1. The ablation table shows that removing SE reduces ScanRefer performance to $z$2, removing multi-level positional encoding to $z$3, removing intra/inter attention to $z$4, and removing language loss to $z$5; the full model remains $z$6 (Xu et al., 1 Dec 2025). The paper also states that adding just the SE Attention module to the raw LLaVA-Video backbone yields [email protected] $z$7 and [email protected] $z$8 in Appendix A.

5. Comparative analysis across the three variants

The three modules share a family resemblance only at a high level. Each introduces a structural prior that a more isotropic baseline discards, but the encoded structure is representation-specific.

Variant	Core structural bias	Reported comparison
SliceSemOcc (Huang et al., 4 Sep 2025)	Height-preserving average pooling and per-channel, per-height attention weights	mIoU $z$9 replacing standard SENet
SEFormer (Feng et al., 2022)	Value projection indexed by quantized direction and distance, with multi-radii aggregation	LEVEL1 mAP $$S_{b,c,z} = \frac{1}{X \cdot Y}\sum_{i=1}^{X}\sum_{j=1}^{Y} X_{b,c,i,j,z}, \qquad S \in \mathbb{R}^{B \times C \times Z}.$$0 from baseline Transformer to full SEFormer
S$$S_{b,c,z} = \frac{1}{X \cdot Y}\sum_{i=1}^{X}\sum_{j=1}^{Y} X_{b,c,i,j,z}, \qquad S \in \mathbb{R}^{B \times C \times Z}.$$1-MLLM (Xu et al., 1 Dec 2025)	Intra-view and inter-view attention with 3D coordinate and camera-ray encoding	ScanRefer overall $$S_{b,c,z} = \frac{1}{X \cdot Y}\sum_{i=1}^{X}\sum_{j=1}^{Y} X_{b,c,i,j,z}, \qquad S \in \mathbb{R}^{B \times C \times Z}.$$2; removing SE gives $$S_{b,c,z} = \frac{1}{X \cdot Y}\sum_{i=1}^{X}\sum_{j=1}^{Y} X_{b,c,i,j,z}, \qquad S \in \mathbb{R}^{B \times C \times Z}.$$3

In SliceSemOcc, the attention target is channel weighting conditioned on height layers in a dense voxel tensor. In SEFormer, the attention target is neighborhood aggregation in irregular point clouds, with anisotropy appearing in the value projection rather than in the attention weights themselves. In S$$S_{b,c,z} = \frac{1}{X \cdot Y}\sum_{i=1}^{X}\sum_{j=1}^{Y} X_{b,c,i,j,z}, \qquad S \in \mathbb{R}^{B \times C \times Z}.$$4-MLLM, the target is cross-view feature enhancement for multimodal grounding, and the structural signal is injected jointly through geometry-aware positional encodings and two complementary attention axes. This suggests that “SEAttention3D” is best understood as a recurrent naming pattern for structure-aware 3D attention rather than as a transportable, architecture-agnostic primitive.

A second useful distinction concerns where structure enters the computation. SliceSemOcc modifies the squeeze stage by preserving $$S_{b,c,z} = \frac{1}{X \cdot Y}\sum_{i=1}^{X}\sum_{j=1}^{Y} X_{b,c,i,j,z}, \qquad S \in \mathbb{R}^{B \times C \times Z}.$$5; SEFormer modifies the value operator by indexing it with quantized offset bins; S$$S_{b,c,z} = \frac{1}{X \cdot Y}\sum_{i=1}^{X}\sum_{j=1}^{Y} X_{b,c,i,j,z}, \qquad S \in \mathbb{R}^{B \times C \times Z}.$$6-MLLM modifies both token embeddings and attention topology through positional enrichment plus intra-/inter-view factorization. The baselines also differ: standard SENet for occupancy, vanilla Transformer for point-based detection, and a raw LLaVA-Video-style backbone or ablated multimodal architecture for grounding.

6. Relation to adjacent 3D attention frameworks

A nearby but distinct line of work is the SE(3)-Transformer, which introduces an SE(3)-equivariant self-attention layer for point clouds and graphs rather than a module named SEAttention3D (Fuchs et al., 2020). Its query, key, and value maps are built from type-$$S_{b,c,z} = \frac{1}{X \cdot Y}\sum_{i=1}^{X}\sum_{j=1}^{Y} X_{b,c,i,j,z}, \qquad S \in \mathbb{R}^{B \times C \times Z}.$$7 fields, spherical harmonics, radial MLPs, and Clebsch–Gordan blocks so that the resulting attention is equivariant under continuous 3D roto-translations. Scalar compatibility scores are invariant,

$$S_{b,c,z} = \frac{1}{X \cdot Y}\sum_{i=1}^{X}\sum_{j=1}^{Y} X_{b,c,i,j,z}, \qquad S \in \mathbb{R}^{B \times C \times Z}.$$8

while type-specific value messages transform equivariantly under Wigner-$$S_{b,c,z} = \frac{1}{X \cdot Y}\sum_{i=1}^{X}\sum_{j=1}^{Y} X_{b,c,i,j,z}, \qquad S \in \mathbb{R}^{B \times C \times Z}.$$9 representations. The layer therefore guarantees SE(3)-equivariance, unlike the three SEAttention3D variants discussed above, whose objectives are height sensitivity, local geometric encoding, or multimodal spatial reasoning rather than explicit group equivariance.

The distinction matters because the overlap in abbreviations can invite confusion. SEFormer’s “SE” means “Structure-Embedding”; S$^2$00-MLLM’s “SE” means “Structure-Enhanced”; SliceSemOcc’s SEAttention3D is a height-aware extension of squeeze-and-excitation; and SE(3)-Transformer refers to the symmetry group SE(3). These are conceptually adjacent only in the broad sense that all are attempts to improve 3D representation learning with attention mechanisms that respect structure more faithfully than isotropic baselines.

The empirical record in the supplied papers indicates three different payoff regimes. SliceSemOcc reports a modest but isolated gain from height-wise channel reweighting. SEFormer reports substantial gains for 3D object detection by combining structure preserving and structure encoding. S$^2$01-MLLM reports improvements in 3D visual grounding, including out-of-distribution evaluation on MultiScan and ArkitScenes. Taken together, the literature suggests that the term SEAttention3D names a class of paper-specific solutions to the same general problem: preserving discriminative 3D structure that standard attention, standard pooling, or viewpoint-limited encoders do not represent adequately.