RelFlexformer: Efficient Attention 3D-Transformers for Integrable Relative Positional Encodings

Abstract: We present a new class of efficient attention mechanisms applying universal 3D Relative Positional Encoding (RPE) methods given by arbitrary integrable modulation functions $f$. They lead to the new class of 3D-Transformer models, called \textit{RelFlexformers}, flexibly integrating those RPEs, and characterized by the $O(L \log L)$ time complexity of the attention computation for the $L$-length input sequences. RelFlexformers builds on the theory of the Non-Uniform Fourier Transform (NU-FFT), naturally generalizing several existing efficient RPE-attention methods from structured settings with tokens homogeneously embedded in unweighted grids into general non-structured heterogeneous scenarios, where tokens' positions are arbitrarily distributed in the corresponding 3D spaces. As such, RelFlexformers can be applied in particular to model point clouds. Our extensive empirical evaluation on a large portfolio of 3D datasets confirms quality improvements provided by the NU-FFT-driven attention modulation techniques in the RelFlexformers.

• The paper introduces RelFlexformer, a kernel-based Transformer that efficiently integrates arbitrary relative positional encodings using NU-FFT based FastMult, achieving O(L log L) complexity.
• The paper demonstrates enhanced accuracy on 3D classification and segmentation benchmarks, outperforming or matching standard quadratic Transformers on datasets like ModelNet40 and ScanNet.
• The paper shows that the learnable spectral quadrature enables flexible spatial biasing in unstructured 3D domains, paving the way for robust real-time perception and AR/VR applications.

RelFlexformer: Advancing Efficient Kernel-based 3D Attention with Integrable Relative Positional Encodings

Motivation and Contributions

The paper "RelFlexformer: Efficient Attention 3D-Transformers for Integrable Relative Positional Encodings" (2605.10706) introduces a class of scalable kernel-based self-attention architectures tailored for 3D data modalities, especially point clouds and lifted RGB-D inputs. Standard quadratic-complexity Transformers, while expressive, are inefficient in these domains due to large, irregular, and unordered input sizes and the substantial importance of geometric relationships. While kernelized variants such as Performer provide linear complexity, they generally lack an efficient mechanism for modeling arbitrary relative positional encodings (RPEs), resorting either to restrictive grid-based biases or falling back to $\mathcal{O}(L^2)$ masks.

RelFlexformer addresses this bottleneck by leveraging the Non-Uniform Fast Fourier Transform (NU-FFT) to enable direct integration of arbitrary $L_1$-integrable RPEs into kernel attention with sub-quadratic, $\mathcal{O}(L \log L)$, complexity. This framework extends linear RPE approaches (e.g., Toeplitz-based and string kernel-based methods) to general non-Euclidean token layouts, subsumes existing multidimensional encodings such as RoPE and STRING as special cases, and provides fast, unified geometric mask operators compatible with generic 3D data distributions.

The main contribution is a general efficient algorithm, FastMult, for masked attention via spectral convolution, which approximates the matrix-vector product $\mathbf{w} = \mathbf{M}\mathbf{u}$, where $\mathbf{M}$ is an RPE modulation mask and $\mathbf{u}$ is the input, using stochastic quadrature in the spectral domain. The technique preserves the scalability of linear attention while endowing the model with expressive geometric bias.

Methodology

RelFlexformer expresses distance-dependent modulation of attention between input tokens via an arbitrary integrable function $f(\mathbf{r}_i - \mathbf{r}_j)$ acting on their 3D coordinates. Instead of explicitly materializing the $L\times L$ mask matrix, the approach:

• Observes that the masking operation corresponds to convolution, and thus, by the Convolution Theorem, can be cast as multiplication in the Fourier domain.
• Uses the NU-FFT to compute the (forward and inverse) transforms over irregularly-spaced tokens, which crucially generalizes block-Toeplitz/grid-based FFT approaches to continuous or unstructured domains.
• Employs stochastic quadrature, parameterized by sample points and weights, to approximate the Fourier integral efficiently, with error decaying as $\mathcal{O}(1/\sqrt{S})$ in the number of samples $S$.
• Enables the modulation function $L_1$0 and spectral quadrature to be learnable, allowing flexible and data-driven spatial bias.

The FastMult masked matrix-vector product achieves $L_1$1 complexity and avoids the quadratic cost of explicit masking. See the execution time benchmarks and performance-complexity analysis:

Figure 1: RelFlexformer is compared to standard Transformer and Performer models in terms of both execution time scaling and accuracy; FastMult is shown to outperform naive implementations, and the quadrature error decays with $L_1$2.

The learned (or fixed) kernels can mimic RBF or Laplacian decay and are shown to replicate and generalize RoPE, as well as factorization forms underlying STRING position encoding. The decay and regularity of the kernel across Euclidean distances are visualized below:

Figure 2: The RelFlexformer mask's decay and projection variance are compared to standard RBF and Laplace kernels, demonstrating precise spatially adaptive biasing in 3D.

Experimental Results

RelFlexformer is benchmarked on a comprehensive suite of 3D classification and segmentation datasets, including ModelNet40, ScanObjectNN, S3DIS, ScanNet, ScanNet200, ScanNet++, nuScenes, NYU Depth v2, and SUN RGB-D, and is evaluated as a drop-in replacement for Performer in PCT, PTv3, and DFormer backbones.

Across all tasks, RelFlexformer robustly closes and frequently surpasses the accuracy gap between kernelized attention models and standard ($L_1$3) Transformers. Notable highlights:

• Classification: On ModelNet40 and ScanObjectNN, Performer baseline accuracy is improved from 92.34% to 92.94% and from 83.16% to 84.45% respectively, matching or exceeding vanilla Transformers.
• Semantic Segmentation: On ScanNet, ScanNet200, ScanNet++, and nuScenes, RelFlexformer consistently exhibits multi-point mIoU improvements (e.g., 74.8 to 76.8 mIoU on ScanNet, 28.2 to 34.0 mIoU on ScanNet200), often outperforming dense-attention baselines in complex scenarios (ScanNet++, S3DIS Area 3, nuScenes).
• RGB-D Segmentation: On NYU Depth v2 and SUN RGB-D, RelFlexformer achieves mIoU of 55.32% and 51.04%, respectively, outperforming Performer and STRING, and achieving parity with full-attention Transformer baselines.

Ablation studies indicate that a small quadrature size $L_1$4 suffices for optimal accuracy, and the method is robust to modulation parameter $L_1$5, with the optimal setting varying by dataset but always outperforming the Performer base.

Figure 3: The effect of quadrature sample size $L_1$6 per head on NYU Depth v2, ScanObjectNN, and SUN RGB-D; performance saturates at moderate $L_1$7, demonstrating computational parsimony.

Theoretical and Practical Implications

From a theoretical standpoint, RelFlexformer demonstrates that fast spectral mask application via NU-FFT permits a broad class of geometric priors within efficient attention. The approach not only generalizes and subsumes grid-based and group-theoretic RPEs, but also allows integrating learnable spectral representations for handling highly heterogeneous data distributions. The algorithm is invariant to the intrinsic structure—unstructured clouds, lifted RGB-D pixels, embedded graphs—and does not require auxiliary neighbor graphs or costly pre-processing.

Practically, RelFlexformer is poised to enable high-resolution, global 3D reasoning across domains such as robotics, real-time perception, and AR/VR, where sequence lengths exceed tens of thousands and standard Transformer memory fails. Its architecture serves as a unifying geometric operator, applicable across point clouds, multi-sensor fusion, and dense lifting, while being compatible with additional inductive priors (e.g., PointRoPE).

Limitations and Future Directions

The approach relies on $L_1$8-integrable masks with tractable Fourier transforms. Extending to all smooth functions or non-integrable cases remains an open area. Selection or learning of optimal spectral priors per application, and scaling to even larger or more diverse data, are promising future directions. The interplay of learned and fixed positional kernels in settings with complex semantics (e.g., scene graphs, hybrid visual-lidar domains) also merits further investigation.

Conclusion

RelFlexformer establishes a rigorous algorithmic and practical paradigm for efficient, RPE-modulated kernel attention in 3D vision. By unifying geometric reasoning via the NU-FFT and enabling deep architectural flexibility, it advances scalable, expressive, and adaptable Transformers for modern 3D perception challenges, empirically matching or exceeding the performance of standard dense-attention models while maintaining sub-quadratic time and memory complexity.

(2605.10706)