RelFlexformers: Efficient 3D Attention

• RelFlexformers are efficient 3D Transformer models that integrate continuous, universal relative positional encoding using NU-FFT for irregular token distributions.
• They replace standard softmax attention with a Fourier-based low-rank scheme that achieves O(L log L) complexity while encoding geometrical bias in a continuous convolution.
• Empirical evaluations show that RelFlexformers match or outperform quadratic attention models on point cloud and RGB-D segmentation benchmarks.

RelFlexformers are a class of efficient attention 3D-Transformer models that integrate universal 3D relative positional encoding (RPE) through arbitrary integrable modulation functions $f$, while retaining $O(L \log L)$ attention complexity for input sequences of length $L$. They are formulated for irregular 3D inputs such as point clouds or lifted RGB-D patches, and build on the Non-Uniform Fast Fourier Transform (NU-FFT) to generalize efficient RPE-attention methods from homogeneous grids to non-structured heterogeneous settings in which token positions are arbitrarily distributed in $\mathbb R^3$ (Kim et al., 11 May 2026).

1. Definition and design objective

The defining problem addressed by RelFlexformers is the tension between geometric expressivity and computational efficiency in 3D attention. Standard softmax attention has $O(L^2)$ cost, which is prohibitive for large irregular 3D inputs. At the same time, 3D tasks benefit from encoding relative geometric positions of tokens into attention. RelFlexformer is designed to satisfy both requirements by replacing softmax with a linear-attention kernel, inserting an element-wise RPE mask $M$, and showing that the resulting masked attention operator can still be applied in $O(L \log L)$ time (Kim et al., 11 May 2026).

The RPE mask is defined entrywise by

$M_{ij} = f(\mathbf r_i - \mathbf r_j),$

where $\{\mathbf r_i\}\subset\mathbb R^3$ are token coordinates and $f:\mathbb R^3\to\mathbb R$ is an $O(L \log L)$0-integrable modulation function. The attention operator is written as

$O(L \log L)$1

This construction places RelFlexformers within the family of Performer-style linear-attention architectures, but augments them with continuous 3D relative geometry. A central consequence is that the framework is not restricted to structured lattices or voxel grids: the same formalism applies when token positions are arbitrary points in 3D space. The source explicitly notes point clouds as a target setting.

2. Integrable modulation functions and 3D relative positional encoding

RelFlexformer assumes a continuous modulation function

$O(L \log L)$2

such that

$O(L \log L)$3

This condition supplies the analytic basis for Fourier-domain treatment of the mask and therefore for efficient computation (Kim et al., 11 May 2026).

The paper describes typical choices as radial-decay kernels, for example

$O(L \log L)$4

and also mentions physically-motivated Green’s functions, including heat, Laplace, and Cauchy. The framework therefore treats relative positional encoding not as a fixed discrete lookup or a single parameterization, but as a family indexed by arbitrary integrable spatial modulation functions.

The Fourier transform and inverse transform are given by

$O(L \log L)$5

$O(L \log L)$6

By the Fourier inversion and convolution theorems, convolution in space corresponds to multiplication in frequency. In RelFlexformer, this is the key observation enabling the RPE mask to be applied through frequency-domain operations rather than by direct quadratic construction of $O(L \log L)$7.

A useful way to interpret the model is that geometric bias is moved from explicit pairwise attention computation into a continuous convolutional operator defined over token locations. This suggests a unification of positional encoding and spatial kernel design: different choices of $O(L \log L)$8 induce different relative-geometry priors while preserving the same computational skeleton.

3. NU-FFT formulation of masked attention

The computational core is the fast multiplication of a vector by the RPE mask. For an arbitrary vector $O(L \log L)$9, RelFlexformer computes

$L$0

To do so, it introduces the point-cloud signal distribution

$L$1

so that

$L$2

Applying Fourier inversion and the convolution theorem yields

$L$3

with

$L$4

The integral is approximated by a quadrature with $L$5 frequency samples $L$6 and weights $L$7: $L$8 Defining

$L$9

the values at token locations become

$\mathbb R^3$0

The practical significance is that the sequences $\mathbb R^3$1 and $\mathbb R^3$2 are each computable by a single NU-FFT in time $\mathbb R^3$3. Taking $\mathbb R^3$4, or even constant $\mathbb R^3$5, gives $\mathbb R^3$6. The paper states the underlying complexity theorem in the form: given points $\mathbb R^3$7, frequencies $\mathbb R^3$8, and coefficients $\mathbb R^3$9, the sums

$O(L^2)$0

can be computed in $O(L^2)$1 time (Kim et al., 11 May 2026).

4. Performer-style attention construction

RelFlexformer combines the fast mask multiplication routine with low-rank linear attention. The first core procedure, FastMult$O(L^2)$2, takes $O(L^2)$3, point positions $O(L^2)$4, and modulation Fourier data $O(L^2)$5, where $O(L^2)$6, and returns an approximation to $O(L^2)$7 (Kim et al., 11 May 2026).

Its steps are:

1. a forward NU-FFT to compute

$O(L^2)$8

1. pointwise formation of $O(L^2)$9,
2. an inverse NU-FFT to compute

$M$0

The second procedure is a masked low-rank attention algorithm. With queries $M$1, keys $M$2, values $M$3, feature map $M$4, and FastMask operator FastMult$M$5, the model builds

$M$6

with row $M$7 equal to

$M$8

and

$M$9

with row $O(L \log L)$0 equal to $O(L \log L)$1. FastMult$O(L \log L)$2 is then applied to the columns of both matrices, producing $O(L \log L)$3 and $O(L \log L)$4. The output embedding for token $O(L \log L)$5 is recovered as

$O(L \log L)$6

The stated overall complexity is

$O(L \log L)$7

so for fixed $O(L \log L)$8 and $O(L \log L)$9 the asymptotic cost is $M_{ij} = f(\mathbf r_i - \mathbf r_j),$0. In contrast to dense quadratic attention, the relative-position mask is never materialized as a full $M_{ij} = f(\mathbf r_i - \mathbf r_j),$1 matrix.

5. Relation to existing efficient RPE methods

A central claim of RelFlexformer is that it generalizes several existing efficient RPE-attention constructions. This is not presented as an informal analogy; the source gives explicit reductions for multiple prior families (Kim et al., 11 May 2026).

For Toeplitz or doubly-Toeplitz masks on 1D, 2D, or 3D grids, when the positions $M_{ij} = f(\mathbf r_i - \mathbf r_j),$2 lie on a uniform grid, the NU-FFT reduces to the standard FFT. In that regime the framework recovers prior $M_{ij} = f(\mathbf r_i - \mathbf r_j),$3 RPE methods for regular lattices.

For Rotary Position Embedding (RoPE), choosing

$M_{ij} = f(\mathbf r_i - \mathbf r_j),$4

implies

$M_{ij} = f(\mathbf r_i - \mathbf r_j),$5

Under this choice, the NU-FFT formula collapses to the exact closed-form RoPE rotations.

For STRING, when

$M_{ij} = f(\mathbf r_i - \mathbf r_j),$6

the quadrature frequencies $M_{ij} = f(\mathbf r_i - \mathbf r_j),$7 play the role of $M_{ij} = f(\mathbf r_i - \mathbf r_j),$8. Uniform weights $M_{ij} = f(\mathbf r_i - \mathbf r_j),$9 recover $\{\mathbf r_i\}\subset\mathbb R^3$0–STRING, while learned $\{\mathbf r_i\}\subset\mathbb R^3$1 generalize it.

For graph-based masks implemented as block-Toeplitz via spanning-tree methods, the contrast is operational rather than merely formal: those methods rely on fast tree solves but still require quadratic pre-processing, whereas RelFlexformer requires no graph construction, works on completely unstructured points, and runs in subquadratic time end-to-end.

A frequent misunderstanding would be to treat RelFlexformer as only a point-cloud adaptation of a single prior positional encoding. The formulation in fact subsumes multiple efficient RPE mechanisms under one Fourier-analytic framework, with the modulation function $\{\mathbf r_i\}\subset\mathbb R^3$2 as the unifying object.

6. Empirical evaluation on 3D benchmarks

The reported empirical study incorporates RelFlexformer into three backbone architectures: PCT for point cloud classification, PTv3 for point cloud semantic segmentation, and DFormer-Base for RGB-D semantic segmentation. The benchmarks span ModelNet40, ScanObjectNN, ScanNet v2, ScanNet200, ScanNet++, nuScenes LiDAR, S3DIS 6-fold cross-val, NYU Depth v2, and SUN RGB-D (Kim et al., 11 May 2026).

For point cloud classification with a PCT backbone, ModelNet40 is reported at 93.2% accuracy for a baseline dense Transformer, 92.34% for Performer, 92.94% for RelFlexformer, and 92.55% for PointRoPE. On ScanObjectNN, the reported overall accuracy is 84.0 for Transformer, 83.16 for Performer, and 84.45 for RelFlexformer.

For point cloud semantic segmentation with a PTv3 backbone, ScanNet v2 is reported at 77.6 mIoU for Transformer, 74.8 for Performer, and 76.8 for RelFlexformer. On ScanNet200, the corresponding numbers are 35.3, 28.2, and 34.0. On ScanNet++, they are 48.2, 48.1, and 48.7. On nuScenes LiDAR, they are 80.4, 72.0, and 80.3. On S3DIS 6-fold cross-val, they are 77.70 mIoU, 73.18, and 76.25.

For RGB-D semantic segmentation with a DFormer-Base backbone, NYU Depth v2 is reported at 55.6 mIoU for Transformer, 54.44 for Performer, and 55.32 for RelFlexformer. On SUN RGB-D, the corresponding numbers are 51.2, 48.49, and 51.04.

The source summarizes these results by stating that RelFlexformer closes most of the gap between Performer and the quadratic Transformer and, in ScanObjectNN, ScanNet200, and nuScenes, surpasses the dense-attention model. It also states that RelFlexformer outperforms the recently proposed STRING encoding at comparable complexity $\{\mathbf r_i\}\subset\mathbb R^3$3, and matches or exceeds the performance of the quadratic-attention reference without blowing up runtime or memory.

7. Scope, significance, and limitations of the formulation

RelFlexformer is presented as a principled, Fourier-analysis–driven framework for integrating arbitrary continuous RPE masks into linear-attention Transformers for 3D data (Kim et al., 11 May 2026). Its significance lies in coupling three properties that are often separated in prior work: continuous relative geometry, support for arbitrarily distributed 3D token positions, and subquadratic end-to-end attention.

Within that formulation, the most distinctive technical commitment is the requirement that the modulation function be $\{\mathbf r_i\}\subset\mathbb R^3$4-integrable. This requirement is not incidental: it is what supports the Fourier-domain construction used by FastMult$\{\mathbf r_i\}\subset\mathbb R^3$5. A plausible implication is that model design in this family is naturally expressed through the choice of physically or geometrically meaningful kernels rather than through ad hoc discrete positional tables.

The framework is also explicitly positioned against two narrower views of efficient positional encoding. First, it is not limited to structured settings with tokens embedded in unweighted grids; the stated purpose is to generalize those settings to general non-structured heterogeneous scenarios. Second, it is not dependent on graph construction or spanning-tree preprocessing, because the NU-FFT route operates directly on coordinates.

At the same time, the reported complexity guarantee is tied to the quadrature-based approximation and to NU-FFT computation. The practical quality of the approximation therefore depends on the selected frequency samples $\{\mathbf r_i\}\subset\mathbb R^3$6 and weights $\{\mathbf r_i\}\subset\mathbb R^3$7, even though the source does not recast this dependence as a separate limitation. What is established directly is that, for fixed feature dimensions and suitable frequency sampling, the asymptotic cost remains $\{\mathbf r_i\}\subset\mathbb R^3$8 while preserving flexible 3D RPE integration.