Group Autoregressive Transformer Overview

• The paper introduces a novel framework where autoregression is applied to groups of synchronized sensor views rather than individual tokens.
• The UniT architecture leverages a DINOv2 backbone and dedicated modal fusion to integrate multi-view data for unified geometry perception.
• Empirical results show that adapting the group size balances online streaming with offline processing while reducing pose estimation errors and memory usage.

A Group Autoregressive Transformer is a Transformer architecture in which the basic autoregressive unit is a group rather than an individual frame or token. In the formulation introduced by UniT, the groups are groups of sensor observations, and the model predicts the corresponding point maps through bidirectional attention within each group and causal attention across groups, thereby unifying online perception, offline reconstruction, multi-modal integration, long-horizon scalability, and metric-scale estimation within a single feed-forward model (Wang et al., 20 May 2026). Related literature does not always use the same name, but it develops closely aligned perspectives in which autoregression is organized over grouped views, patches, variables, residual branches, target buffers, or dimensions, rather than over a single flat token stream (Lu et al., 11 Feb 2025, Lu et al., 2024, Hassan et al., 10 Oct 2025, Patacchiola et al., 2024).

1. Group autoregression as a formal factorization

In UniT, the starting point is the conditional geometry perception problem

$$p\!\left(\{\mathbf{X}_t\}_{t=1}^{N} \,\middle|\, \{\mathbf{I}_t\}_{t=1}^{N}, \{\mathcal{O}_t\}_{t=1}^{N}\right),$$

where $$\mathcal{O}_t \subseteq \{\mathbf{D}_t,\mathbf{K}_t,[\mathbf{R}|\mathbf{T}]_t\}$$. A standard autoregressive factorization over frames is

$$p\!\left(\{\mathbf{X}_t\}_{t=1}^{N} \,\middle|\, \{\mathbf{I}_t\}_{t=1}^{N}, \{\mathcal{O}_t\}_{t=1}^{N}\right) = \prod_{t=1}^{N} p\!\left(\mathbf{X}_t \,\middle|\, \mathbf{I}_{\le t}, \mathcal{O}_{\le t}\right).$$

UniT generalizes this to group autoregression, where the unit at step $t$ is a group of $G$ synchronized views $\{\mathbf{I}_t^{g}\}_{g=1}^{G}$ with corresponding point maps $\{\mathbf{X}_t^{g}\}_{g=1}^{G}$:

$$\{\mathbf{X}_t^{1:G}\}_{t=1}^{N/G} \leftarrow \arg\max_{\Theta} \prod_{t=1}^{N/G} p\!\left(\mathbf{X}_t^{1:G} \,\middle|\, \mathbf{I}_{\le t}^{1:G}, \mathcal{O}_{\le t}^{1:G}\right).$$

Within a group, attention is bidirectional; across groups, attention is causal (Wang et al., 20 May 2026).

This definition makes group size $G$ the principal control variable. When $G=1$, the model operates in fully online streaming mode with one frame per autoregressive step. When $$\mathcal{O}_t \subseteq \{\mathbf{D}_t,\mathbf{K}_t,[\mathbf{R}|\mathbf{T}]_t\}$$0, all frames form one group and the model becomes fully offline. Intermediate settings support multi-camera synchronous groups or short offline chunks inside a longer stream. The term “group” therefore denotes a structural unit in the factorization rather than a fixed semantic category such as a head or a channel (Wang et al., 20 May 2026).

A common misconception is that group autoregression is merely ordinary causal decoding applied to a batched input. UniT’s formulation is stricter: the attention mask is group-causal, so the model is bidirectional inside each group and causal only between groups. This makes the architecture neither a standard decoder-only Transformer nor a purely bidirectional encoder (Wang et al., 20 May 2026).

2. UniT architecture: group-causal attention, anchor-free geometry, and modal fusion

UniT is built on a DINOv2 vision transformer backbone, followed by a “frame attention” layer that refines tokens within each frame and a “global attention” layer that attends across all frames in the active memory under the group causal mask. With modal fusion, the updated formulation is

$$\mathcal{O}_t \subseteq \{\mathbf{D}_t,\mathbf{K}_t,[\mathbf{R}|\mathbf{T}]_t\}$$1

where $$\mathcal{O}_t \subseteq \{\mathbf{D}_t,\mathbf{K}_t,[\mathbf{R}|\mathbf{T}]_t\}$$2 are modal-fused tokens. The decoder heads then produce local point maps $$\mathcal{O}_t \subseteq \{\mathbf{D}_t,\mathbf{K}_t,[\mathbf{R}|\mathbf{T}]_t\}$$3, camera extrinsics $$\mathcal{O}_t \subseteq \{\mathbf{D}_t,\mathbf{K}_t,[\mathbf{R}|\mathbf{T}]_t\}$$4, confidences, and normals (Wang et al., 20 May 2026).

The geometry representation is the point map representation popularized by DUSt3R/VGGT: a global point map $$\mathcal{O}_t \subseteq \{\mathbf{D}_t,\mathbf{K}_t,[\mathbf{R}|\mathbf{T}]_t\}$$5 and a local point map $$\mathcal{O}_t \subseteq \{\mathbf{D}_t,\mathbf{K}_t,[\mathbf{R}|\mathbf{T}]_t\}$$6 in camera coordinates, related by

$$\mathcal{O}_t \subseteq \{\mathbf{D}_t,\mathbf{K}_t,[\mathbf{R}|\mathbf{T}]_t\}$$7

UniT adopts an anchor-free design inspired by $$\mathcal{O}_t \subseteq \{\mathbf{D}_t,\mathbf{K}_t,[\mathbf{R}|\mathbf{T}]_t\}$$8. Camera poses are predicted in absolute form but evaluated through relative transformations,

$$\mathcal{O}_t \subseteq \{\mathbf{D}_t,\mathbf{K}_t,[\mathbf{R}|\mathbf{T}]_t\}$$9

and the losses are imposed on induced relative poses and point maps rather than on a designated reference view (Wang et al., 20 May 2026).

The multi-modal component is a dedicated Modal Attention module. DINOv2 provides image tokens; a two-layer MLP with SP-Normalization encodes optional modalities into point tokens and pose tokens; and ModalAttn concatenates image and modal tokens at aligned spatial positions before attention. The module is inserted at four layers, specifically at layers $$p\!\left(\{\mathbf{X}_t\}_{t=1}^{N} \,\middle|\, \{\mathbf{I}_t\}_{t=1}^{N}, \{\mathcal{O}_t\}_{t=1}^{N}\right) = \prod_{t=1}^{N} p\!\left(\mathbf{X}_t \,\middle|\, \mathbf{I}_{\le t}, \mathcal{O}_{\le t}\right).$$0 in a 24-layer stack, and these modules are only approximately $$p\!\left(\{\mathbf{X}_t\}_{t=1}^{N} \,\middle|\, \{\mathbf{I}_t\}_{t=1}^{N}, \{\mathcal{O}_t\}_{t=1}^{N}\right) = \prod_{t=1}^{N} p\!\left(\mathbf{X}_t \,\middle|\, \mathbf{I}_{\le t}, \mathcal{O}_{\le t}\right).$$1 of the parameters. Missing modalities are encoded as zero matrices. This arrangement is designed to preserve pixel-wise spatial correspondence while allowing joint reasoning over image, depth, intrinsics, and extrinsics (Wang et al., 20 May 2026).

This architecture establishes a precise meaning for “group” in the original term: a group is a set of synchronized observations whose internal relations are modeled bidirectionally, while causal structure is reserved for relations among groups. That distinction is central to the model’s unification of online and offline geometry perception (Wang et al., 20 May 2026).

3. Online/offline unification, bounded memory, and scale-adaptive learning

UniT’s online and offline modes are not separate models. They are the same Group Autoregressive Transformer run with different group sizes. Larger groups enable more bidirectional multi-view fusion in a single step, while smaller groups induce more autoregressive steps and more sequential adaptation. The paper reports that pose estimation on ScanNetV2 with 90 frames improves as group size increases, with larger groups consistently reducing ATE/RPE, while UniT remains stable across a wide range of $$p\!\left(\{\mathbf{X}_t\}_{t=1}^{N} \,\middle|\, \{\mathbf{I}_t\}_{t=1}^{N}, \{\mathcal{O}_t\}_{t=1}^{N}\right) = \prod_{t=1}^{N} p\!\left(\mathbf{X}_t \,\middle|\, \mathbf{I}_{\le t}, \mathcal{O}_{\le t}\right).$$2 (Wang et al., 20 May 2026).

Long-horizon scalability is handled through a queue-style KV caching mechanism. Standard autoregressive Transformers retain all past keys and values, so memory grows with sequence length. UniT instead keeps a fixed-length queue of cached tokens per layer. When the queue exceeds capacity $$p\!\left(\{\mathbf{X}_t\}_{t=1}^{N} \,\middle|\, \{\mathbf{I}_t\}_{t=1}^{N}, \{\mathcal{O}_t\}_{t=1}^{N}\right) = \prod_{t=1}^{N} p\!\left(\mathbf{X}_t \,\middle|\, \mathbf{I}_{\le t}, \mathcal{O}_{\le t}\right).$$3, old entries are dropped according to a policy such as first-in–first-out, random drop, token merge, or stride drop. Under this mechanism, per-step complexity is $$p\!\left(\{\mathbf{X}_t\}_{t=1}^{N} \,\middle|\, \{\mathbf{I}_t\}_{t=1}^{N}, \{\mathcal{O}_t\}_{t=1}^{N}\right) = \prod_{t=1}^{N} p\!\left(\mathbf{X}_t \,\middle|\, \mathbf{I}_{\le t}, \mathcal{O}_{\le t}\right).$$4 and memory is $$p\!\left(\{\mathbf{X}_t\}_{t=1}^{N} \,\middle|\, \{\mathbf{I}_t\}_{t=1}^{N}, \{\mathcal{O}_t\}_{t=1}^{N}\right) = \prod_{t=1}^{N} p\!\left(\mathbf{X}_t \,\middle|\, \mathbf{I}_{\le t}, \mathcal{O}_{\le t}\right).$$5, independent of total sequence length $$p\!\left(\{\mathbf{X}_t\}_{t=1}^{N} \,\middle|\, \{\mathbf{I}_t\}_{t=1}^{N}, \{\mathcal{O}_t\}_{t=1}^{N}\right) = \prod_{t=1}^{N} p\!\left(\mathbf{X}_t \,\middle|\, \mathbf{I}_{\le t}, \mathcal{O}_{\le t}\right).$$6; stride-based drop gives the best trade-off and near-full-cache performance (Wang et al., 20 May 2026).

The feasibility of bounded memory is tied to anchor-free relational modeling. Because losses are imposed on relative poses and relative or partially absolute point-map constraints rather than on coordinates fixed to the first frame, late predictions need not depend on the earliest frames. This reduces long-range dependence on a specific anchor and permits outdated memory to be discarded on the fly. In the paper’s interpretation, geometry is continually re-anchored via relative relationships among recent views (Wang et al., 20 May 2026).

Metric-scale estimation is addressed by a scale-adaptive geometry loss. UniT couples relative, scale-invariant constraints with a partial absolute point-map term:

$$p\!\left(\{\mathbf{X}_t\}_{t=1}^{N} \,\middle|\, \{\mathbf{I}_t\}_{t=1}^{N}, \{\mathcal{O}_t\}_{t=1}^{N}\right) = \prod_{t=1}^{N} p\!\left(\mathbf{X}_t \,\middle|\, \mathbf{I}_{\le t}, \mathcal{O}_{\le t}\right).$$7

The intended behavior is a progressive transition from scale-invariant geometry to metric-scale solutions. In the reported ablation, direct metric regression yields an average error of $$p\!\left(\{\mathbf{X}_t\}_{t=1}^{N} \,\middle|\, \{\mathbf{I}_t\}_{t=1}^{N}, \{\mathcal{O}_t\}_{t=1}^{N}\right) = \prod_{t=1}^{N} p\!\left(\mathbf{X}_t \,\middle|\, \mathbf{I}_{\le t}, \mathcal{O}_{\le t}\right).$$8, the scale-adaptive design reduces the average to $$p\!\left(\{\mathbf{X}_t\}_{t=1}^{N} \,\middle|\, \{\mathbf{I}_t\}_{t=1}^{N}, \{\mathcal{O}_t\}_{t=1}^{N}\right) = \prod_{t=1}^{N} p\!\left(\mathbf{X}_t \,\middle|\, \mathbf{I}_{\le t}, \mathcal{O}_{\le t}\right).$$9, and adding shuffled normal further improves it to $t$0 (Wang et al., 20 May 2026).

4. Related autoregressive lineages in time series, probabilistic inference, and density estimation

Although UniT introduces the term explicitly for 3D geometry, adjacent literatures provide compatible constructions in which groups or structured blocks become the effective autoregressive unit.

In multivariate time-series forecasting, a single linear attention layer without softmax can be written as

$t$1

which is interpreted as a dynamic VAR($t$2). After rearranging the architecture, multi-layer linear attention remains a dynamic VAR over the original keys with recursively defined coefficients

$t$3

and the resulting model, SAMoVAR, is described as a Structural Aligned Mixture of VAR (Lu et al., 11 Feb 2025). The phrase “Group Autoregressive Transformer” is not used there, but the paper explicitly argues that block-diagonal, block-sparse, clustered, or mixture-based structure on the dynamic VAR weights $t$4 would be a natural grouped extension. This suggests a grouped autoregressive interpretation in which groups are blocks of variables, sectors, spatial clusters, or regimes (Lu et al., 11 Feb 2025).

WAVE develops a different decomposition by embedding a full ARMA structure into autoregressive attention. The AR term is the usual causal attention output, while the MA term is another attention pass over residuals

$t$5

with implicitly generated MA weights. The paper treats heads, channels, and time-scales as natural axes of decoupling, and the details explicitly reinterpret this as a route to a “Group Autoregressive Transformer” in which each head or feature group carries its own AR+MA dynamics (Lu et al., 2024).

GBT offers a staged interpretation rather than a masked-attention one. It separates a time-series forecasting Transformer into an Auto-Regression stage that maps past inputs to a future horizon and a Self-Regression stage that refines the resulting “Good Beginning” using masked self-attention plus Error Score Modification. The paper does not define explicit groups, but it describes the architecture as a two-stage organization of autoregressive roles and is presented in the details as a grouped or staged autoregressive Transformer design (Shen et al., 2023).

In amortized probabilistic inference, the causal autoregressive buffer separates a static context cache from a dynamic target buffer. The factorization

$t$6

preserves set-based conditioning while enabling efficient autoregressive generation and one-pass joint log-likelihood evaluation (Hassan et al., 10 Oct 2025). The paper’s details state that block-causal masks over grouped targets are a direct extension, making the buffer a concrete template for groupwise autoregression over target blocks rather than single targets (Hassan et al., 10 Oct 2025).

In density estimation, Transformer Neural Autoregressive Flows treat each dimension as a token and use an autoregressive attention mask so that the transformer outputs the parameters of an invertible per-dimension transform. The paper does not define groups, but its details state that a grouped version would treat each group as a super-token and parameterize a groupwise invertible map, yielding block-lower-triangular Jacobian structure (Patacchiola et al., 2024). This suggests that “group autoregressive transformer” can also denote an ordering over blocks of dimensions rather than over time or views.

5. Optimization behavior, design trade-offs, and recurrent variants

Grouped autoregression does not eliminate optimization problems. Tiny Autoregressive Recursive Models provide a controlled comparison between standard autoregressive Transformers and several recurrent, two-stream, and nested-refinement alternatives under matched compute. The paper finds that some two-level refinement baselines show strong performance, especially a flat two-stream recurrent model, but reports no reliable performance gains from the full Autoregressive TRM architecture (Rauba et al., 9 Mar 2026).

The main technical distinction is between flat two-stream refinement and nested hierarchical refinement. In the original TRM-style schedule,

$t$7

the solution stream is updated only after multiple inner updates of the auxiliary stream. In the autoregressive projection studied in the paper, this nested structure, combined with terminal-iterate readout, creates a difficult credit-assignment problem and does not reliably improve generalization per unit compute (Rauba et al., 9 Mar 2026).

This caution matters for group autoregressive designs more broadly. WAVE argues that separate AR and MA branches are superior to folding local smoothing directly into the AR path, because AR attention can remain globally sensitive while the MA branch handles short-term fluctuations (Lu et al., 2024). SAMoVAR likewise argues that structural alignment with the autoregressive objective improves interpretability and generalization, whereas standard multi-layer Transformer stacks create a mismatch between residual-space computations and lag-based forecasting structure (Lu et al., 11 Feb 2025). Taken together, these works indicate that “grouping” is useful when it expresses a genuine decomposition of dependencies, but not when it merely adds nested computation without a well-aligned supervisory path.

A second misconception is that the term always refers to the same grouping axis. In the cited literature, “group” can mean synchronized views, variable blocks, AR and MA branches, staged forecasting roles, target buffers, heads, channels, time-scales, or dimension blocks. The common denominator is not the semantic identity of the group; it is the replacement of a flat tokenwise autoregressive chain with a structured autoregressive unit.

6. Empirical status, limitations, and future directions

Empirically, UniT reports state-of-the-art performance in unified geometry perception on ten benchmarks spanning seven representative tasks, with a single model trained once on 21 public metric-scale datasets (Wang et al., 20 May 2026). The same architecture supports multi-view reconstruction, camera pose estimation, video depth estimation, monocular depth estimation, long-horizon perception, multi-modal reconstruction, and depth completion by varying inference configuration rather than retraining. The paper also reports that offline UniT($t$8) has comparable parameters, approximately $t$9B, and FPS to VGGT, while online UniT($G$0) with queue capacity $G$1 has bounded memory of $G$2 GiB and FPS of $G$3 (Wang et al., 20 May 2026).

In time-series forecasting, the structurally aligned view has likewise been empirically favorable. SAMoVAR is reported as top-1 on most datasets, with the best average rank, and its gains are especially large on stably structured data where the dynamics are more VAR-like (Lu et al., 11 Feb 2025). WAVE reports that ARMA attention consistently improves various autoregressive attentions on time-series forecasting tasks and achieves state-of-the-art results (Lu et al., 2024). GBT reports that its two-stage framework outperforms SOTA TSFTs and many other forecasting models while using only canonical attention and convolution and less time and space complexity (Shen et al., 2023).

The limitations are domain-specific but structurally informative. UniT is a large model requiring significant training resources, specifically 64 H100 GPUs for approximately seven days, and its current ModalAttn handles image, depth, intrinsics, and extrinsics rather than a broader sensor suite (Wang et al., 20 May 2026). SAMoVAR notes that its experiments are on time-series forecasting rather than general sequence modeling, that the VAR structure is linear in observations, and that it does not explicitly model hierarchical group structures (Lu et al., 11 Feb 2025). The causal autoregressive buffer still incurs a $G$4 term in buffer length, although it replaces naive autoregressive re-encoding with $G$5 complexity and can deliver up to 20 times faster joint sampling (Hassan et al., 10 Oct 2025). T-NAFs retain quadratic attention cost in the number of dimensions or groups, even though they reduce parameter count and improve stability relative to constrained masked-flow alternatives (Patacchiola et al., 2024).

Future directions stated or strongly suggested in these papers converge on a common agenda: block-sparse or hierarchical masks for grouped dependencies, explicit mixture-of-VAR components with gates, richer feature maps or nonlinear kernels, broader modality sets, and more aggressive memory compression (Lu et al., 11 Feb 2025, Wang et al., 20 May 2026). A plausible implication is that the Group Autoregressive Transformer is best understood not as a single architecture but as a general design pattern: define an autoregressive unit larger than a token, impose a causal structure over those units, and preserve whatever within-group symmetry or bidirectionality the application requires.