TensFormer: Hierarchical Occupancy Forecasting

• TensFormer is a hierarchical Transformer for occupancy world modeling that employs Temporal Next-Scale Prediction to decouple temporal scene evolution from spatial scale refinement.
• It integrates multi-scale scene tokenization with holistic pose aggregation, converting 3D occupancy scenes and motion into a unified autoregressive sequence.
• Empirical results on nuScenes demonstrate improved mIoU and efficiency, offering a controllable trade-off between generation speed and scene fidelity.

Searching arXiv for the exact and closely related terms to ground the article in current papers. arXiv search query: "TensFormer OccTENS temporal next-scale prediction" TensFormer is the generative transformer at the core of OccTENS, a 3D occupancy world model for autonomous driving. It is designed to predict future occupancy scenes and ego motion from historical observations after those observations have been discretized by a scene tokenizer and a motion tokenizer. Its central design choice is to replace flat next-token autoregression with Temporal Next-Scale Prediction (TENS), which factorizes generation into temporal scene-by-scene prediction and spatial scale-by-scale generation. In the formulation reported for OccTENS, TensFormer is intended to address the inefficiency, temporal degradation, and weak controllability attributed to prior autoregressive occupancy world models (Jin et al., 4 Sep 2025).

1. Problem setting and representational scope

TensFormer operates in the setting of occupancy world modeling, where the target is a sequence of future 3D occupancy scenes rather than ordinary image frames. The motivating difficulty is that occupancy generation must capture fine-grained 3D geometry, semantic occupancy structure, temporal evolution, long-range scene coherence, and ego-motion dependence. The model is therefore embedded in a pipeline with two tokenizers. A scene tokenizer converts a 3D occupancy scene into multi-scale discrete tokens, and a motion tokenizer discretizes ego-motion through relative pose $(x,y,\theta)$. TensFormer then performs sequence generation over these occupancy and motion tokens (Jin et al., 4 Sep 2025).

In the reported formulation, a scene $\mathbf{S}$ is encoded into a BEV latent feature map

$\mathbf{F} \in \mathbb{R}^{H \times W \times C},$

after which a multi-scale quantizer discretizes $\mathbf{F}$ into

$$\mathbf{F} = (\mathbf{f}^1,\mathbf{f}^2,\dots,\mathbf{f}^M),$$

where each $\mathbf{f}^m$ corresponds to one spatial scale. This is explicitly contrasted with local VQ tokenization that assigns codebook entries only to small local patches; the reported argument is that multi-scale tokenization better preserves global context. Motion is discretized separately and inserted into the same autoregressive sequence as a token $\mathbf{f}_t^0$, so that each frame is represented by

$$\{ \mathbf{f}_t^0, \mathbf{f}_t^1,\dots,\mathbf{f}_t^M \}.$$

The paper does not describe TensFormer as a generic tensorized Transformer in the sense of tensor decomposition or multilinear attention. Its domain specificity is instead architectural and generative: it is a hierarchical Transformer for occupancy forecasting, with explicit handling of time, scale, and motion in a unified sequence model (Jin et al., 4 Sep 2025).

2. Temporal Next-Scale Prediction

The defining formalism of TensFormer is Temporal Next-Scale Prediction (TENS). Standard autoregressive modeling over a flattened $n \times n$ feature map is written as

$$p(x)=\prod_{i=1}^{n\times n} p(x_i \mid x_1,x_2,\ldots,x_{i-1}),$$

which corresponds to next-token factorization over a single long sequence. OccTENS reformulates this problem so that temporal dependencies are modeled scene by scene, while spatial detail is modeled scale by scale (Jin et al., 4 Sep 2025).

The temporal factorization is given as

$\mathbf{S}$0

where $\mathbf{S}$1 denotes the tokenized representation of frame $\mathbf{S}$2. Within a frame, scales are generated autoregressively: $\mathbf{S}$3 After motion is integrated, the joint occupancy-motion model becomes

$\mathbf{S}$4

The stated purpose of this factorization is to separate temporal causality from intra-frame spatial dependency. The paper also notes that a naive next-scale adaptation with $\mathbf{S}$5 scales for an $\mathbf{S}$6 feature map has approximate total token count

$\mathbf{S}$7

and uses this discussion to motivate a structured decomposition rather than a flat extension of ordinary autoregression. In the reported interpretation, coarse scales capture global scene layout, while finer scales refine local detail; time modeling then governs scene evolution across frames (Jin et al., 4 Sep 2025).

3. Architectural organization and attention structure

TensFormer has two main generation components: temporal scene-by-scene prediction and spatial scale-by-scale generation. The scene-by-scene module is further decomposed into scale-wise temporal causal attention and frame-wise spatial attention. This decomposition is one of the central architectural claims of the model (Jin et al., 4 Sep 2025).

For temporal prediction, the model preserves frame causality: $\mathbf{S}$8 The paper argues that plain frame-wise causal attention can induce structure collapse when generating large-scale occupancy tokens because it mixes inter-frame temporal causality with intra-frame bidirectional spatial dependency. TensFormer therefore decouples these dependencies. For tokens $\mathbf{S}$9 at scale $\mathbf{F} \in \mathbb{R}^{H \times W \times C},$0 in frame $\mathbf{F} \in \mathbb{R}^{H \times W \times C},$1, scale-wise temporal causal attention is restricted to

$\mathbf{F} \in \mathbb{R}^{H \times W \times C},$2

while frame-wise spatial attention uses full attention within the current frame. Spatial refinement then proceeds through a block-wise causal attention mask so that each token at scale $\mathbf{F} \in \mathbb{R}^{H \times W \times C},$3 attends only to its prefix: $\mathbf{F} \in \mathbb{R}^{H \times W \times C},$4

The architectural details reported explicitly are limited. TensFormer contains three blocks, with 4 layers each, hidden dimension 128, and 4 attention heads. Three 1-D sine-cosine embeddings are added: position embedding, scale embedding, and time embedding. The paper does not provide detailed equations for Q/K/V projections, FFN structure, normalization style, or exact per-scale token layout, and it explicitly notes that some low-level block internals are not specified in the text (Jin et al., 4 Sep 2025).

4. Pose aggregation and controllable generation

A distinctive aspect of TensFormer is the treatment of ego motion as part of the same token sequence as occupancy. Motion is encoded from relative pose $\mathbf{F} \in \mathbb{R}^{H \times W \times C},$5, with the $\mathbf{F} \in \mathbb{R}^{H \times W \times C},$6-axis ignored, and the tokenizer maps it to an embedding

$\mathbf{F} \in \mathbb{R}^{H \times W \times C},$7

In the resulting sequence, the motion token is treated as the 0-th scale token: $\mathbf{F} \in \mathbb{R}^{H \times W \times C},$8 so that each frame is represented as

$\mathbf{F} \in \mathbb{R}^{H \times W \times C},$9

This design is termed holistic pose aggregation or multi-modal camera pose aggregation in the supplied description (Jin et al., 4 Sep 2025).

The reported significance of this arrangement is that occupancy generation and ego-motion prediction are not handled in separate branches. Instead, they are modeled through one unified autoregressive sequence. The paper states that this enables two modes of use: conditioning occupancy generation on a specified trajectory, or predicting motion jointly for planning. The relevant joint factorization is

$\mathbf{F}$0

Qualitatively, the paper reports that manipulated camera pose inputs yield occupancy generations consistent with turning and lane changing. The mechanism of controllability is therefore sequence-level conditioning through joint motion-occupancy token modeling, rather than an external controller or a separate conditioning network (Jin et al., 4 Sep 2025).

5. Training setup, objectives, and empirical behavior

OccTENS trains TensFormer in a two-stage pipeline: tokenizer training followed by world-model training. For the scene tokenizer, the loss is

$\mathbf{F}$1

with

$\mathbf{F}$2

For TensFormer itself, the world model loss is

$\mathbf{F}$3

with

$\mathbf{F}$4

Both $\mathbf{F}$5 and $\mathbf{F}$6 are cross-entropy losses over discrete tokens (Jin et al., 4 Sep 2025).

The reported setup uses 2-second historical context = 4 frames, predicts 3-second future = 6 frames, applies occupancy downsampling factor 8, and uses codebook size 4096, codebook embedding dimension 128, and 6 scales with $\mathbf{F}$7. The world model uses the architectural settings already noted: 3 blocks, 4 layers each, hidden dimension 128, and 4 heads.

The main forecasting results are reported on nuScenes 4D occupancy forecasting. Using occupancy as input, the average scores are:

• OccWorld-O: mIoU $\mathbf{F}$8, IoU $\mathbf{F}$9
• OccLLaMA-O: mIoU $$\mathbf{F} = (\mathbf{f}^1,\mathbf{f}^2,\dots,\mathbf{f}^M),$$0, IoU $$\mathbf{F} = (\mathbf{f}^1,\mathbf{f}^2,\dots,\mathbf{f}^M),$$1
• OccTENS-O: mIoU $$\mathbf{F} = (\mathbf{f}^1,\mathbf{f}^2,\dots,\mathbf{f}^M),$$2, IoU $$\mathbf{F} = (\mathbf{f}^1,\mathbf{f}^2,\dots,\mathbf{f}^M),$$3

Using camera-derived occupancy as input, the average scores are:

• OccWorld-F: mIoU $$\mathbf{F} = (\mathbf{f}^1,\mathbf{f}^2,\dots,\mathbf{f}^M),$$4, IoU $$\mathbf{F} = (\mathbf{f}^1,\mathbf{f}^2,\dots,\mathbf{f}^M),$$5
• OccLLaMA-F: mIoU $$\mathbf{F} = (\mathbf{f}^1,\mathbf{f}^2,\dots,\mathbf{f}^M),$$6, IoU $$\mathbf{F} = (\mathbf{f}^1,\mathbf{f}^2,\dots,\mathbf{f}^M),$$7
• OccTENS-F: mIoU $$\mathbf{F} = (\mathbf{f}^1,\mathbf{f}^2,\dots,\mathbf{f}^M),$$8, IoU $$\mathbf{F} = (\mathbf{f}^1,\mathbf{f}^2,\dots,\mathbf{f}^M),$$9

For planning, the reported averages are:

• OccWorld: avg $\mathbf{f}^m$0 $\mathbf{f}^m$1, avg collision $\mathbf{f}^m$2
• OccLLaMA: avg $\mathbf{f}^m$3 $\mathbf{f}^m$4, avg collision $\mathbf{f}^m$5
• OccTENS: avg $\mathbf{f}^m$6 $\mathbf{f}^m$7, avg collision $\mathbf{f}^m$8

Efficiency is reported through a scale-number ablation. Latencies are:

• OccWorld: $\mathbf{f}^m$9 s
• OccTENS, 2 scales: $\mathbf{f}_t^0$0 s
• OccTENS, 4 scales: $\mathbf{f}_t^0$1 s
• OccTENS, 6 scales: $\mathbf{f}_t^0$2 s
• OccTENS, 8 scales: $\mathbf{f}_t^0$3 s
• OccSora: around $\mathbf{f}_t^0$4 s

The paper uses these results to argue for a controllable quality-efficiency trade-off through the number of scales: 2 scales is the fastest and already stronger than OccWorld, while larger numbers of scales increase fidelity at increased latency. It also reports qualitatively that OccWorld exhibits repetition artifacts, whereas OccTENS produces more diverse and realistic occupancy scenes and maintains stronger long-term coherence (Jin et al., 4 Sep 2025).

6. Position within transformer research and terminological boundaries

Within the supplied sources, the exact term TensFormer appears as the internal Transformer of OccTENS (Jin et al., 4 Sep 2025). This should be distinguished from several adjacent but non-identical uses of tensor-oriented or similarly named Transformer work.

TEAFormer is a TEnsor-Augmented Transformer for multi-dimensional time series forecasting. It preserves matrix- and tensor-variate structure through tensor expansion and Tucker-based compression inside existing time-series Transformer pipelines. The supplied description explicitly states that TEAFormer is related to the query “TensFormer” in the intuitive sense of a tensor-aware Transformer, but also states that the paper’s exact term is TEAFormer, not TensFormer (Kong et al., 2024).

TensorLens is not a new Transformer architecture, but a tensor-based formulation for analyzing an existing Transformer as a single input-dependent linear operator encoded by a 4th-order attention-interaction tensor

$\mathbf{f}_t^0$5

Its contribution is a global tensor view of attention, FFNs, activations, LayerNorm, residuals, and embeddings, rather than a new generative model for occupancy forecasting (Atad et al., 25 Jan 2026).

Accordingly, TensFormer in the strict sense is not a general-purpose tensorized Transformer, nor an analysis framework, nor a Tucker-compressed sequence model. It is a domain-specific hierarchical generator that combines temporal scene-by-scene prediction, spatial scale-by-scale generation, and holistic pose aggregation for occupancy world modeling (Jin et al., 4 Sep 2025). The broader literature nevertheless places it within a family of Transformer research that uses additional structure—scale, tensor organization, or high-order operators—to address weaknesses of flat sequence modeling.