C^2RoPE: Causal Continuous Rotary Encoding

• C^2RoPE is a positional encoding method for 3D multimodal models that integrates a spatio-temporal triplet (m, x, y) to restore spatial continuity.
• It addresses limitations of standard RoPE by incorporating 2D spatial coordinates and implementing Chebyshev causal masking to reduce long-term attention decay.
• Empirical evaluations on benchmarks like ScanQA and SQA3D show that C^2RoPE improves visual question answering and 3D scene reasoning performance.

C$^2$RoPE, short for Causal Continuous Rotary Positional Encoding, is a positional encoding method introduced for 3D Large Multimodal Models (LMMs) built on LLMs. It is designed to address two limitations inherited from standard Rotary Position Embedding (RoPE) in multimodal visual processing: the loss of 2D spatial continuity caused by flattening images into 1D token sequences, and the long-term decay in attention allocation that causes earlier visual tokens to be progressively neglected as sequence length grows. The method combines a spatio-temporal continuous positional embedding mechanism based on a triplet index $(m,x,y)$ with Chebyshev Causal Masking, which defines causal dependencies from 2D spatial neighborhoods rather than purely from 1D sequence order. The formulation was presented for 3D scene reasoning and 3D visual question answering, where it was evaluated on ScanQA and SQA3D (Ye et al., 11 Feb 2026).

1. Problem setting and motivation

In the setting targeted by C$^2$RoPE, 3D LMMs follow the now-dominant paradigm of aligning 3D visual features with LLM representations. Within this architecture, positional encoding remains inherited from language-oriented transformer practice, especially standard RoPE. The central claim of the method is that this inheritance is not neutral for visual reasoning: the positional prior that is useful for autoregressive text can be misaligned with the structure of image and multi-view 3D inputs (Ye et al., 11 Feb 2026).

The first limitation is spatial-continuity loss. In standard RoPE, a 2D image is flattened into a 1D token sequence using row-major, or raster-scan, ordering. Each patch token receives a single index $m \in \{1,\dots,v\}$. Although neighboring tokens within a row are assigned consecutive indices, tokens occupying the same column across adjacent rows become far apart in the 1D ordering. This creates a break in the column dimension, so visual neighbors in 2D can be treated as distant under RoPE. In the terminology of the paper, this disrupts spatial locality.

The second limitation is long-term attention decay, described in the paper as visual tokens neglect. RoPE injects a relative positional bias through rotations $R_{n-m}$ whose sinusoidal structure decays as $|n-m|$ grows. In natural language this encourages emphasis on recent context, but for long multimodal sequences, such as multi-view 3D scenes with thousands of patches, the same bias causes early visual tokens to receive exponentially attenuated attention. The paper reports information-flow analyses on a 3K subset of 3D VQA questions, showing concentrated attention on the final few visual tokens and severe neglect of the bulk of earlier tokens (Ye et al., 11 Feb 2026).

These two observations motivate the design goal of C$^2$RoPE: to preserve local spatial continuity while redefining causal structure for visual tokens in a way that is grounded in 2D layout rather than only in 1D temporal order.

2. Spatio-temporal triplet hybrid positional index

C$^2$RoPE replaces the single visual positional index with a triplet hybrid positional index $(m,x,y)$. This construction is intended to restore spatial continuity without discarding the original sequence order used by the underlying LLM stack (Ye et al., 11 Feb 2026).

The components of the triplet have distinct roles. The variable $m \in \{1,\dots,v\}$ is the original row-major RoPE sequence index and is shared with text tokens so as not to disturb text processing. The variables $(m,x,y)$0 are integer Cartesian coordinates of the patch within its image view. The image is centered at $(m,x,y)$1; $(m,x,y)$2 increases rightward and $(m,x,y)$3 increases upward, with each patch step incrementing the corresponding coordinate by $(m,x,y)$4.

The encoding allocates different subsets of the rotary dimensions to different components of the triplet. The paper states that, with $(m,x,y)$5 in the experiments, dimensions 1 through 96 encode the temporal component $(m,x,y)$6 at high and mid frequencies using the original RoPE bands $(m,x,y)$7, while dimensions 97 through 112 encode $(m,x,y)$8 at lower-frequency bands and dimensions 113 through 128 encode $(m,x,y)$9 similarly at lower frequencies. The intended effect is asymmetric: temporal order is preserved with the usual RoPE spectrum, whereas spatial coordinates are encoded more coarsely to emphasize locality.

The exposition describes the resulting embedding as a continuous rotary embedding $^2$0 formed by interleaving sine-cosine pairs over the reserved frequency bands. No additional normalization is applied beyond centering $^2$1 at zero. This suggests a design in which the visual token simultaneously carries a standard sequence-compatible phase component and an explicitly spatial phase component, rather than relying on the 1D flattening order to implicitly represent both.

A plausible implication is that C$^2$2RoPE preserves compatibility with pretrained text-side positional behavior because the original sequence index $^2$3 remains present, while augmenting the visual side with coordinate-aware structure that 1D RoPE does not capture.

3. Chebyshev causal masking

The second mechanism in C$^2$4RoPE is Chebyshev Causal Masking, introduced to mitigate the long-term decay bias associated with purely 1D causal masking. Instead of treating causality as a function only of sequence lag, the mask determines allowable dependencies by 2D Chebyshev distance between image tokens (Ye et al., 11 Feb 2026).

For tokens $^2$5 and $^2$6 with spatial coordinates $^2$7 and $^2$8, the Chebyshev distance is defined as

$^2$9

Using this distance, the additive attention mask is defined as

$m \in \{1,\dots,v\}$0

where $m \in \{1,\dots,v\}$1 is a chosen radius, often the maximum Chebyshev distance within one view.

The attention computation then becomes

$m \in \{1,\dots,v\}$2

Under this rule, a token may attend only to visual tokens lying within its Chebyshev neighborhood, regardless of the 1D distance in $m \in \{1,\dots,v\}$3. The paper’s framing is not that causality is removed, but that it is redefined spatially for visual processing. This is intended to preserve access to spatially adjacent tokens even when raster-scan ordering places them far apart in the flattened sequence.

The significance of the Chebyshev choice, rather than a purely axial metric, is that it defines square neighborhoods in 2D grid space and therefore treats horizontal, vertical, and diagonal adjacency uniformly at a given radius. The paper contrasts this with the prior embedded in standard RoPE, where tokens that are temporally closer are assumed to be more causally related.

4. Integration into transformer-based multimodal architectures

C$m \in \{1,\dots,v\}$4RoPE is integrated as a replacement for standard RoPE in the attention stack of a multimodal LMM. The paper specifies that it replaces standard RoPE in all self-attention layers of the multimodal model, both during visual-text cross-attention and during autoregressive decoding (Ye et al., 11 Feb 2026).

After projection, the continuous embedding $m \in \{1,\dots,v\}$5 yields token embeddings $m \in \{1,\dots,v\}$6, and rotary rotations $m \in \{1,\dots,v\}$7 are applied to the query and key tensors in each attention head. The new rotation is built by stacking $m \in \{1,\dots,v\}$8 blocks of cosine and sine terms for the temporal bands and likewise for the $m \in \{1,\dots,v\}$9 and $R_{n-m}$0 bands. In parallel, the Chebyshev mask $R_{n-m}$1 is added to the pre-softmax attention logits.

The paper characterizes this as a dual mechanism. The continuous triplet indexing is responsible for restoring spatial adjacency inside the rotary encoding itself, while the spatial mask prevents the model from over-decaying attention to tokens that are distant in 1D order but near in 2D space. In that sense, C$R_{n-m}$2RoPE modifies both the relative phase geometry of attention and the allowable dependency graph over visual tokens.

Because the original sequence index $R_{n-m}$3 is preserved and shared with text tokens, the method is explicitly designed not to disturb text processing. This places it within a class of multimodal positional adaptations that attempt to alter the visual inductive bias while leaving the text-side LLM prior largely intact.

5. Empirical evaluation

The empirical evaluation reported for C$R_{n-m}$4RoPE covers 3D scene reasoning and 3D visual question answering. The benchmarks named in the paper are ScanQA, described as containing 33.4K human-annotated Q&A and measuring spatial reasoning in 3D scenes, and SQA3D, described as containing approximately 19K GPT-4-generated questions over dynamic 3D scenes (Ye et al., 11 Feb 2026).

The main quantitative gains are reported relative to the LLaVA-3D baseline.

Benchmark	Metric changes over LLaVA-3D baseline
ScanQA	EM@1: $R_{n-m}$5 $R_{n-m}$6; BLEU-4: $R_{n-m}$7 $R_{n-m}$8; METEOR: $R_{n-m}$9 $|n-m|$0; ROUGE: $|n-m|$1 $|n-m|$2; CIDEr: $|n-m|$3 $|n-m|$4
SQA3D	EM@1: $|n-m|$5 $|n-m|$6; Refined-EM: $|n-m|$7 $|n-m|$8

The paper also reports an ablation study on the SQA3D test/val split comparing standard RoPE with alternative causal-attention designs.

Method	SQA3D test metrics
LLaVA-3D (RoPE)	EM@1 = 55.6, EM@R = 53.1
+MCA (Manhattan Causal Attention)	55.5 / 53.7
+CCA (Concentric Causal Attention)	56.2 / 53.9
+C$|n-m|$9RoPE (ours)	56.8 / 54.3

These ablations are described as isolating the benefits of the proposed spatio-temporal indexing and Chebyshev masking over heuristic 1D re-indexing methods. The paper further states that case studies confirm that C$^2$0RoPE avoids hallucinations and more accurately grounds responses in the visual input.

In aggregate, the empirical record presented in the paper supports three claims: restoration of 2D spatial continuity, alleviation of long-term attention decay for early visual tokens, and consistent gains on 3D reasoning and VQA tasks. The strongest improvements reported numerically occur on ScanQA, especially for BLEU-4, METEOR, and CIDEr.

6. Limitations, extensions, and naming ambiguity

The paper explicitly notes several limitations and possible extensions of C$^2$1RoPE. The current $^2$2 indexing does not exploit full 3D geometry such as depth $^2$3 or point-cloud coordinates; one proposed extension is to generalize the index to $^2$4. The Chebyshev mask uses a fixed radius $^2$5, and the paper suggests that this radius might instead be adaptive or learned per scene or per head. It also suggests that the same combination of continuous indexing and causal masking could be applied to video or to higher-dimensional modalities, such as a 4D spatio-temporal volume (Ye et al., 11 Feb 2026).

A separate source introduces a potential nomenclature issue. In the exposition associated with "Jordan-RoPE: Non-Semisimple Relative Positional Encoding via Complex Jordan Blocks", the label C$^2$6RoPE is used differently, as "Complex-Coupled Rotary plus Polynomial Extension", for a non-semisimple relative positional encoding based on complex Jordan blocks (Zhang, 5 May 2026). That use is distinct from the 3D multimodal method discussed here, where C$^2$7RoPE means Causal Continuous Rotary Positional Encoding (Ye et al., 11 Feb 2026).

This naming overlap can lead to confusion because the two methods operate in different technical regimes. The 3D multimodal C$^2$8RoPE modifies visual positional structure through a triplet index and spatial masking, whereas Jordan-RoPE introduces oscillatory-polynomial features such as $^2$9 and $^2$0 through defective Jordan blocks. The overlap is therefore terminological rather than methodological.

For readers tracking positional encoding research, the most precise interpretation is contextual: in the domain of 3D LMM reasoning, C$^2$1RoPE refers to the causal continuous formulation for multimodal visual processing; in the Jordan-RoPE exposition, the same string is reused for a complex-coupled polynomial extension of rotary encoding.