Cumulative-Depth RoPE in Transformers

• Cumulative-depth RoPE is a descriptive term for the layerwise intensification of phase effects observed in standard RoPE within Transformer architectures.
• It demonstrates how nonlinear transformations in attention and FFN layers reshape oscillatory phase signals, leading to wavelet-like, multi-resolution representations.
• Empirical analyses across LLaMA models reveal that phase differences can persist or even amplify with depth, emphasizing the interplay between encoding schemes and network dynamics.

Searching arXiv for papers on cumulative-depth RoPE and related rotary positional embedding work. Cumulative-depth Rotary Position Embeddings denotes an informal, descriptive reading of how standard Rotary Position Embeddings (RoPE) can exhibit increasingly differentiated phase-sensitive behavior across successive Transformer layers, rather than a formally defined positional encoding variant. In the literature block considered here, the closest source is "Beyond Position: the emergence of wavelet-like properties in Transformers" (Ruscio et al., 2024), which studies RoPE as a phase-rotation mechanism inside Transformer attention and FFNs and argues that deeper layers show increasingly different activation statistics for aligned versus misaligned positional phases. That paper does not introduce a new architecture called cumulative-depth RoPE, does not provide a depth-recursive RoPE formula, and does not compare a depth-accumulating rotary mechanism against standard RoPE. The term is therefore best understood as a compact label for an observed layerwise phenomenon under ordinary RoPE rather than as a standardized method.

1. Terminological status and scope

Within the cited literature, cumulative-depth RoPE is not a formal model class, theorem, or named algorithm. The relevant paper explicitly states that it is about RoPE as a phase-rotation mechanism inside Transformer attention and FFNs, not about a formally defined cumulative-depth RoPE variant (Ruscio et al., 2024). The closest notion is the repeated claim that RoPE-induced phase effects can “accumulate” across layers in the sense that deeper layers show increasingly different activation statistics for aligned versus misaligned positional phases.

This distinction is technically important. Standard RoPE rotates query and key vectors as a function of position; the paper analyzes the consequences of those rotations under repeated nonlinear processing. It does not define a new layerwise compounded rotary embedding, a cumulative-depth update rule, or a modified positional operator. A common misconception is therefore to treat cumulative-depth RoPE as if it were comparable to named positional encodings such as standard RoPE. The evidence presented instead supports a descriptive statement: phase sensitivity can persist, be reshaped, and sometimes intensify with depth.

Term or claim	Status in the cited literature
Cumulative-depth RoPE as a new architecture	Not introduced
Depth-recursive RoPE law or theorem	Not derived
Layerwise amplification of phase effects under standard RoPE	Observed descriptively
Formal comparison against a depth-accumulating RoPE baseline	Not provided

A plausible implication is that the phrase is best reserved for analyses of emergent depthwise behavior in RoPE-based Transformers, not for a standalone positional encoding design.

2. RoPE as phase rotation and relative-position encoding

In the framing used by the relevant paper, RoPE is a position-dependent rotation applied to query and key vectors, so position is encoded by rotating embedding subspaces by an angle proportional to position rather than by adding a position vector (Ruscio et al., 2024). The paper writes the rotated embedding as

$$e(p)=e(p)\cos(\theta_kp)+e^\perp(p)\sin(\theta_kp),$$

and gives the explicit per-pair form

$$q_p^{(2k)} = q^{(2k)} \cos(\theta_k p) - q^{(2k+1)} \sin(\theta_k p),$$

$$q_p^{(2k+1)} = q^{(2k)} \sin(\theta_k p) + q^{(2k+1)} \cos(\theta_k p),$$

with the same form for keys. Here $\theta_k$ is the frequency for the $k$-th $2$D subspace.

The principal structural property emphasized in the paper is that RoPE realizes relative-position encoding through rotations. Using

$Q'(p)=R(p)Q(p), \qquad K'(p)=R(p)K(p),$

the attention score is written as

$S(p,q)=\frac{Q(p)^{\top}R(q-p)K(q)}{\sqrt{d_k}}.$

This makes the score depend on $q-p$, not on absolute positions. In this view, RoPE functions as a frequency modulation or phase modulation scheme: positional information is carried by oscillations and relative phase.

That phase-based interpretation underlies the cumulative-depth reading. If positional information is represented as oscillatory structure, then subsequent attention, softmax, and FFN transformations can act on that structure as signal-processing operations rather than merely passing forward a static embedding offset.

3. Spectral limitations motivating a depthwise interpretation

The same paper argues that RoPE, despite its empirical effectiveness, has several theoretical limitations when viewed spectrally (Ruscio et al., 2024). First, similarity is modulated by cosine terms,

$$\mathbf{q}_p^\top \mathbf{k}_q \propto \cos\left(\theta_k(p-q)\right),$$

so similarity does not simply decay with distance. It oscillates, and for $$q_p^{(2k)} = q^{(2k)} \cos(\theta_k p) - q^{(2k+1)} \sin(\theta_k p),$$0, the cosine becomes negative. The paper explicitly notes that this can make an “identical” key look less similar than a random one.

Second, distance sensitivity depends strongly on $$q_p^{(2k)} = q^{(2k)} \cos(\theta_k p) - q^{(2k+1)} \sin(\theta_k p),$$1. Small $$q_p^{(2k)} = q^{(2k)} \cos(\theta_k p) - q^{(2k+1)} \sin(\theta_k p),$$2 yields slower phase evolution and longer-range retention, whereas large $$q_p^{(2k)} = q^{(2k)} \cos(\theta_k p) - q^{(2k+1)} \sin(\theta_k p),$$3 yields faster oscillation and more local behavior. The resulting distance dependence is therefore frequency-dependent and potentially brittle across scales.

Third, the paper does not claim that RoPE itself guarantees a stable multiresolution representation. Instead, it argues that richer frequency structure may be induced by the network’s nonlinearities. Fourth, it does not derive a rigorous new theory of depth-wise phase accumulation. These limitations clarify why “cumulative-depth” is not a property of the RoPE formula alone. The phenomenon, as described, emerges only after repeated processing by standard Transformer components.

This suggests that cumulative-depth behavior should be interpreted as an interaction between the rotational encoding and the model’s nonlinear dynamics, not as a direct consequence of the base positional map.

4. Mechanisms of depthwise phase transformation

The paper’s central explanatory claim is that RoPE creates frequency-modulated embeddings, after which attention and FFNs transform those oscillatory signals through selective filtering, frequency mixing, and harmonic generation (Ruscio et al., 2024). This is the basis for its wavelet-like or multi-resolution analogy.

A frequency-sum memory proxy is defined as

$$q_p^{(2k)} = q^{(2k)} \cos(\theta_k p) - q^{(2k+1)} \sin(\theta_k p),$$4

alongside the Fourier-style representation

$$q_p^{(2k)} = q^{(2k)} \cos(\theta_k p) - q^{(2k+1)} \sin(\theta_k p),$$5

These expressions formalize memory across lag as a superposition of frequency components. The paper then approximates softmax as

$$q_p^{(2k)} = q^{(2k)} \cos(\theta_k p) - q^{(2k+1)} \sin(\theta_k p),$$6

and argues that higher-order terms generate harmonics and frequency mixing.

The FFN is modeled as

$$q_p^{(2k)} = q^{(2k)} \cos(\theta_k p) - q^{(2k+1)} \sin(\theta_k p),$$7

with $$q_p^{(2k)} = q^{(2k)} \cos(\theta_k p) - q^{(2k+1)} \sin(\theta_k p),$$8, so the FFN directly receives phase-modulated inputs. The position-dependent hidden activation is given by

$$q_p^{(2k)} = q^{(2k)} \cos(\theta_k p) - q^{(2k+1)} \sin(\theta_k p),$$9

This expression is the paper’s key statement of how RoPE and FFN weights mix frequencies.

For nonlinearities, the paper uses the Fourier-series-style expansion

$$q_p^{(2k+1)} = q^{(2k)} \sin(\theta_k p) + q^{(2k+1)} \cos(\theta_k p),$$0

to argue that ReLU or GeLU can generate higher-order harmonics from RoPE-modulated signals. It also invokes the interference identity

$$q_p^{(2k+1)} = q^{(2k)} \sin(\theta_k p) + q^{(2k+1)} \cos(\theta_k p),$$1

to explain constructive and destructive interference and amplitude modulation.

Under this account, cumulative-depth behavior is the repeated reshaping of phase information by nonlinear layers. It is therefore an emergent depth-evolving frequency profile, not a formal wavelet decomposition and not a new positional encoding operator.

5. Empirical evidence across layers, models, and checkpoints

The empirical study uses autoregressive Transformer models with RoPE, specifically LLaMA 2, LLaMA 3, and LLaMA 3.1; the primary results are reported for LLaMA 3, with additional experiments in the appendix (Ruscio et al., 2024). The layerwise analyses span 32 layers for each model variant.

One experimental setup applies phase-shift simulation on real embeddings. It uses 1,000 text samples, each with 200 tokens, sourced from BookCorpus. Token embeddings are extracted before positional encoding, manually rotated, and then fed back to study attention-score sensitivity to phase shifts. A second setup constructs 250 synthetic sequences divided into aligned and misaligned cases. Aligned sequences consist of repeated instances of the same token, so identical tokens have similar pre-RoPE embeddings and differ after RoPE primarily by phase. Misaligned sequences alternate different tokens, combining different embeddings with different positions and thereby creating phase misalignment and interference.

For each layer, the paper computes mean, standard deviation, variance, kurtosis, entropy, number of peaks, KS statistic and p-value, t-statistic and t-test p-value, and PCA visualizations of FFN activations. The software and hardware stack is Google Colab Pro, NVIDIA A100 GPU, and Hugging Face Transformers.

The paper’s depthwise findings are model-specific. In Llama 2, misaligned sequences often show larger variance, kurtosis, entropy, and peak counts in deeper layers, which the authors interpret as accumulating interference and instability. In Llama 3, the distributions are often very close numerically, but PCA still separates aligned versus misaligned sequences well, suggesting more structured latent representations even when raw statistics are similar. In Llama 3.1, misaligned activations often have higher variance, kurtosis, and peak counts, and PCA separation is weaker than in Llama 3, suggesting some degradation or altered handling of positional alignment.

These observations are the strongest empirical basis for the phrase cumulative-depth RoPE. They indicate that RoPE-induced phase differences are not static and can persist or intensify with depth. The paper nevertheless treats this as an observational result rather than as evidence for a new cumulative-depth mechanism.

6. Relation to other RoPE adaptations and boundaries of the concept

A useful contrast is provided by "Geotokens and Geotransformers" (Unlu, 2024), which modifies the RoPE idea for geographical rather than sequential position. That work introduces geotokens, each linked to a specific geological location, and argues that the primary consideration is not sequence index but geographic coordinates and relation to other geotokens. It reviews standard RoPE as a rotation-based mechanism in which

$$q_p^{(2k+1)} = q^{(2k)} \sin(\theta_k p) + q^{(2k+1)} \cos(\theta_k p),$$2

with

$$q_p^{(2k+1)} = q^{(2k)} \sin(\theta_k p) + q^{(2k+1)} \cos(\theta_k p),$$3

where $$q_p^{(2k+1)} = q^{(2k)} \sin(\theta_k p) + q^{(2k+1)} \cos(\theta_k p),$$4 is a block-diagonal matrix of $$q_p^{(2k+1)} = q^{(2k)} \sin(\theta_k p) + q^{(2k+1)} \cos(\theta_k p),$$5 planar rotations and

$$q_p^{(2k+1)} = q^{(2k)} \sin(\theta_k p) + q^{(2k+1)} \cos(\theta_k p),$$6

Its contribution is a spherical adaptation in which longitude and latitude, denoted $$q_p^{(2k+1)} = q^{(2k)} \sin(\theta_k p) + q^{(2k+1)} \cos(\theta_k p),$$7 and $$q_p^{(2k+1)} = q^{(2k)} \sin(\theta_k p) + q^{(2k+1)} \cos(\theta_k p),$$8, are used directly as angular coordinates. Starting from a $$q_p^{(2k+1)} = q^{(2k)} \sin(\theta_k p) + q^{(2k+1)} \cos(\theta_k p),$$9D Euler-angle rotation matrix and setting the $\theta_k$0-axis rotation to zero, $\theta_k$1, the paper defines a spherical rotation basis

$\theta_k$2

and repeats it block-diagonally across an embedding space whose dimension is assumed divisible by three. The resulting positional encoding matrix $\theta_k$3 adapts RoPE from one-dimensional sequence offsets to spherical geographic displacement.

This contrast clarifies the boundary of cumulative-depth usage. Geotransformers change the positional geometry itself; cumulative-depth RoPE, as discussed above, refers instead to what happens when ordinary RoPE-induced phase structure is repeatedly transformed across layers. The two notions address different axes of variation: one modifies the positional map, while the other describes a layerwise effect under an unchanged map.

The geotransformer experiment uses a transformer encoder-decoder with one block and one attention head, applies the proposed spherical rotation matrix to query and key vectors, uses no position encoding in the ordinary sequence sense, tokenizes coordinate strings from a small character vocabulary, and sets the embedding dimension to 27 so that it is divisible by three. Its main finding is that the properly geo-encoded version achieves much lower training loss than a baseline using randomly fabricated latitudes and longitudes in the same scheme. A plausible implication is that the literature on RoPE extensions currently contains explicit domain adaptations of the rotational idea, whereas cumulative-depth effects remain primarily an interpretive description of standard RoPE dynamics.

7. Conceptual significance and open interpretive questions

The cumulative-depth reading of RoPE is significant because it relocates some of the explanatory burden for Transformer positional competence from the encoding formula itself to the interaction between that formula and the model’s learned nonlinear processing (Ruscio et al., 2024). In this view, the practical effectiveness of RoPE is not exhausted by the usual statement that attention depends on relative position. The deeper claim is that phase-encoded signals can be selectively filtered, mixed, and reorganized across the network into progressively more structured latent representations.

At the same time, the literature block imposes clear limits on what can be claimed. There is no explicit cumulative-depth RoPE architecture, no cumulative-depth theorem, no formal wavelet construction, and no benchmark comparison against a named depth-accumulating rotary baseline. The wavelet language remains analogical, grounded in observed multi-resolution behavior, harmonic generation, and layerwise changes in activation statistics. The safest formulation is therefore that standard RoPE can exhibit cumulative-depth effects under repeated nonlinear transformation, not that there exists an established positional encoding method called Cumulative-depth Rotary Position Embeddings.

This suggests two interpretive consequences. First, future work could formalize whether the observed layerwise amplification of phase effects admits a precise recursive theory. Second, the distinction between modifying RoPE itself and analyzing emergent dynamics under unchanged RoPE is likely to remain important, especially as domain-specific rotational encodings continue to appear in settings such as spherical geography.