--- title: Learnable Angle Matrices (ComRoPE) url: https://www.emergentmind.com/topics/learnable-angle-matrices-comrope type: topic --- # Learnable Angle Matrices (ComRoPE) Learnable Angle Matrices, or Commuting Rounds of Parallelized Elementary rotations (ComRoPE), generalize rotary positional encodings (RoPE) by parameterizing position-dependent rotations using trainable angle (skew-symmetric) matrices with strict commutativity requirements. ComRoPE is designed to address the limitations of fixed rotary mechanisms in transformer architectures, guaranteeing robustness to position offsets, scalability to high-resolution domains, and adaptability across one- and multi-dimensional structured inputs. The theoretical foundation relies on the algebra of commuting skew-symmetric generators that define families of orthogonal transformations satisfying essential properties of relative positional encoding, and supports efficient implementation via block-structured Givens rotations or round-robin decompositions. Empirical results demonstrate ComRoPE's superiority on tasks including large-scale image classification and object detection, with strict gains in both in-distribution and out-of-distribution settings [2506.03737]. ## 1. Theoretical Foundations: Rotary Positional Embedding and the RoPE Equation Rotary Positional Embedding (RoPE) integrates positional information into the attention mechanism through block-diagonal rotation matrices acting on embedding vectors. Standard RoPE operates by partitioning the embedding dimension $d$ into $d/2$ independent $2\times2$ blocks, each corresponding to a planar rotation with manually designed angle schedules: \[ R(p) = \operatorname{diag}\bigl(R_1(p), R_2(p), \dots, R_{d/2}(p)\bigr) \in SO(d) \] with \[ R_j(p) = \begin{pmatrix} \cos\theta_j(p) & -\sin\theta_j(p) \\ \sin\theta_j(p) & \cos\theta_j(p) \end{pmatrix}, \quad \theta_j(p) = \frac{p}{10000^{2j/d}} \] The attention calculation involves rotated queries and keys $q' = R(p_q)q$, $k' = R(p_k)k$, yielding \[ q'^\top k' = q^\top R(p_q)^\top R(p_k) k. \] A central constraint for robust, relative positional encoding is the RoPE equation: \[ R(p_q)^\top R(p_k) = R(p_k - p_q) \quad \text{for all positions} \] which guarantees attention only depends on relative offset, thereby ensuring shift-invariance and scalability to arbitrary input lengths or resolutions. ComRoPE generalizes $R(p)$ to \[ R(x;\mathcal{A}) = \exp\left(\sum_{i=1}^N A_i x_i\right) \] where $\mathcal{A} = \{A_1,\dots,A_N\}$ are $N$ real skew-symmetric matrices in $\mathbb{R}^{d\times d}$, and $x\in\mathbb{R}^N$ encodes position along each axis. The necessity and sufficiency theorem (Theorem 3.1, [2506.03737]) asserts that $R(x;\mathcal{A})$ satisfies the RoPE equation for arbitrary offsets if and only if all $A_i$ commute: $[A_i, A_j]=0$ for all $i, j$. This property enables consistent relative position encoding for both 1D and multidimensional data. ## 2. Constructing and Parameterizing Learnable Commuting Angle Matrices ComRoPE provides explicit parameterizations that guarantee commutativity, leading to two principal variants: ### Axial-Partition (AP) Parameterization Partition $d$ into $m$ blocks of size $b$, $d = m b$. For each axis $i\in\{1,\dots,N\}$ and block $j\in\{1,\dots,m\}$, \[ B_{ij} = \begin{cases} P_j - P_j^\top, & j\equiv i\ (\mathrm{mod}\ N) \\ 0_{b\times b}, & \text{otherwise} \end{cases} \] where $P_j$ is an unconstrained trainable $b\times b$ matrix. Then, \[ A_i = \operatorname{diag}(B_{i1}, B_{i2},\dots, B_{im}). \] Only one axis contributes a non-zero skew block per partition, enforcing $[A_i,A_j]=0$. ### Linearly Dependent (LD) Parameterization Produce a single base skew-symmetric matrix $S = P - P^\top$ and per-axis scalars $\theta_i$. Form \[ B_{i} = \theta_i S, \quad A_i = \operatorname{diag}(B_i, \ldots, B_i) \] with $d/b$ repeats per axis. Since all $A_i$ are scalar multiples of a common block, they trivially commute. Both parameterizations scale efficiently, use learnable blocks of small matrix size, and result in $O(d b)$ or $O(d (b+N/b))$ free parameters. ## 3. Efficient Implementation: Givens Rotations, Round-Robin Scheduling, and GPU Utilization ComRoPE extends the classical FFT-like or round-robin decompositions of orthogonal matrices using angle-parameterized Givens rotations [1404.7195, 2106.00003]. For head dimension $n=2^L$, a rotation matrix $Q$ is represented as \[ Q \approx Q^{(L)} Q^{(L-1)} \cdots Q^{(1)} = \prod_{s=1}^{L} \prod_{p=1}^{n/2} R_{i_{s,p},j_{s,p}}(\theta_{s,p}) \] Here, each $Q^{(s)}$ is a block-sparse matrix applying independent planar rotations (Givens) on disjoint pairs. The index-pair schedule is engineered in a "butterfly" or "round-robin" pattern for maximal parallelism; with $n/2$ rotations per layer and $L=\log_2 n$ layers, the approach yields $O(n\log n)$ total operations for forward and backward passes [1404.7195]. In the round-robin method [2106.00003], all $n(n-1)/2$ Givens rotations are organized into $n-1$ blocks of $n/2$ non-overlapping pairs, admitting $O(n)$ sequential depth in forward computation and $O(n\log n)$ for backpropagation, ideally suited to GPU architectures. Common implementation steps: - Store and update only the angle parameters and small per-block intermediates. - Forward and backward passes update activations layer/block by layer/block without forming full dense matrices. - After each update, re-project $2\times2$ blocks to $SO(2)$ to ensure orthogonality. - Runtime overhead is negligible ($\approx 2n\log n$ FLOPs per vector for $n=64$). ## 4. Integration into Transformers and Attention Mechanisms Within transformer-based models, ComRoPE replaces fixed RoPE with dynamically learned, commuting angle-based rotations. For each token (or patch) and each axis, positional coordinates modulate the associated angle matrices: \[ \text{For each position } p,~ M_p = \sum_{i=1}^N P_{p,i} A_i,~ R_p = \exp(M_p) \] The query/key embeddings at position $p$ are updated to $Q_p^\prime = R_p Q_p$, $K_p^\prime = R_p K_p$. Standard attention proceeds using $Q^\prime$, $K^\prime$: \[ \text{Attn}(Q_p, K_{p'}, V_{p'}) = \operatorname{softmax}\bigl( (Q_p^\prime) (K_{p'}^\prime)^\top \bigr) V_{p'} \] All rotation parameters are shared across the batch and sequence dimensions, with gradients accumulated during backpropagation. Practical instantiations initialize $\theta_{s,p}$ using FFT "twiddle" factors, RoPE sinusoids, or uniform randomization. ## 5. Empirical Performance and Robustness ComRoPE's effectiveness is most evident in settings where positional robustness and extrapolation to out-of-distribution resolutions or input lengths are critical. ### Classification and Detection On ImageNet-1K using ViT-B/16, at $224 \times 224$ input, ComRoPE-LD yields 65.49% top-1 accuracy (+2.4% absolute over LieRE), and at $512 \times 512$ extrapolated resolution, 55.29% (+2.9% absolute) [2506.03737]. Object detection experiments (MS COCO, ViT-S backbone) yield ComRoPE-LD at 44.7 AP, slightly outperforming LieRE at 44.5 AP, using approximately half the additional parameters. ### Ablation and Stress Testing - ComRoPE variants display invariance to coordinate shifts, while non-commuting formulations (e.g., LieRE) degrade under uniform coordinate perturbations. - Optimal block size $b$ balances performance and computational cost, peaking near $b=8$. - Robustness to training-time position perturbation is intrinsic in ComRoPE; gains from artificial perturbation are minimal compared to vulnerable baselines (APE +19.5% vs. ComRoPE-LD +2.9%). ## 6. Generalization Properties and Relation to Prior Work ComRoPE formally subsumes both absolute and rotary positional encoding as special cases: - If all $A_i=0$, the mechanism reduces to absolute position encoding. - With $b=2$ and fixed $2\times2$ skew-symmetric blocks, standard RoPE is recovered. Thus, ComRoPE constitutes a strict superset of previous rotary-based encodings [2506.03737]. Expressivity is governed by parameter count and block size; larger blocks allow richer, higher-dimensional transformations at increased computational overhead. ## 7. Future Directions and Open Problems Key avenues for future research highlighted include: - Developing optimized routines for small-matrix exponentiation (exact closed-form, lookup tables) to minimize computational cost for block sizes $b \leq 8$. - Investigating weaker commutativity constraints that may permit a wider space of trainable angle matrices. - Extending the approach to more general structured data (multi-dimensional grids, point clouds, videos) and scaling to large language models where efficient exponentiation is paramount. A plausible implication is that such directions may enhance generalization, facilitate efficient fine-tuning for new modalities, and support architectures requiring relative encoding over long spatial-temporal contexts. --- **Summary Table: ComRoPE Parameterizations** | Variant | Structure | Parameters (per block) | |--------------|----------------------|--------------------------| | AP | One axis per block | $d \cdot b$ | | LD | Shared skew matrix | $b^2 + N$ | ComRoPE establishes a mathematically principled, scalable, and empirically robust foundation for learnable rotary embeddings in transformer models, unifying previous positional encoding strategies and advancing state-of-the-art accuracy and generalization [2506.03737, 2106.00003, 1404.7195]. Source: https://www.emergentmind.com/topics/learnable-angle-matrices-comrope