Rotary Camera Encoding (RoCE)

• Rotary Camera Encoding (RoCE) is a technique that fuses geometric, spatial, temporal, and camera information via phase rotations to enable high-speed imaging and advanced neural representation.
• Optical implementations, such as compressive coded rotating mirror cameras, use calibrated mask encoding and precise motion shifts to achieve high compression ratios and recover thousands of frames in a single exposure.
• Neural adaptations of RoCE integrate spatiotemporal rotary embeddings and camera-conditioned phase adjustments, improving video understanding, 3D detection, and retake generation in transformer systems.

Rotary Camera Encoding (RoCE) refers to a class of encoding techniques that integrate geometric, spatial, temporal, and camera-condition information via phase rotation mechanisms, most often in optical compressive sensing systems or as an extension of rotary positional encoding (RoPE) in neural architectures. RoCE aims to enhance either high-speed frame capture or the ability of neural models to represent complex camera/View dynamics and time—depending on the domain of deployment. Initial implementations emerged in compressive imaging hardware, while recent developments focus on multi-view, spatiotemporal, and camera-aware transformer models for video understanding and generation. Multiple independent lines of research converge on these principles, embedding rotation-based codes into the measurement process or deep model attention layers for improved information disentanglement and recovery.

1. Optical Realization: Compressive High-Speed Imaging

The original instantiation of Rotary Camera Encoding appears in the compressive coded rotating mirror (CCRM) camera, where RoCE enables high-frame-rate passive capture without sacrificing spatial resolution or incurring prohibitive cost (Matin et al., 2020). The CCRM employs a static amplitude encoding mask, a motorized rotating mirror configured for precise single-pixel shift per frame, and a low-cost CMOS detector to generate a highly compressed measurement: $y = \Phi x + \eta, \quad \Phi = TCA,$ where $x$ is the vectorized stack of $F$ unknown frames (each $M \times N$), $T$ enforces pixel shifts, $C$ calibrates (per-frame) motion-induced jitter, $A$ applies amplitude (mask) encoding, and $\eta \sim \mathcal{N}(0,\sigma^2 I)$ represents additive measurement noise.

Key parameters:

• Mask: $A$ is block-diagonal across $F$ frames, each block encoding a 1:1 transparent/blocked pixel ratio; calibration blocks correct for mask non-idealities.
• Motion calibration: $C$ encodes per-frame vertical misalignments, extracted automatically via printed pixel blocks.
• Temporal encoding: $T$ produces exact one-pixel vertical shift per frame, ensuring overlapping summation on the detector.

Native optical compression ratio is: $CR = \frac{NF}{N + F - 1},$ reaching up to $368:1$ (e.g., $F=1400$, $N\approx 200-500$ in practice).

The rotating mirror’s sweep velocity ($v = \omega L$) is physically synchronized with detector pixel pitch ($p$) to achieve: $\Delta x = p \implies F_r = \frac{\omega L}{p}.$ Consequently, the effective frame rate is determined exclusively by mirror angular velocity $\omega$, mirror-detector distance $L$, and sensor pitch $p$, being independent of detector readout speed.

2. Reconstruction Algorithms and Trade-offs

Because the measurement operator $\Phi = TCA$ does not in general satisfy the Restricted Isometry Property (RIP), the recovery of unknown frame stacks $x$ from $y$ is performed via a regularized inverse problem, specifically with Total Variation (TV) regularization and ADMM optimization: $$\hat{x} = \arg\min_x \frac{1}{2}\|y - TCA x\|_2^2 + \lambda \operatorname{TV}(x).$$ TV priors are enforced along spatial and temporal axes, with weights empirically tuned. The algorithm alternates between $\ell_2$ residual minimization and fast TV-denoising steps. This structure enables successful reconstruction even at extreme optical compression, with per-channel RGB video recovery handled independently.

Key trade-offs:

• Spatial resolution is set by amplitude mask print quality and sensor pitch.
• Temporal resolution is limited by mirror angular stability; higher angular velocities increase frame rate but also risk introducing aberrations or field curvature.
• Sequence depth $F$ cannot exceed the difference in detector column count and frame width plus one: $F \leq (\#\text{columns} - N) + 1$.
• At high $F$ (higher compression), the inverse problem becomes harder and signal quality degrades due to increased smoothing and SNR loss.

Experimental benchmarks show successful recovery of up to $F=1400$ frames in a single exposure at 120 kfps; extrapolation suggests up to 20 Gfps is physically feasible with improved mechanics.

3. Extensions to Deep Learning: Spatiotemporal Rotary Embedding

Contemporary RoCE appears as a generalized rotary embedding equipped to integrate spatiotemporal and camera parameters in transformer architectures, particularly for tasks requiring joint modeling of space, time, and motion—such as 3D detection or video retake generation (Ji et al., 17 Apr 2025, Park et al., 25 Nov 2025).

A representative formulation:

• Let $(x, y)$ denote normalized BEV-plane coordinates, and $t$ the normalized timestamp;
• Frequency vectors $\omega^{(x)}, \omega^{(y)}, \omega^{(t)}$ encode per-axis periodicities;
• Rotation angles are composed additively: $$\theta_k = x\,\omega^{(x)}_k + y\,\omega^{(y)}_k + t\,\omega^{(t)}_k,$$ applied pairwise to embedding channels to rotate $(Q, K)$ vectors within each attention head.

In StreamPETR and RoPETR, RoCE effectively replaces absolute/additive position embeddings: positional geometry and temporal cues are encoded directly into the pairwise attention mechanism via rotation, allowing the model to align tokens according to their spatiotemporal context (Ji et al., 17 Apr 2025). In video-to-video retake architectures, RoCE is further enhanced to include a camera-conditioned phase shift—integrating per-frame extrinsics and intrinsics into the attention phase, thereby enabling the model to distinguish otherwise-ambiguous spatial/temporal positions across different camera views (Park et al., 25 Nov 2025).

4. Camera-Conditioned Phase: Theory and Implementation

The camera-conditioned RoCE introduces a learnable phase shift $\Delta(n, c)$ per token, computed from camera pose embeddings (e.g., Plücker-ray features), and combined multiplicatively (as a complex exponential) with standard RoPE phases: $\widetilde{R}(n, c) = e^{i\,\Delta(n, c)},$ leading to the composite attention operation,

$$A_{\mathrm{RoCE}}(n, m) = \Re \left[ \bar{\mathbf{q}}_n \overline{\bar{\mathbf{k}}_m} \exp (i [\omega_c(n - m) + \Delta(n,c) - \Delta(m,c)]) \right].$$

Specialization: Temporal channels’ phase shifts are zeroed to avoid disturbing temporal alignment, constraining geometry-awareness to spatial features. Attention to the value vector is also modulated by (inverse/forward) camera phases.

Implementation details:

• MLPs ($\operatorname{MLP}_{\text{cam}}$) map camera embeddings to per-token phase shifts, initialized to output zero at training start.
• All transformer self-attention layers are replaced by RoPE+RoCE-equipped variants, using phase shifts computed per token and channel.

5. Empirical Performance and Evaluation

In compressive imaging, RoCE (as realized in CCRM) achieves:

• Up to 1400 frames recovered in a single exposure at 120 kfps, with potential for 20 Gfps operation.
• Optical compression ratios up to 368:1.
• Approximately 700× speedup over conventional mechanical rotating-mirror devices for comparable spatial resolution (Matin et al., 2020).

In video transformer applications, RoCE:

• Substantially reduces 3D object velocity error in camera-only 3D object detection, cutting mean Average Velocity Error (mAVE) by ~30% compared to baseline and increasing NuScenes Detection Score (NDS) by 1.4–3.3 points (Ji et al., 17 Apr 2025).
• In video retake generation, achieves SOTA geometric consistency and camera controllability; e.g., TransErr 0.0165 (vs 0.0292 for strong pixel-aligned baselines) and consistent improvement on DAVIS video metrics (Park et al., 25 Nov 2025).
• Supports arbitrary test-time video lengths, with no need for new embedding tables, and generalizes to unseen camera trajectories.

6. Comparative Methods and Limitations

RoCE offers distinct advantages over pixel-aligned encodings (which require global world frames and risk overparameterization) and frame-level encodings (which lack integration with pretrained models or require retraining from scratch on static scenes). RoCE’s compatibility with standard RoPE machinery and its implementation as minor phase shifts positions it as a minimally invasive, but geometrically expressive, alternative.

Limitations and trade-offs depend on domain:

• Optical RoCE: Physical limits (mask accuracy, mirror jitter, frame count bounded by detector size); more aggressive compression impairs recovery fidelity.
• Neural RoCE: Camera-conditioned attention relies on encoder quality (e.g., Plücker-ray extractor must be robust); efficacy of geometry encoding depends on MLP capacity and dataset diversity.

7. Future Directions and Improvement Potential

Advances for rotary camera encoding include:

• Optics: Adoption of air-bearing/magnetic-levitation mirrors for increased rotational stability and speed; improved multi-level amplitude/phase masks to approach RIP conditions.
• Algorithms: Integration of deep denoisers or learned priors in recovery optimization; 3D and spectral extension via multi-view mirrors or light-field setups (Matin et al., 2020).
• Neural models: Broader adoption of camera-conditioned RoCE in video diffusion and transformer pipelines, incorporation of geometry-aware attention on values, and scaling to higher resolution and trajectory complexity (Park et al., 25 Nov 2025).
• Hardware: Use of miniaturized optics, smaller-pitch CMOS sensors, or multi-wavelength detectors for further increases in sequence and frame rate.

By unifying rotary encoding principles in both physical and deep learning systems, RoCE provides a scalable route to high-dimensional, geometry-aware inference in both capture hardware and neural architectures.