Trajectory-to-Camera Formulation

• Trajectory-to-camera formulation is a method that maps 3D motion trajectories to camera parameters, ensuring physically consistent control and view synthesis.
• It employs mathematical models such as SE(3) transformations and Lie groups along with optimization techniques to achieve smooth pose interpolation and calibration.
• This approach underpins applications in robotics, visual servoing, novel-view video synthesis, and generative models, facilitating improved scene understanding and control.

A trajectory-to-camera formulation refers to the explicit modeling, conversion, or coupling between a 3D trajectory (typically a sequence of positions and/or motions in space) and the parameters or signals controlling a camera system—most often its extrinsic pose (rotation, translation), and frequently, additional camera controls such as intrinsics, zoom, or field-of-view. Such formulations are foundational in diverse areas including novel-view video synthesis, vision-based control, visual servoing, coverage planning, structure-from-motion, and blind deblurring. The technical landscape is characterized by a variety of mathematical mappings, optimization objectives, and training strategies that enable trajectory-informed camera modeling, rendering, or estimation under different task constraints and data modalities.

1. Foundations and Mathematical Mappings

Trajectory-to-camera formulations often start by parameterizing a reference or user-prescribed motion as a series of waypoints, control points, or continuous curves in Euclidean or Lie group manifolds. These trajectories are then mapped to camera extrinsics, typically as SE(3) transformations (rotation matrices $R_t$ and translation vectors $c_t$ per timestamp $t$). In many works, the focus is on capturing the relative transformations between trajectories, or expressing trajectory reparameterization in a differentiable and geometrically consistent way.

For example, in novel-trajectory video synthesis, given reference and target trajectories $T^{(r)} = \{t_i^{(r)}\}$ and $T^{(t)} = \{t_i^{(t)}\}$, the formulation proceeds by computing per-frame camera pose pairs $P_i^{(r)}$, $P_i^{(t)}$ and expressing the relative motion as

$$\Delta R_i = R_i^{(t)} R_i^{(r)\,T}, \qquad \Delta c_i = c_i^{(t)} - \Delta R_i c_i^{(r)}$$

with the set $\{\Delta R_i, \Delta c_i\}_{i=1}^N$ forming the conditioning signal for subsequent modules (Li et al., 3 Dec 2025). In continuous-time and optimization-based settings, cubic splines, radial-basis functions, or direct parameterizations on Lie groups ($\mathbb{SE}(3)$, $\mathbb{SO}(3)$, $\mathbb{R}^3$) are used to yield smooth and differentiable pose curves (Ovrén et al., 2018, Liu et al., 2024).

2. Conditioning and Control in Generative Frameworks

The translation of trajectory signals into actionable camera control is critical in view-synthesis and video generation models. Approaches differ in how the conditioning is encoded and fused with generative backbones. In latent diffusion transformers, this entails embedding trajectory-derived pose differentials (and in advanced settings, rendering-based features) via dedicated encoders and cross-attention mechanisms at multiple layers and training stages (Li et al., 3 Dec 2025, YU et al., 7 Mar 2025). For InfCam (Kim et al., 18 Dec 2025), the infinite homography $H_\infty = K_t R K_s^{-1}$ is applied directly to spatial video latents, separating the exact rotational warp from residual translation/parallax, with network modules trained to predict the latter as a learned offset.

Other models, such as MotionFlow, map the trajectory directly into pixelwise motion maps using Plücker embeddings, enabling explicit per-pixel supervision of how camera motion is intended to affect the image sequence, which is then fused throughout the video synthesis pipeline (Lei et al., 25 Sep 2025).

3. Inverse Trajectory-to-Pose Inference

In some domains, the task is to infer the camera pose from observed or hypothesized trajectories. One clear example is estimating camera calibration from the 2D image-plane tracks of objects with assumed or constrained dynamics. By training neural sequence models that map pedestrian or object trajectories $T \in \mathbb{R}^{2 \times N}$ to pose parameters $P = (t, q)$ (translation, unit quaternion) in $SE(3)$, the system estimates global height and orientation of static cameras without explicit 3D-to-2D correspondences (Xu et al., 2019). Under multiple unsynchronized cameras, joint bundle adjustment can recover both trajectory coefficients and unknown camera timing/rotational parameters by enforcing cross-time triangulation on polynomial trajectory models (Huang et al., 31 May 2025).

4. Trajectory-Based Optimization in Robotics and Control

In coverage planning and visual servoing, trajectory-to-camera formulations underpin the real-time joint optimization of robot/UAV state and camera controls. The camera’s movement is coupled to physical dynamics and environment visibility criteria, expressed through mixed-integer programming or continuous nonlinear optimization. Constraints include vehicle kinematics, collision avoidance, camera field-of-view geometry, and precise ray-tracing of visible surface elements over predicted trajectories (Papaioannou et al., 8 Apr 2025, Tang et al., 2020). Visual servoing further utilizes homography decomposition and minimal-geodesic interpolation in $SO(3)$ for end-effector trajectories, yielding trajectory-informed image-space references for closed-loop control (Fu et al., 2023).

5. Supervision, Data Curation, and Training

Modern approaches require large-scale parallel or aligned trajectory–video data, often unavailable in real-world driving or robotics scenarios. Cross-trajectory curation pipelines, such as ParaDrive (Li et al., 3 Dec 2025), use scene reconstruction (e.g., 3D Gaussian Splatting or NeRF) to simulate novel, laterally shifted camera trajectories and render synthetic target videos. The curation process yields diverse, densely aligned trajectory pairs enabling effective end-to-end training of camera-controlled generative models. Data augmentation along axes such as trajectory diversity, viewing direction, and focal length is critical for overcoming distribution gaps and ensuring robust generalization (Kim et al., 18 Dec 2025).

6. Geometric and Consistency Losses

Losses in trajectory-to-camera formulations fall into several categories: (i) deterministic flow matching losses in diffusion models, where targets are latent “velocity fields” under camera conditioning (Li et al., 3 Dec 2025); (ii) reprojection and cycle-consistency losses enforcing geometric integrity under simulated trajectory changes (YU et al., 7 Mar 2025); (iii) photometric or blur re-creation losses tied to differentiable forward models that blur or deblur images based on sampled or inferred camera motion (Carbajal et al., 23 Oct 2025); and (iv) bidirectional consistency objectives in 3D tracking and map prediction, where priors on smoothness, pose consistency, and dynamic masking prevent degenerate solutions (Miao et al., 4 Feb 2026). Optimization objectives in planning and robotics settings incorporate coverage, visibility, aesthetics, and path-efficiency, using differentiable or combinatorial solvers (Liu et al., 2024, Wang et al., 13 May 2025, Tang et al., 2020).

7. Impact and Applications

The trajectory-to-camera paradigm—encompassing both trajectory-informed camera control and pose estimation from observed movement—serves as a unifying concept across generative vision, robotics, cinematography, and geometric perception. In camera-controlled video generation models, it enables explicit, physically grounded manipulation of viewpoint and trajectory distinct from scene content, achieving higher pose fidelity and geometric consistency than previous depth- or flow-based methods (Li et al., 3 Dec 2025, Kim et al., 18 Dec 2025). In robotics and vision-based control, it facilitates joint optimization of agent and camera states for efficient data acquisition, immersive cinematography, and precise control (Papaioannou et al., 8 Apr 2025, Wu et al., 2023). Its broad adoption as a principled coupling between geometry, control, and perception continues to shape advances in both foundational machine perception and downstream applications.