ActCam: Zero-Shot Video Generation with Camera Control

• ActCam is a zero-shot video generation method that jointly controls actor motion and camera trajectory through geometrically consistent pose and depth conditioning.
• It employs a two-phase conditioning schedule where early diffusion uses sparse depth to enforce scene structure and later refinement applies pose-only guidance for detail.
• Empirical results show ActCam's superior performance in moving-camera scenarios, reducing motion errors and enhancing visual coherence compared to prior methods.

ActCam is a zero-shot method for video generation that jointly transfers character motion from a driving video into a new scene and enables per-frame control of intrinsic and extrinsic camera parameters. It is designed for the setting in which a reference image specifies target identity and scene appearance, an acting video supplies the desired performance, and a target camera trajectory specifies the intended cinematography. The method builds on a pretrained image-to-video diffusion model that accepts conditioning in terms of scene depth and character pose, and it addresses the central difficulty that actor motion and camera motion are entangled in image space. Its defining contribution is to construct pose and depth conditions that remain geometrically consistent across frames, then use a single sampling process with a two-phase conditioning schedule: early denoising uses pose and sparse depth to enforce scene structure, and later denoising drops depth and keeps pose-only guidance for detail refinement (Khalifi et al., 7 May 2026).

1. Problem formulation and design objective

ActCam targets joint control of performance and cinematography. The inputs are a reference image $I_{\mathrm{ref}}$, an acting video $$V_{\mathrm{act}} = \{ I^{\mathrm{act}}_\tau \}_{\tau=1}^{T}$$, and a target camera trajectory

$$\mathcal{C} = \{(K_\tau, R_\tau, \mathbf{t}_\tau)\}_{\tau=1}^{T},$$

where $K_\tau \in \mathbb{R}^{3 \times 3}$ is the intrinsic matrix, $R_\tau \in SO(3)$ is the camera rotation, and $\mathbf{t}_\tau \in \mathbb{R}^3$ is the camera translation. The output is a generated video $V = \{ I_\tau \}_{\tau=1}^{T}$ whose subject identity and overall appearance match the reference image, whose motion matches the acting video, and whose viewpoint follows the prescribed camera path.

The method is motivated by three limitations identified in prior practice. Pose-only control is not camera-aware, because $2$D pose signals are view-locked and ambiguous under strong viewpoint change. Camera-only or geometry-only control does not transfer articulated performance reliably. Existing joint methods such as Uni3C require task-specific training or finetuning, whereas ActCam is framed as zero-shot and training-free at inference time. A plausible implication is that ActCam is less a new generative backbone than a conditioning and control layer for compatible pretrained video generators.

2. Conditioning interface and backbone assumptions

ActCam assumes a pretrained image-to-video diffusion model with dense conditioning interfaces for scene depth and character pose. In experiments, the backbone is VACE, and for comparison with Uni3C in moving-camera experiments both methods are based on the same Wan 2.1 14B backbone. The backbone is formalized as a mapping from conditioning to video, and its latent evolution is written as

$$\frac{\mathrm{d}z_t}{\mathrm{d}t} = v_\theta(z_t, t, C),$$

with conditioning tuple

$$C = (I_{\mathrm{ref}}, c_{\mathrm{pose+depth}}, c_{\mathrm{pose}}).$$

Two target-view control videos are synthesized. The first is a pose-only control video,

$$V_{\mathrm{act}} = \{ I^{\mathrm{act}}_\tau \}_{\tau=1}^{T}$$0

constructed from a standard OpenPose-style representation encoding $$V_{\mathrm{act}} = \{ I^{\mathrm{act}}_\tau \}_{\tau=1}^{T}$$1D joint locations and limb connectivity, rendered on a black background from the target camera viewpoint. The second is a depth+pose control video,

$$V_{\mathrm{act}} = \{ I^{\mathrm{act}}_\tau \}_{\tau=1}^{T}$$2

formed by rendering background depth under the target camera and superimposing pose.

Several preprocessing components are required. GroundingDINO-based segmentation removes the reference character, inpainting fills the exposed background, MoGe estimates background-only depth, and GVHMR recovers $$V_{\mathrm{act}} = \{ I^{\mathrm{act}}_\tau \}_{\tau=1}^{T}$$3D human motion from the acting video. The paper emphasizes that mesh-based rendering is used because it provides better geometric consistency than point clouds, which can appear sparse or artifact-prone under camera motion. This design indicates that ActCam treats camera control as a problem of generating camera-aligned dense conditions rather than camera tokens or explicit trajectory embeddings.

3. Geometric alignment, scene transfer, and motion fitting

A central obstacle is that naive depth estimation on the reference image includes the static reference person. If that person remains in the depth map while pose conditioning tries to animate a moving person, the controls conflict and can produce duplicated or frozen characters. ActCam therefore estimates a background-only depth map $$V_{\mathrm{act}} = \{ I^{\mathrm{act}}_\tau \}_{\tau=1}^{T}$$4 after character removal and inpainting.

Because the original reference depth and the inpainted background depth are estimated separately, the method performs a scene-transfer step to place the character consistently into the background geometry. Let $$V_{\mathrm{act}} = \{ I^{\mathrm{act}}_\tau \}_{\tau=1}^{T}$$5 be the binary character mask. For pixels outside the mask, an importance weight is defined as

$$V_{\mathrm{act}} = \{ I^{\mathrm{act}}_\tau \}_{\tau=1}^{T}$$6

so that points near the character boundary receive larger weight. Weighted centroids are then computed in the reference and background depth spaces, and character depth is aligned by an affine transform along the depth axis. The stated goal is to compensate for scale and translation mismatch while preserving correct local scene interaction and contact relationships. This suggests that the method is explicitly concerned with occlusion ordering and layout preservation under viewpoint change.

Recovered $$V_{\mathrm{act}} = \{ I^{\mathrm{act}}_\tau \}_{\tau=1}^{T}$$7D motion from the acting video is then aligned to the target scene and reference character. Following Uni3C, a rigid alignment at $$V_{\mathrm{act}} = \{ I^{\mathrm{act}}_\tau \}_{\tau=1}^{T}$$8 is estimated with Umeyama least-squares fitting, producing a rotation $$V_{\mathrm{act}} = \{ I^{\mathrm{act}}_\tau \}_{\tau=1}^{T}$$9, a translation $$\mathcal{C} = \{(K_\tau, R_\tau, \mathbf{t}_\tau)\}_{\tau=1}^{T},$$0, and a scale $$\mathcal{C} = \{(K_\tau, R_\tau, \mathbf{t}_\tau)\}_{\tau=1}^{T},$$1. These are applied to the entire motion sequence: $$\mathcal{C} = \{(K_\tau, R_\tau, \mathbf{t}_\tau)\}_{\tau=1}^{T},$$2 The transformed sequence $$\mathcal{C} = \{(K_\tau, R_\tau, \mathbf{t}_\tau)\}_{\tau=1}^{T},$$3 is the motion actually rendered into the target-view conditioning videos. In effect, ActCam separates the problems of recovering articulated motion, placing that motion coherently in a new scene, and rendering it under a prescribed camera.

4. Two-phase conditioning schedule

The most distinctive inference mechanism in ActCam is its staged guidance schedule. The method runs a single diffusion sampling process, but the conditioning changes over denoising time. Early denoising uses depth+pose to enforce scene structure and camera-consistent geometry; later denoising drops depth and keeps pose-only guidance to refine high-frequency details without over-constraining the generation.

The schedule is written as

$$\mathcal{C} = \{(K_\tau, R_\tau, \mathbf{t}_\tau)\}_{\tau=1}^{T},$$4

The rationale is asymmetrical. Early denoising determines global structure, so sparse or coarse depth is sufficient to communicate viewpoint change, parallax, and static scene geometry. Later denoising determines texture, fine detail, and dynamic interaction; if coarse monocular depth remains active at that stage, it can rigidify the scene and propagate depth errors into visible artifacts.

The ablation chooses a canonical $$\mathcal{C} = \{(K_\tau, R_\tau, \mathbf{t}_\tau)\}_{\tau=1}^{T},$$5, and the paper gives the example that if the total denoising steps are $$\mathcal{C} = \{(K_\tau, R_\tau, \mathbf{t}_\tau)\}_{\tau=1}^{T},$$6, then $$\mathcal{C} = \{(K_\tau, R_\tau, \mathbf{t}_\tau)\}_{\tau=1}^{T},$$7. No depth guidance ($$\mathcal{C} = \{(K_\tau, R_\tau, \mathbf{t}_\tau)\}_{\tau=1}^{T},$$8) leaves camera and body motion ambiguous, often yielding a moving person against an almost fixed background. Full-depth guidance through all denoising steps makes the scene too rigid. The selected intermediate schedule is therefore a compromise between camera adherence and late-stage visual flexibility.

5. Empirical evaluation

ActCam is evaluated in both moving-camera and static-camera settings. The moving-camera benchmark is inspired by Uni3C, uses $$\mathcal{C} = \{(K_\tau, R_\tau, \mathbf{t}_\tau)\}_{\tau=1}^{T},$$9 camera presets with common cinematic motions, and evaluates $K_\tau \in \mathbb{R}^{3 \times 3}$0 reference clips from RealisDance-Val per preset. Reported metrics include VBench components—Subject Consistency (SC), Background Consistency (BC), Appearance Fidelity (AF), Imaging Quality (IQ), Temporal Consistency (TC), and Motion Smoothness (MS)—as well as MPJPE, Sampson Error (SE), and WorldScore metrics $K_\tau \in \mathbb{R}^{3 \times 3}$1D-C and OC (Khalifi et al., 7 May 2026).

Model	VBench Avg ↑	MPJPE ↓	SE ↓	3D-C ↑	OC ↑
Uni3C	0.8370	0.2121	0.5665	0.5390	0.9878
ActCam	0.8497	0.2087	0.4546	0.6304	0.9953

The moving-camera comparison shows that ActCam improves overall VBench average, MPJPE, SE, $K_\tau \in \mathbb{R}^{3 \times 3}$2D-C, and OC relative to Uni3C. The reduction in SE from $K_\tau \in \mathbb{R}^{3 \times 3}$3 to $K_\tau \in \mathbb{R}^{3 \times 3}$4 is the paper’s clearest numerical indication of improved camera and geometric consistency. The same table also reports higher SC, AF, IQ, and MS for ActCam, with slightly lower BC and TC.

In the static-camera setting on RealisDance-Val, ActCam also leads the listed baselines on all reported VBench submetrics. Its reported averages are $K_\tau \in \mathbb{R}^{3 \times 3}$5 for overall score, $K_\tau \in \mathbb{R}^{3 \times 3}$6 for SC, $K_\tau \in \mathbb{R}^{3 \times 3}$7 for BC, $K_\tau \in \mathbb{R}^{3 \times 3}$8 for AF, $K_\tau \in \mathbb{R}^{3 \times 3}$9 for IQ, $R_\tau \in SO(3)$0 for TC, and $R_\tau \in SO(3)$1 for MS. The paper interprets this as evidence that the $R_\tau \in SO(3)$2D-motion-based conditioning pipeline improves over $R_\tau \in SO(3)$3D-pose-based animation even when the camera is fixed.

A $R_\tau \in SO(3)$4AFC user study with $R_\tau \in SO(3)$5 users compares ActCam to Uni3C on camera adherence, motion faithfulness, and overall visual quality. The paper states that ActCam is strongly preferred on all three axes, especially under large viewpoint changes.

6. Context, limitations, and significance

ActCam occupies a specific position within controllable visual generation. It is not a language-to-camera system such as ChatCam, which generates trajectories in radiance-field scenes from conversational instructions (Liu et al., 2024), nor is it a test-time viewpoint adaptation layer for robot policies such as AnyCamVLA (Heo et al., 6 Mar 2026). Its domain is image-conditioned video generation, and its distinctive claim is that strong joint camera and motion control can be achieved without task-specific training by constructing camera-aligned conditioning and using staged guidance.

The paper also makes its limitations explicit or strongly implied. Monocular depth is coarse or locally inaccurate, which is why full-time depth conditioning is harmful. The method depends on GVHMR for $R_\tau \in SO(3)$6D motion recovery, so pose estimation errors propagate into control. Incorrect actor/background alignment can lead to tearing or implausible occlusions. Extreme camera motion and complex scenes stress the monocular depth proxy and mesh rendering assumptions. Static or dynamic scene conflicts remain possible if depth conditioning encodes objects that should move independently. More fundamentally, the environment geometry is approximated from a single reference image, so unseen content must still be hallucinated by the backbone.

Within those constraints, ActCam is significant because it reframes joint performance-and-cinematography control as a conditioning problem rather than a retraining problem. Its main technical elements—reference-character removal, weighted scene transfer, target-view rendering of pose and background depth, and a two-phase denoising schedule—form a coherent recipe for training-free control over both actor motion and camera trajectory. In that sense, ActCam defines a methodological bridge between pose-guided human animation and camera-consistent video synthesis, with its strongest advantages appearing under large viewpoint changes and moving-camera scenarios (Khalifi et al., 7 May 2026).