ActCam: Zero-Shot Joint Camera and 3D Motion Control for Video Generation

Abstract: For artistic applications, video generation requires fine-grained control over both performance and cinematography, i.e., the actor's motion and the camera trajectory. We present ActCam, a zero-shot method for video generation that jointly transfers character motion from a driving video into a new scene and enables per-frame control of intrinsic and extrinsic camera parameters. ActCam builds on any pretrained image-to-video diffusion model that accepts conditioning in terms of scene depth and character pose. Given a source video with a moving character and a target camera motion, ActCam generates pose and depth conditions that remain geometrically consistent across frames. We then run a single sampling process with a two-phase conditioning schedule: early denoising steps condition on both pose and sparse depth to enforce scene structure, after which depth is dropped and pose-only guidance refines high-frequency details without over-constraining the generation. We evaluate ActCam on multiple benchmarks spanning diverse character motions and challenging viewpoint changes. We find that, compared to pose-only control and other pose and camera methods, ActCam improves camera adherence and motion fidelity, and is preferred in human evaluations, especially under large viewpoint changes. Our results highlight that careful camera-consistent conditioning and staged guidance can enable strong joint camera and motion control without training. Project page: https://elkhomar.github.io/actcam/.

• The paper introduces a zero-shot framework that enables joint control of camera and actor motion in video generation.
• It employs a two-phase denoising schedule and 3D geometry-aware conditioning to maintain spatial and temporal consistency.
• Results show improved 3D consistency and motion fidelity over baseline methods without requiring model retraining.

ActCam: Zero-Shot Joint Camera and 3D Motion Control for Video Generation

Introduction

Image-conditioned video generation demands not only accurate character motion synthesis but also precise and controllable camera trajectories, essential for applications in cinematic content creation and visual storytelling. Existing paradigms in video generation often struggle to provide synchronized control over both human and camera motion, typically requiring model retraining or extensive fine-tuning, which impedes transferability and stylistic coherence across different video backbones. "ActCam: Zero-Shot Joint Camera and 3D Motion Control for Video Generation" (2605.06667) introduces a training-free framework that achieves zero-shot joint control over actor motion and per-frame camera parameters in video synthesis, without modification or specialization of the backbone diffusion model.

Methodology

Problem Formulation and Conditioning Strategy

ActCam operates on three inputs: a single reference image specifying the subject and scene, an acting video providing the target motion, and a user-defined camera trajectory. The method utilizes a pretrained video diffusion model (such as VACE) that supports conditioning on dense spatial signals, specifically 3D scene depth and articulated pose. The central challenge addressed is the construction of camera-aligned, temporally consistent conditioning signals that disambiguate between camera and character motion, achieving stable synthesis under large viewpoint changes.

The conditioning strategy in ActCam is built around two pivotal mechanisms: (i) compositional scene understanding and (ii) two-phase denoising. Background depth is estimated from an inpainted reference (with the main actor removed), while the actor's motion is reconstructed as 3D articulated pose sequences using state-of-the-art monocular motion estimators (e.g., GVHMR). These are dynamically aligned in 3D via geometric fitting, allowing camera and actor motion to be independently but coherently rasterized into dense conditioning videos from arbitrary camera viewpoints.

Figure 1: ActCam pipeline — background depth estimation, motion recovery and alignment, and control signal rasterization for pose and depth under arbitrary viewpoints, with a two-phase denoising schedule.

Scene Transfer and Geometry-Aware Fusion

To guarantee spatial coherence between inserted dynamic characters and the static scene, ActCam performs a scene transfer operation aligning mesh-based background and character geometries through weighted centroids focused on the boundary region. The depth of the inserted character is adjusted with importance weighting to resolve occlusions and maintain relative placement, which is shown to enhance both background/character consistency and visual plausibility.

Character removal from the reference depth map is critical to avoiding duplicated subjects and motion conflicts; ablations empirically confirm that failing to inpaint the reference actor yields severe duplication artifacts.

Figure 2: Character removal — reference character present in the depth map yields duplicate actors, emphasizing the importance of correct inpainting.

Two-Phase Conditioning Schedule

Monocular depth estimates are locally noisy but provide valuable global geometric structure. To balance over-constraint from static depth conditioning and under-specified guidance from pose-only control, ActCam employs a two-phase denoising scheme. Early denoising steps are conditioned on both pose and depth, locking in background geometry and enforcing parallax, while later stages drop depth, relying solely on pose for high-frequency detail refinement and motion adherence.

Ablation reveals the necessity of this schedule: full-depth guidance produces static backgrounds, while pose-only fails to resolve ambiguities between camera and character motion.

Figure 3: Importance of conditioning schedule — persistent depth constraints inhibit dynamic backgrounds, while the two-phase scheme enables flexible motion rendering.

Figure 4: Importance of depth — pose-only control yields confounded camera/character motion, whereas depth inclusion correctly guides movement.

Quantitative and Qualitative Evaluation

Metrics and Benchmarks

ActCam was evaluated on both moving and static camera settings using the RealisDance-Val dataset and four canonical cinematic camera movements. The backbone models are held constant for comparisons, ensuring fairness. Metrics include VBench (subject/background consistency, imaging quality, etc.), WorldScore (3D consistency, object control), and geometric controls such as MPJPE and Sampson Error.

Results consistently show that ActCam outperforms Uni3C—a strong baseline requiring finetuning—across all visual and geometric metrics. Notably, ActCam demonstrates substantial margin in 3D consistency and motion error reduction, indicating not only framewise but also spatial/temporal alignment.

Figure 5: User study — ActCam is preferred over Uni3C for camera adherence, motion faithfulness, and visual quality.

Figure 6: Comparison with Uni3C — ActCam offers sharper camera control and more realistic articulated motion across varying scenarios.

On static camera motion control, ActCam surpasses 2D keypoint-based state-of-the-art methods (e.g., AnimateAnyone, HumanVid, SteadyDancer) in subject/background/appearance fidelity and temporal consistency, attributed to robust 3D pose-based conditioning.

Generalization and Scene Diversity

Demonstrations highlight robustness to large camera trajectories, transfer to varied scenes, and support for multi-character scenarios (when permitted by the underlying model architecture). Scene transfer and depth/pose conditioning confer adaptability to broad content domains.

Figure 7: Different scenes — same actor performance and camera trajectory rendered in scene-diverse outputs with consistent result quality.

Ablation and Design Analysis

Ablations validate:

• The criticality of reference character removal to avoid compositing artifacts and duplication (Figure 2).
• The improvement yielded by full scene transfer with importance weighting for precise occlusion and actor-scene interaction (Figure 8).
• The necessity of the two-phase denoising schedule for balancing geometric stability and motion flexibility (Figures 3–5).

  Figure 8: Importance of scene transfer — without proper alignment, depth and position inconsistencies degrade compositing and realism.

Implications and Future Directions

The ActCam approach provides a model-agnostic paradigm for joint, zero-shot control over both acting and camera trajectories, freeing video generation from task-specific training requirements. This presents direct implications for scalable content creation pipelines—especially in post-production and synthetic data generation—where artistic control, style continuity, and rapid adaptation to new backbones are important.

From a theoretical perspective, the method underscores the benefits of explicit geometric grounding in conditioning signals and the pitfalls of static/dynamic entanglement. Future research could extend ActCam by leveraging more precise or learned monocular depth estimators, integrating environment-aware sound and lighting controls, or scaling to multi-character, object-rich environments with richer holistic scene understanding.

Conclusion

ActCam sets a new standard for zero-shot, backbone-independent controllable video generation, integrating 3D geometry-aware conditioning, robust actor-scene alignment, and a two-phase denoising schedule to achieve fine-grained joint control over cinematographic camera and human motion. Its strong empirical results across geometric, perceptual, and user-centric metrics validate the approach and motivate further exploration of 3D-consistent conditioning strategies for video synthesis (2605.06667).