SierpinskiCam: Fractal-Guided Video Retaking

• SierpinskiCam is a video retaking system that synthesizes new views from a single monocular video using geometry-based proxy rendering and synthetic fractal dome cues.
• It combines 4D reconstruction with reference-video conditioning and a novel NegRoPE mechanism to disentangle positional signals and improve camera controllability.
• Empirical evaluations on DAVIS and MultiCamVideo show significant gains in rotation, translation, and visual quality metrics over state-of-the-art baselines.

SierpinskiCam denotes a video-to-video generative system for camera-controlled video retaking: given a single monocular source video and a user-specified target camera trajectory, it synthesizes a new video that follows the requested camera motion while preserving the appearance and dynamics of the original scene. In current arXiv usage, the term refers specifically to the system introduced in 2026, which combines monocular 4D reconstruction, synthetic Sierpinski triangle dome cues, and reference-video conditioning using negative rotary positional embeddings (NegRoPE) inside a Wan video diffusion transformer (Wizadwongsa et al., 15 Jun 2026). A broader antecedent usage, drawn from fractal-image generation in the complex plane, treats a SierpinskiCam as a camera-like system that navigates and warps Sierpinski-type fractals by applying complex maps $f(z)$ to a generated gasket (0903.4053).

1. Problem setting and system definition

SierpinskiCam addresses video retaking from a single monocular video. The input is a monocular source video showing a dynamic scene together with a target camera trajectory, and the output is a retaken video depicting the same scene and motion as if captured along the target trajectory. The central difficulty is that geometry reconstructed from a single source video is necessarily partial: when the target camera departs strongly from the source path, newly revealed regions become sparse or entirely missing in geometry-guided warps, so a diffusion model must inpaint under weak conditioning. The cited failure modes include rotation and translation errors, objects remaining in frame when they should leave it, and hallucinated geometry in disoccluded regions (Wizadwongsa et al., 15 Jun 2026).

The architecture is organized around three components. First, it uses geometry-based proxy rendering from monocular 4D reconstruction. Second, it augments the warped proxy with Sierpinski dome texture cues, rendered on a synthetic spherical dome and composited into pixels where geometry is absent. Third, it appends source-video tokens to target-video tokens inside the diffusion transformer and separates the two streams with negative RoPE indices, enabling appearance grounding without architectural modification or per-video adaptation. This arrangement is designed to improve camera controllability, geometric consistency, and video quality under large viewpoint change (Wizadwongsa et al., 15 Jun 2026).

A common misconception is that the Sierpinski component replaces geometric guidance. The formulation is explicitly hybrid: geometry remains the primary proxy, while the Sierpinski dome supplies dense motion-consistent cues precisely where the proxy becomes sparse. Another misconception is that the method requires physical fiducials in the captured scene. The integration is stated to be purely synthetic; no physical markers in real scenes are required (Wizadwongsa et al., 15 Jun 2026).

2. Geometry proxy and Sierpinski dome cues

The geometry branch follows an explicit reconstruction-and-warp pipeline. For each source frame, SierpinskiCam estimates depth and camera parameters with DepthAnything-V3, producing a per-pixel depth map $D_t$, intrinsics $\mathbf{K}_t$, and extrinsics $\mathbf{T}_t=[\mathbf{R}_t\mid \mathbf{t}_t]$, with temporal smoothing of extrinsics to remove jitter. Back-projection uses the standard pinhole model

$$\lambda \mathbf{p} = \mathbf{K}_t(\mathbf{R}_t \mathbf{X} + \mathbf{t}_t), \qquad \mathbf{X} = \mathbf{R}_t^\top \left(D_t(u,v)\mathbf{K}_t^{-1}\mathbf{p} - \mathbf{t}_t\right),$$

yielding a dense colored point cloud. A second component, SpaTracker-V2, provides sparse 3D point tracks rendered as colored points in the target view for additional motion anchors. Forward splatting to the target camera produces a warped RGB proxy $I_{\text{warp},t}$ and a validity mask $M_t\in\{0,1\}$ (Wizadwongsa et al., 15 Jun 2026).

The Sierpinski dome is introduced when $M_t$ becomes sparse. The dome is a sphere of radius

$R=\min(R_{\max}, d_{\max}),$

where $d_{\max}$ is the maximum distance of confident proxy points and $D_t$0. For each target pixel, a ray is intersected with the sphere, the intersection point is converted to spherical coordinates

$D_t$1

and a Sierpinski texture is sampled to form $D_t$2. The conditioning frame is

$D_t$3

The texture itself is a $D_t$4 RGB image divided into a $D_t$5 grid of cells, with a depth-3 recursive Sierpinski triangle in each cell, alternating orientation across rows and columns and using half-cell-shifted triangles for the opposite orientation to avoid large uniform regions (Wizadwongsa et al., 15 Jun 2026).

The stated rationale is feature-theoretic. The Sierpinski pattern provides high-contrast edges and corners at multiple scales, is described as rich in trackable features, and induces coherent parallax on the sphere under camera motion. The paper reports that, under SIFT + RANSAC analysis, Sierpinski yields the highest inlier matches compared to checkerboard, triangle grid, circle fractal, and uniform textures. This makes the camera-control signal explicit even where scene geometry is absent (Wizadwongsa et al., 15 Jun 2026).

3. Reference-video conditioning and NegRoPE

The appearance-preservation component is implemented inside a Wan video diffusion transformer. Let $D_t$6 denote the noisy target-video latent at diffusion step $D_t$7, $D_t$8 the clean camera-controlling latent from geometry plus dome, $D_t$9 the noisy source-video latent, and $\mathbf{K}_t$0 the clean source-video latent. The model forms two token streams: $\mathbf{K}_t$1 then concatenates them along the token dimension: $\mathbf{K}_t$2 This lets standard self-attention propagate information between source and target without introducing new cross-attention blocks (Wizadwongsa et al., 15 Jun 2026).

The crucial positional mechanism is NegRoPE. RoPE rotates query and key channels according to position index $\mathbf{K}_t$3: $\mathbf{K}_t$4 Naively assigning the same spatial indices to source and target streams over-couples tokens that merely share grid coordinates. SierpinskiCam instead assigns positive spatial RoPE indices to target tokens and negative spatial RoPE indices to source tokens. The embedding at $\mathbf{K}_t$5 is the complex conjugate of the embedding at $\mathbf{K}_t$6, so the positional signatures are distinct but related. Temporal RoPE is left unchanged; only the height and width components are manipulated. The claimed effect is position-disentangled joint attention: source-target interactions are guided by semantic compatibility rather than accidental positional equality, and this is achieved with no new modules and no per-video finetuning (Wizadwongsa et al., 15 Jun 2026).

Ablations compare frame concatenation, token concatenation with no RoPE modification, token concatenation with a large positive offset such as $\mathbf{K}_t$7, and token concatenation with NegRoPE. NegRoPE is reported to yield the best CLIP, DINO, and FID metrics, supporting its role as the preferred source-injection strategy in this architecture (Wizadwongsa et al., 15 Jun 2026).

4. Training protocol and empirical performance

The base model is Wan2.1 Fun-Control 14B, with a 1.3B version for ablations. Training uses MultiCamVideo, a synthetic Unreal Engine dataset with 10 synchronized cameras per scene, and RealEstate10K, used to augment realism and to train operation with camera-controlling signals even without a reference-video stream. Optimization is LoRA fine-tuning with rank $\mathbf{K}_t$8, learning rate $\mathbf{K}_t$9, and $\mathbf{T}_t=[\mathbf{R}_t\mid \mathbf{t}_t]$0 epochs; the first-frame signal is randomly dropped $\mathbf{T}_t=[\mathbf{R}_t\mid \mathbf{t}_t]$1 of the time for robustness. At inference, the pipeline reconstructs geometry, renders $\mathbf{T}_t=[\mathbf{R}_t\mid \mathbf{t}_t]$2, $\mathbf{T}_t=[\mathbf{R}_t\mid \mathbf{t}_t]$3, and $\mathbf{T}_t=[\mathbf{R}_t\mid \mathbf{t}_t]$4, encodes $\mathbf{T}_t=[\mathbf{R}_t\mid \mathbf{t}_t]$5, prepares source conditioning, and samples the target latent $\mathbf{T}_t=[\mathbf{R}_t\mid \mathbf{t}_t]$6 through the standard diffusion schedule before VAE decoding to RGB frames (Wizadwongsa et al., 15 Jun 2026).

Evaluation targets three axes. Camera controllability uses Rotation Error (RotErr), Translation Error (TransErr), and Absolute Trajectory Error (ATE), with camera poses estimated from generated videos by VGGT for static scenes and DepthAnything-V3 for dynamic scenes. Geometric consistency uses Dyn-MEt3R and per-frame MEt3R. Visual quality uses VBench metrics including subject consistency, background consistency, aesthetic quality, imaging quality, and flickering or temporal consistency (Wizadwongsa et al., 15 Jun 2026).

On DAVIS (90 videos) with 14 target camera trajectories, amounting to 1,260 video-trajectory pairs, SierpinskiCam is reported as best or second-best on most VBench metrics, and specifically achieves Dyn-MEt3R $\mathbf{T}_t=[\mathbf{R}_t\mid \mathbf{t}_t]$7, RotErr $\mathbf{T}_t=[\mathbf{R}_t\mid \mathbf{t}_t]$8, and ATE $\mathbf{T}_t=[\mathbf{R}_t\mid \mathbf{t}_t]$9, all marked as best. On MultiCamVideo, reported reconstruction metrics are PSNR $$\lambda \mathbf{p} = \mathbf{K}_t(\mathbf{R}_t \mathbf{X} + \mathbf{t}_t), \qquad \mathbf{X} = \mathbf{R}_t^\top \left(D_t(u,v)\mathbf{K}_t^{-1}\mathbf{p} - \mathbf{t}_t\right),$$0 versus approximately $$\lambda \mathbf{p} = \mathbf{K}_t(\mathbf{R}_t \mathbf{X} + \mathbf{t}_t), \qquad \mathbf{X} = \mathbf{R}_t^\top \left(D_t(u,v)\mathbf{K}_t^{-1}\mathbf{p} - \mathbf{t}_t\right),$$1–$$\lambda \mathbf{p} = \mathbf{K}_t(\mathbf{R}_t \mathbf{X} + \mathbf{t}_t), \qquad \mathbf{X} = \mathbf{R}_t^\top \left(D_t(u,v)\mathbf{K}_t^{-1}\mathbf{p} - \mathbf{t}_t\right),$$2 for baselines, LPIPS $$\lambda \mathbf{p} = \mathbf{K}_t(\mathbf{R}_t \mathbf{X} + \mathbf{t}_t), \qquad \mathbf{X} = \mathbf{R}_t^\top \left(D_t(u,v)\mathbf{K}_t^{-1}\mathbf{p} - \mathbf{t}_t\right),$$3 versus approximately $$\lambda \mathbf{p} = \mathbf{K}_t(\mathbf{R}_t \mathbf{X} + \mathbf{t}_t), \qquad \mathbf{X} = \mathbf{R}_t^\top \left(D_t(u,v)\mathbf{K}_t^{-1}\mathbf{p} - \mathbf{t}_t\right),$$4–$$\lambda \mathbf{p} = \mathbf{K}_t(\mathbf{R}_t \mathbf{X} + \mathbf{t}_t), \qquad \mathbf{X} = \mathbf{R}_t^\top \left(D_t(u,v)\mathbf{K}_t^{-1}\mathbf{p} - \mathbf{t}_t\right),$$5, and FID $$\lambda \mathbf{p} = \mathbf{K}_t(\mathbf{R}_t \mathbf{X} + \mathbf{t}_t), \qquad \mathbf{X} = \mathbf{R}_t^\top \left(D_t(u,v)\mathbf{K}_t^{-1}\mathbf{p} - \mathbf{t}_t\right),$$6 versus $$\lambda \mathbf{p} = \mathbf{K}_t(\mathbf{R}_t \mathbf{X} + \mathbf{t}_t), \qquad \mathbf{X} = \mathbf{R}_t^\top \left(D_t(u,v)\mathbf{K}_t^{-1}\mathbf{p} - \mathbf{t}_t\right),$$7–$$\lambda \mathbf{p} = \mathbf{K}_t(\mathbf{R}_t \mathbf{X} + \mathbf{t}_t), \qquad \mathbf{X} = \mathbf{R}_t^\top \left(D_t(u,v)\mathbf{K}_t^{-1}\mathbf{p} - \mathbf{t}_t\right),$$8. In a user study with 41 participants on 10 DAVIS videos, mean ratings on a 1–5 scale are Overall $$\lambda \mathbf{p} = \mathbf{K}_t(\mathbf{R}_t \mathbf{X} + \mathbf{t}_t), \qquad \mathbf{X} = \mathbf{R}_t^\top \left(D_t(u,v)\mathbf{K}_t^{-1}\mathbf{p} - \mathbf{t}_t\right),$$9 versus $I_{\text{warp},t}$0 for the next best method, Camera motion accuracy $I_{\text{warp},t}$1 versus approximately $I_{\text{warp},t}$2–$I_{\text{warp},t}$3, and Source stability/consistency $I_{\text{warp},t}$4 versus $I_{\text{warp},t}$5 for the next best method (Wizadwongsa et al., 15 Jun 2026).

Setting	Metric	Reported value
DAVIS	Dyn-MEt3R	0.7477
DAVIS	RotErr	0.2058
DAVIS	ATE	0.6921
MultiCamVideo	PSNR	19.64
MultiCamVideo	LPIPS	0.29
MultiCamVideo	FID	32.80

These results are presented as evidence that the dome prevents guidance collapse when target trajectories depart strongly from the source path, while NegRoPE preserves appearance grounding without architectural expansion (Wizadwongsa et al., 15 Jun 2026).

5. Ablations, baseline comparisons, and limitations

The ablation studies separate the influence of explicit 3D guidance and texture cues. On RealEstate10K under hard trajectories, the reported camera-control results are: Plücker rays with RotErr $I_{\text{warp},t}$6 and TransErr $I_{\text{warp},t}$7, Texture only with $I_{\text{warp},t}$8 and $I_{\text{warp},t}$9, 3D only with $M_t\in\{0,1\}$0 and $M_t\in\{0,1\}$1, and Texture + 3D (full SierpinskiCam) with $M_t\in\{0,1\}$2 and $M_t\in\{0,1\}$3, marked as best. In the texture-only comparison, Checkerboard gives RotErr $M_t\in\{0,1\}$4 and TransErr $M_t\in\{0,1\}$5, while Sierpinski gives $M_t\in\{0,1\}$6 and $M_t\in\{0,1\}$7. The intended conclusion is narrow and empirical: geometry plus Sierpinski texture is better than geometry alone, texture alone, or Plücker-ray implicit encoding for camera control, and the Sierpinski pattern is a better motion cue than a simple checkerboard (Wizadwongsa et al., 15 Jun 2026).

Qualitative comparisons are also sharply delimited. The paper states that implicit baselines such as ReCamMaster and ReDirector frequently keep main objects in frame when the target camera should move away and conflate subject motion with camera motion. TrajectoryCrafter is described as better at disentanglement but prone to failure when valid warp coverage is small, including hallucinated objects and misplacement when the subject becomes small or leaves the frustum. By contrast, SierpinskiCam is reported to maintain expected subject motion relative to camera motion, such as a swan disappearing correctly during zoom-out or a pedestrian moving out of frame when the camera is static but the subject moves. A plug-and-play experiment also shows that replacing black disocclusion fill in ReAngle-A-Video with the Sierpinski dome improves camera ATE substantially (Wizadwongsa et al., 15 Jun 2026).

The stated limitations are threefold. First, the system depends on the quality of 4D reconstruction and on video diffusion priors, so poor depth or camera estimation and strong non-rigid motion degrade guidance. Second, scenes with extremely complex dynamic motion, heavy occlusions, or severe motion blur remain challenging. Third, the synthetic dome introduces an artificial background that may not always be aesthetically ideal, so careful compositing is needed. Proposed future directions include stronger or 4D-complete reconstruction such as FreeOrbit4D and Vista4D, learned camera-flow-field priors that do not require explicit synthetic textures, extension to fully dynamic 4D scenes with moving backgrounds or multi-object interactions, and greater efficiency at higher resolution or longer sequence length (Wizadwongsa et al., 15 Jun 2026).

6. Fractal-visualization antecedents and broader technical interpretation

Before the 2026 video-retaking system, a broader tutorial-style synthesis based on “The generating of Fractal Images Using MathCAD Program” described a SierpinskiCam system as essentially a “camera” that navigates and warps Sierpinski-type fractals. In that formulation, the first ingredient is the generation of the Sierpinski gasket in the complex plane, and the second is the application of complex maps $M_t\in\{0,1\}$8 to the generated gasket to change its appearance (0903.4053).

The underlying geometry is the standard gasket obtained by repeated midpoint subdivision and removal of the central triangle. In the complex plane, a point is represented as $M_t\in\{0,1\}$9. Iterations are implemented through midpoint rules such as

$M_t$0

and the limit set may be written as

$M_t$1

The paper’s workflow constructs matrices $M_t$2 of complex points and then applies a complex map elementwise: $M_t$3 The specific functions examined are $M_t$4, $M_t$5, $M_t$6, $M_t$7, $M_t$8, and $M_t$9. The stated conclusion is that “The degree of the transforming function $R=\min(R_{\max}, d_{\max}),$0 produces a rotation of the image. Thus, for the first degree function the image of the fractals is represented on the first frame; for the second degree function the image of the fractals is represented on the two frames, and so on.” The paper also states that “If the transforming functions are modified the domain covered by the map of the fractals is modified” (0903.4053).

This earlier usage does not describe camera-controlled video retaking. It instead frames SierpinskiCam as a camera-like lens for fractal geometry, with $R=\min(R_{\max}, d_{\max}),$1 functioning as the lens map and the complex plane as the imaging domain. The later video-retaking system can therefore be read as a different technical instantiation of the same naming logic: a camera is controlled by Sierpinski-structured cues, but now those cues are synthetic textures rendered on a dome rather than direct complex-plane warp maps (0903.4053).

7. Conceptual significance

SierpinskiCam sits at the intersection of explicit geometry, synthetic feature design, and transformer-based video generation. Its defining move is not merely the insertion of a fractal texture, but the use of a Sierpinski triangle pattern as a dense, multi-scale camera-motion prior in regions where 4D warping is weak. In parallel, its NegRoPE mechanism reuses a standard DiT-style self-attention pipeline while disentangling positional signatures between source and target streams (Wizadwongsa et al., 15 Jun 2026).

The broader significance of the name is therefore twofold. In the 2026 sense, SierpinskiCam is a concrete retaking system with specified reconstruction, conditioning, and evaluation protocols. In the earlier fractal-imaging sense, it suggests a camera-like apparatus for exploring self-similar geometry through analytic transformations. The coexistence of these uses is technically coherent rather than contradictory: both rely on the Sierpinski motif as a structured, multi-scale representation whose utility derives from self-similarity, dense corner structure, and predictable transformation under mapping or viewpoint change (Wizadwongsa et al., 15 Jun 2026).