HoloStateData: Unified Video Control Dataset

• HoloStateData is a source-to-state video dataset that unifies structured controls (camera, object, weather) to enable first-frame anchored generation.
• It combines real, simulated, and V2V subsets with strict separation between source controls and target-weather supervision.
• Its factorized design supports methods like Holo-World, achieving state-of-the-art scene preservation alongside effective weather transformations.

Searching arXiv for the specified paper to ground the article in the cited source. HoloStateData is a source-to-state video dataset introduced as the dataset backbone of Holo-World, a unified controllable video world model that starts from one first frame and then follows explicit camera and object controls while optionally transforming the scene into a target weather state (Yin et al., 18 Jun 2026). Its central design principle is that controllability should not be split across separate datasets or tasks. Instead, it unifies world-preservation supervision and weather-state supervision in one shared interface, so that the source frame plus source-side controls define what must be preserved, while target-weather supervision defines what must change.

1. Definition and motivating problem

HoloStateData was created for a target problem that prior controllable video datasets do not fully cover. In the formulation associated with Holo-World, camera-control datasets supervise viewpoint change, object-motion datasets supervise dynamic entities, and weather datasets usually assume a full source video or a reconstructed scene already provides the future structure (Yin et al., 18 Jun 2026). That setup is insufficient for first-frame-anchored generation with camera + object control and weather transfer in the same model.

The dataset therefore provides two complementary forms of supervision. First, it provides source-side controls that define the world to preserve. Second, it provides target-weather supervision that defines how that world should look under a new environmental state. The paper emphasizes that this is a source-to-state setting, and that the source side is never contaminated by target-weather evidence. A common misconception would be to view HoloStateData as a generic captioned video corpus or as a conventional weather-edit dataset; the paper instead positions it as a unified training substrate for controllable world models.

2. Source-to-state formulation

Each clip in HoloStateData is represented as a source-to-state record rather than a generic captioned video (Yin et al., 18 Jun 2026). The model sees only the first frame, denoted $I_0$, as the visual anchor. Source-side controls define the observed world through camera-conditioned background motion, static geometry anchors, and object boxes. If available, a target weather video plus a weather prompt define the state to synthesize.

The paper stresses a strict separation between source and target evidence. Source controls are built from the source side only, and target-weather pixels are never used to render model-facing controls. This is the condition that makes first-frame-anchored generation possible: future scene evolution must be inferred from the first frame together with structured controls, rather than from a complete source video. A plausible implication is that the dataset is organized not merely around visual diversity, but around causal separation between preserved structure and transformed environmental state.

3. Data sources and construction pipeline

HoloStateData is built from three sources that are converted into a common control format (Yin et al., 18 Jun 2026). The Real subset uses real videos from SpatialVID-HQ, characterized as in-the-wild scenes with diverse camera trajectories and object motion. The Simulation subset uses paired simulated weather videos rendered in Unreal Engine, where the same scene is captured under clear and target weather. The V2V subset uses weather-transferred videos for real clips, generated using the paper’s weather-transfer model and proprietary video editing models, extending weather supervision to real trajectories and dynamics.

The construction pipeline has three stages: data collection, video annotation, and scene construction. After these stages, clips are converted into unified controls: scene text is factorized, object masks become bounding-box controls, geometry is rendered into RGB/depth/normal controls, and paired weather clips provide target supervision. Every sample is normalized into the same interface consisting of source world controls, source appearance anchor, scene prompt, and weather prompt if applicable. This normalization allows very different videos to become comparable training samples for camera, object, and weather supervision.

4. Annotation schema and control signals

HoloStateData includes text annotations, object controls, and rendered camera/geometry/background controls in a single supervision schema (Yin et al., 18 Jun 2026). The scene prompt is produced from Qwen3-VL and covers scene identity, spatial layout, visible objects, object relations, temporal evolution, and camera motion, while explicitly excluding weather. The weather prompt describes only the target weather state and is restricted to a small taxonomy: cloud, rain, snow, fog. Real samples have no weather-edit prompt.

Object controls are produced through first-frame object detection with Grounding DINO, followed by mask propagation with SAM-2, after which each mask is converted into a tight per-frame bounding box. These boxes are the model-facing object control signals. Camera, geometry, and background controls are produced using MegaSaM for camera parameters, Depth Anything v2 and UniDepth for monocular depth priors, and MoGe for dense depth and normal anchors. The rendered controls include RGB background, depth maps, and normal maps. The rendered RGB has a dual role: it is used both as world control and as the source appearance anchor for state transfer.

These signals jointly supervise camera motion, object motion, background/scene structure, and weather-state transfer. The dataset’s control interface is therefore factorized but not fragmented: scene structure, object layout, and environmental state are represented separately while remaining aligned within one sample schema.

5. Role in Holo-World’s factorized method design

HoloStateData is presented not only as a dataset but also as the condition that makes the associated method design trainable (Yin et al., 18 Jun 2026). Its factorized supervision supports the Unified Scene Adapter, or UniSA, which splits learning into two disjoint residual spaces. The World Adapter learns residuals for camera, background geometry, and object layout, while the State Adapter learns residuals for weather appearance, atmospheric effects, and particle effects. The paper further injects these residuals into the frozen backbone through

$$x_\ell = F_\ell(x_{\ell-1},c) + \lambda_w h^w_\ell + \lambda_s h^s_\ell .$$

At inference time, the same factorization underlies Scene-Weather Decomposed CFG, or SW-CFG. The sampler evaluates separate branches and decomposes guidance into a scene component and a weather component. In the paper’s interpretation, $\Delta_{\mathrm{scene}}$ preserves the controlled world, while $\Delta_{\mathrm{weather}}$ adds weather-specific effects, thereby avoiding a single monolithic CFG direction that would entangle scene preservation with weather transfer. This decomposition is meaningful only because HoloStateData provides source-world controls for preservation and separate weather supervision for state transfer.

6. Scale, subsets, curation, and reported function

The appendix reports the full split statistics for HoloStateData (Yin et al., 18 Jun 2026).

Subset	Training / Benchmark	Role
Real	7,571 / 50	source-world preservation
Simulation	3,541 / 50	aligned structure, controlled weather change
V2V	3,954 / 50	real camera/object dynamics with weather supervision

The total training set contains 15,066 samples, and the benchmark contains 150 samples. The weather training subset spans four weather families: cloud, rain, snow, and fog.

The dataset is curated with several explicit rules. Only 81-frame clips are retained; source/target separation is enforced; target-weather pixels are never used to generate controls; and for V2V samples only examples are kept where target weather is clearly visible, source layout is preserved, no severe temporal artifacts appear, and no major object distortions are present. In addition, the first rendered frame is pinned to the observed source frame to reduce drift. These rules indicate that the dataset is intended to teach structured control rather than accidental editing artifacts.

The paper distinguishes HoloStateData from prior video or world-model datasets in three ways. It is source-to-state rather than just video captioning or unconditional video; it jointly covers camera, object, and weather control; and it separates world evidence from state supervision by ensuring that rendered controls come only from the source side while target weather is used only as output supervision. In the reported experiments, this dataset foundation enables Holo-World to perform controllable scene preservation on the Real subset and weather-state generation on the Weather subset. On the Real subset, Holo-World achieves the best overall VBench-I2V score and the lowest camera/object control errors. On the Weather subset, it achieves 86.00% Weather Alignment and 68.51 VLM Evaluation, together with strong human preference over V2V weather-editing baselines (Yin et al., 18 Jun 2026). The ablations further indicate that geometry/G-buffer controls help, UniSA improves preservation and structured transfer, and SW-CFG is crucial for boosting weather effects without degrading the preserved scene.