Kubric MOVi-C Dataset for Object-Centric Learning

• Kubric MOVi-C Dataset is a synthetic video dataset designed for object-centric tasks by blending photorealistic rendering, physics simulation, and diverse 3D scan data.
• It employs the Kubric Python framework with PyBullet and Blender Cycles to achieve realistic physics and rendering under HDRI lighting conditions.
• The dataset supports benchmarking of unsupervised object discovery, motion-based grouping, and temporal segmentation while providing rich, per-frame ground-truth annotations.

The Kubric MOVi-C dataset constitutes the third tier of the Multi-Object Video (MOVi) series, explicitly designed to advance object-centric discovery and segmentation methodologies within highly realistic and dynamic synthetic video environments. It targets benchmarking of unsupervised object discovery, motion-based grouping, and assessment of temporal consistency for methods at the intersection of visual perception and object-centric learning. MOVi-C distinguishes itself by leveraging photorealistically rendered, scanned household objects against real-world High Dynamic Range Image (HDRI) backgrounds, thereby introducing a level of texture and scene fidelity absent from earlier MOVi dataset tiers (Greff et al., 2022).

1. Hierarchical Position and Comparative Context

MOVi-C sits at an intermediate complexity level within the MOVi series. Unlike MOVi-A and MOVi-B—which rely on geometric primitives (spheres, cubes, cones, cylinders, pastel colorings, and uniform lighting) and simple scene arrangements—MOVi-C employs 5–10 high-quality 3D object scans per scene, drawn from both Google Scanned Objects (GSO, approximately 1,000 unique meshes) and a 51,300-model subset of ShapeNetCore.v2 encompassing 55 object categories. Each object carries real-world Physically-Based Rendering (PBR) textures. Environments feature randomized HDRI backgrounds from Polyhaven, and lighting is further modulated via a synthetic point source with color jitter. In contrast, MOVi-D increases object counts and introduces sparse movers, and MOVi-E adds a moving camera trajectory to further complicate temporal analysis (Greff et al., 2022).

Dataset	Object Set	Scene Complexity	Camera Motion
MOVi-A	8 primitives	Flat, simple	Static
MOVi-B	8 primitives + cones, capsules	Checkerboard, pastel	Static
MOVi-C	5–10 scanned meshes	HDRI backgrounds	Static
MOVi-D	up to 23 (dynamic + static)	Mixed motion	Static
MOVi-E	up to 23	Mixed, moving camera	Dynamic

2. Data Generation Pipeline and Framework Architecture

The Kubric Python framework orchestrates MOVi-C data synthesis by interfacing PyBullet for physical simulation and Blender’s Cycles for photorealistic rendering. The workflow encompasses:

• Scene Initialization: A Scene object encapsulates global settings (gravity, frame count, camera intrinsics, lighting).
• Asset Ingestion: Assets sourced via KuBasic, ShapeNetCore.v2, or GSO manifests provide mesh data, real-world PBR textures, and URDF collision geometry.
• Physics Simulation: PyBullet manages rigid-body dynamics, executing 24 simulation steps ($\Delta t=1/24$ s per step) to produce per-frame object trajectories and collision logs. Physical parameters include standard gravity ($g = [0,0,-9.81]$), restitution (0.5), and friction (0.4).
• Rendering: Blender Cycles, synchronized via shared scene graphs, interpolates object motion and ray-traces output modalities at 64 samples per pixel, 256×256 px resolution, with maximum bounce count set to 8 and no denoising.
• Export: RGB, instance segmentation, depth, optical flow, and surface normals are exported in standardized formats, augmented by JSON metadata per scene (Greff et al., 2022).

Orchestration ensures that each asset is congruently represented in both simulation and rendering domains—position, orientation, and scale are mirrored and governed by the same random initializations.

3. Scene, Object, and Motion Configuration

Each MOVi-C sequence instantiates $n \sim Uniform\{5, ..., 10\}$ objects per scene, with object placement assured non-overlapping via rejection sampling. Object scale per instance is sampled as $s \sim \mathcal{U}([0.8,1.2])$, allowing modest in-category variation. Appearance leverages real-world textures mapped from the associated 3D mesh, with scene backdrops stochastically drawn from an HDRI pool and lighting augmented by a point-light of intensity $I \sim \mathcal{U}([5,15])$ W, color sampled componentwise as $\mathcal{U}([0.8,1.2])$.

Initial object velocities follow $\|\vec{v}\| \sim \mathcal{U}(0,2)$ units/s, with directions sampled uniformly from the unit sphere. No extraneous external forces act beyond gravity. Rotational inertia tensors derive from convex-decomposed URDFs, ensuring accurate dynamic response. This parameterization provides a high-variance, yet physically plausible, set of motion conditions for object discovery methods to exploit (Greff et al., 2022).

4. Camera Model, Lighting, and Rendering Modalities

The camera adopts a Blender perspective model with focal length $f = 50$ mm, sensor width 36 mm, yielding 256×256 pixel outputs. Its location is randomized about the scene center: elevation $\theta \sim \mathcal{U}(30^\circ, 60^\circ)$, azimuth $\phi \sim \mathcal{U}(0^\circ, 360^\circ)$, but is always static per sequence in MOVi-C (unlike MOVi-E). Backgrounds rely on Polyhaven-provided HDRI, delivering varied, realistic global illumination.

Output modalities include:

• RGB: 8-bit PNG per frame
• Instance Segmentation: 16-bit PNG (object IDs)
• Depth: 32-bit float EXR
• Optical Flow: .flo format (Middlebury convention)
• Surface Normals: 32-bit EXR in world coordinates

Rendering uses Cycles with 64 samples per pixel and a maximum of 8 light bounces, without denoising or motion blur; the system supports automated depth-of-field should it be enabled (Greff et al., 2022).

5. Annotations, Metadata, Storage, and Dataset Statistics

Rich ground-truth annotation accompanies every sequence:

• Pixel-level: instance masks, optical flow vectors, depth maps, normal maps
• Object-centric: 3D pose ($g = [0,0,-9.81]$0), bounding box, mass, friction, velocity, per-object collision logs
• Camera: intrinsic matrix $g = [0,0,-9.81]$1, extrinsic $g = [0,0,-9.81]$2

Data organization is per-scene:

$g = [0,0,-9.81]$7

The full dataset comprises 100,000 video sequences (24 frames each), totaling approximately 2.4 million frames and ~1.5 TB. Dataset splits are 80% training (80,000 scenes), 10% validation (10,000 scenes), and 10% test (10,000 scenes). The object count per scene is mean $g = [0,0,-9.81]$3, with a pool of 1,000 unique object meshes and 55 ShapeNet categories (Greff et al., 2022).

6. Application Domains and Benchmarks

MOVi-C directly targets the evaluation of unsupervised object discovery and segmentation from video, with additional utility for examining temporal consistency and the role of motion cues in object grouping. Practical use cases include:

• Slot-attention and spatial-slot models: assessment of robustness to real-world textures and backgrounds
• Motion-based grouping: evaluation of generalization to complex object geometry
• Temporal segmentation: measuring trajectory consistency

Reported benchmarks for unsupervised object discovery (Foreground Adjusted Rand Index, ARI) are as follows:

Method	Foreground ARI (%)
SAVi [Anon ’22]	$g = [0,0,-9.81]$4
SIMONe [Kabra ’21]	$g = [0,0,-9.81]$5
SAVi + BBox Init Cue	$g = [0,0,-9.81]$6

A plausible implication is that the complexity and realism of MOVi-C substantially challenge current object-centric video models, especially with respect to segmentation under realistic appearance and lighting statistics (Greff et al., 2022).

7. Significance, Limitations, and Future Context

MOVi-C, as part of the Kubric datasets, exemplifies the scalability and annotation richness afforded by synthetic data generation. Its design addresses the limitations of uniform, low-level synthetic scenes and facilitates controlled experimentation with complex objects and photorealistic environments. This framework enables reproducible, large-scale evaluation across object segmentation, motion analysis, and temporal consistency tasks. A plausible implication is that future research leveraging MOVi-C will contribute to closing the gap between synthetic and real data performance, but performance benchmarks indicate existing methods face substantially increased challenges as photorealism and scene complexity rise (Greff et al., 2022).