WorldCam-50h: Interactive 3D Video Dataset

• WorldCam-50h is a comprehensive video dataset featuring 50 hours of first-person gameplay with per-frame SE(3) pose annotations and synchronized action logs.
• It integrates diverse 3D environments from Counter-Strike, Xonotic, and Unvanquished, offering detailed geometric, behavioral, and textual metadata for fine-grained action control.
• The dataset is organized into stratified clips with hierarchical structure and standardized preprocessing, making it ideal for evaluating long-horizon, viewpoint-consistent video generation models.

WorldCam-50h is a large-scale, richly annotated video dataset designed to advance research in interactive 3D world modeling, with an explicit focus on fine-grained action control and long-horizon, view-consistent video generation. Collected from authentic human gameplay across three complex 3D environments—Counter-Strike (proprietary), Xonotic (CC BY-SA 2.5), and Unvanquished (GPL v3)—the dataset contains over 50 hours of continuous first-person exploration, with detailed per-frame camera trajectories in $SE(3)$, raw action logs, and textual summaries of each static environment. Its unique fusion of geometric pose annotation, multimodal metadata, and stratified environmental diversity aims to unify immediate action grounding with persistent 3D scene understanding for next-generation generative world models (Nam et al., 17 Mar 2026).

1. Dataset Composition and Scope

WorldCam-50h consists of approximately 3,000 minutes (50 hours) of single-player first-person video, recorded at 30 frames per second, downsampled to 480×832 resolution for modeling. Data were captured in static maps, with participants using all standard keyboard and mouse controls (W/A/S/D, jump, crouch, mouse movement), but with no non-player characters or adversarial agents. Each of the three games contributed about 1,000 minutes of footage, sampling diverse geometric, thematic, and architectural styles: Counter-Strike offers urban corridors and plazas (5 maps), Xonotic provides sci-fi arenas and tunnels (7 levels), and Unvanquished covers medieval and outdoor courtyard settings (6 maps).

Table 1 summarizes the distribution:

Game	Minutes (hrs)	Unique Maps/Levels
Counter-Strike	~1000 (17)	5 (urban, plaza)
Xonotic	~1000 (17)	7 (sci-fi)
Unvanquished	~1000 (17)	6 (medieval, outdoor)

Clips are segmented into non-overlapping one-minute intervals (1,800 frames per clip), resulting in approximately 100 gameplay videos per game. Each clip is assigned a textual description and a comprehensive set of geometric and behavioral annotations.

2. Data Annotation and Representation

Each frame of WorldCam-50h is annotated with a global camera pose $P_k \in SE(3)$, represented as a homogeneous transformation:

$$P_k = \begin{bmatrix} R_k & t_k \ 0^T & 1 \end{bmatrix}, \quad R_k \in SO(3),\ t_k \in \mathbb{R}^3,$$

where $R_k$ encodes orientation and $t_k$ encodes position. Pose estimation is performed at 30 Hz using the ViPE engine, with frames exceeding a translational magnitude threshold ($\|\Delta t\| > 0.5$ units/frame) excluded from training and validation; such frames comprise less than 2% of the data.

Action logs record simultaneous keyboard events and mouse deltas $(\Delta x, \Delta y)$, time-synchronized with video frames. For modeling, each action $A_k = [v_k; \omega_k] \in \mathbb{R}^6$—with $v_k$ linear and $\omega_k$ angular velocity—is mapped to $𝔰𝔢(3)$ via the twist matrix:

$$\hat{A}_k = \begin{bmatrix} \hat{\omega}_k & v_k \ 0^T & 0 \end{bmatrix}.$$

Relative camera motion derives from the matrix exponential $\Delta P_k = \exp(\hat{A}_k)$, accumulating as $P_k = P_{k-1} \circ \Delta P_k$ across frames.

Textual annotations are generated by Qwen2.5-VL-7B, delivering one concise paragraph per clip. Prompts emphasize layout, topology, primary regions, key objects, visual themes (color, material, architecture), and ambient conditions (lighting, weather). The average caption length is 42 words ($\sigma \approx 8$), with a vocabulary of approximately 2,200 unique tokens.

3. Dataset Organization and Access

WorldCam-50h adopts a hierarchical file structure:

• Each game has a subdirectory containing:
  • Raw video (.mp4)
  • Per-clip directories with:
  • Frames (lossless PNG, 480×832)
  • poses.json (array of 1,800 annotated poses per clip)
  • caption.txt (UTF-8 description)
  • Raw keyboard/mouse logs (.csv)
  • VAE precomputed latents for model efficiency

Clips are further split into 64-frame modeling segments (32 frames as context, 32 as generation target). Data splits are stratified by game: 80% training (~2,400 min), 10% validation (~300 min), 10% test (~300 min). Non-overlapping clips ensure full independence across partitions.

Licensing is structured by content: Counter-Strike data adhere to closed licensing with a research-specific fair-use agreement; Xonotic and Unvanquished samples are CC BY-SA 2.5 and GPL v3, respectively. The aggregate release (frames, poses, captions) is available under a CC BY 4.0 license via the project website.

4. Quantitative Statistics and Scene Dynamics

All clips comprise exactly 1,800 frames (60 seconds at 30 FPS). The pose sampling rate is 30 Hz. Camera motion statistics demonstrate mean linear speed of 0.15 units/frame ($\sigma\approx 0.07$) and mean angular speed of 0.9 degrees/frame ($\sigma\approx 0.4$). The average inter-frame translation is approximately 0.02 units; average inter-frame rotation angle is 0.9 degrees.

Pose quality filtering retains about 98% of frames after thresholding. Environment coverage encompasses 18 unique maps/levels distributed across three genres, enhancing geometric and thematic diversity.

Descriptive statistics for textual metadata:

• ~3,000 clip-level captions (one per minute)
• Average length: 42 words
• Vocabulary size: ~2,200 unique tokens
• Captions encapsulate spatial configuration, semantic themes, lighting, and prominent architectural or structural features.

5. Mathematical Foundations and Retrieval Metrics

Camera pose annotation leverages $SE(3)$ and $𝔰𝔢(3)$ representations, supporting precise modeling of 6-DoF trajectories. The action-to-pose mapping via the matrix exponential enables direct grounding of human actions in trajectory space. Pose composition is realized as $P_k = P_{k-1} \Delta P_k$.

Memory retrieval within modeling tasks operates through metric-based candidate ranking. Given a query pose $P_i = [R_i, t_i]$ and a set of candidate poses $\{P_j\}$, primary selection occurs via translational proximity:

$\text{score}_t(P_j) = -\|t_j - t_i\|_2,$

with the top $K_1$ retained. Among these, orientation alignment is evaluated as:

$\text{score}_r(P_j) = \mathrm{tr}(R_j^T R_i),$

with the top $K_2$ selected for further processing. This ensures retrieval of both spatially and rotationally consistent reference frames during long-horizon navigation or revisitation.

6. Preprocessing and Latent Encoding Pipeline

Raw videos, approximately eight minutes each, are segmented into standardized one-minute clips. Per-clip intrinsics and extrinsics are extracted via ViPE, and clips are subdivided into 64-frame modeling windows. A VAE encoder (Wan-2.1-T2V) precomputes latent tensors $z \in \mathbb{R}^{f \times h \times w \times c}$ for each segment, enabling resource-efficient downstream training. Metadata—including boundaries, poses, captions, and interaction logs—are compiled as comprehensive JSON records, supporting reproducible experimental splits and distributed training.

7. Research Impact and Applications

WorldCam-50h provides a high-fidelity foundation for the design and evaluation of generative models that require persistent 3D consistency and interactive action alignment. By furnishing precise, per-frame camera trajectories in $SE(3)$, synchronized action logs, and linguistically rich scene descriptions, the dataset supports tasks ranging from autoregressive world modeling, video synthesis, viewpoint-consistent memory retrieval, to imitation learning with geometric grounding. Its explicit stratification, licensing clarity, and focus on mapping human actions to geometric pose evolution make it a benchmark for evaluating long-horizon, action-aware 3D video generation (Nam et al., 17 Mar 2026).