Holi-World: Building Holistic World Intelligence from Human-Ego Experience

Abstract: Human egocentric experience offers a rich foundation for learning how the world is organized and how it changes through interaction. Realizing this potential at scale requires turning growing volumes of recordings into learning resources that connect scene structure, object semantics, and interaction histories across viewpoints and time, while reducing dependence on costly manual annotation. We introduce HOLI-WORLD, an integrated ecosystem connecting experience capture, world representation, automated data construction, model learning, and independent evaluation. To sup- port learning at scale, HOLIWORLD-Foundry combines complementary static and dynamic perception to automatically construct world records that connect persistent scene structure and evolving object states with interaction histories and supporting observations. HOLIWORLD-Data organizes these records from approximately 100 self-collected scenes into grounded learning resources, including approximately 80K QA pairs used to train HOLIWORLD-VL-4B. Independent human annotations of heldout recordings form HOLIWORLD-Bench, supporting evaluation of both annotation quality and learned spatial and temporal understanding. Experiments on ScanNet, ScanNet++, Aria Digital Twin, and HOLIWORLD-Bench demonstrate strong performance in static perception and dynamic tracking, while training on HOLIWORLD-Data substantially improves HOLIWORLD-VL-4B's spatial and temporal reasoning on HOLI- WORLD-Bench and EgoDynamic4D. HOLIWORLD provides a practical foundation for expanding learning resources from the continuous flow of human experience and advancing machine understanding of the dynamic world.

• The paper develops the HoliWorld ecosystem which combines static 3D object detection, dynamic tracking, and vision-language learning from human-egocentric data to enable evidence-driven supervision and reasoning tasks with an emphasis on creating reliable 3D-world annotations and answering questions.
• Evaluations indicate significant improvements in static 3D object perception across datasets like HoliWorld-Bench, ScanNet, and ScanNet++, with 0.55 AP$_{50}$ and 54.31% HOTA accuracies on object detection and tracking, respectively.
• Using this comprehensive framework, fine-tuning the downstream models on HoliWorld Data led to 78% F1 accuracy for HoliVision-Language 4B on Static tasks, showing significant performance enhancements in spatial reasoning.

Problem formulation and system scope

HoliWorld addresses the construction of spatially and temporally grounded learning supervision from human egocentric experience. Its central premise is that egocentric recordings provide a common substrate for three forms of information that are often handled independently: persistent scene geometry, object semantics, and interaction histories. The paper therefore treats world understanding not as a single perception task, but as an end-to-end data construction problem spanning capture, reconstruction, annotation, supervision generation, model training, and evaluation (2610.26549).

The proposed ecosystem comprises four tightly coupled components: an egocentric capture platform, HoliWorld-Foundry for automated world-record construction, HoliWorld-Data for organizing annotations and generating QA supervision, and HoliWorld-VL for downstream vision-language learning. HoliWorld-Bench provides independently annotated held-out recordings for evaluating both the automatically constructed records and the model trained on them.

Figure 1: HoliWorld links egocentric capture, automated static and dynamic annotation, and vision-language learning through aligned scene, object, and interaction representations.

The empirical claim is consequently broader than improved object detection or tracking in isolation. The paper argues that an automated perception stack can produce sufficiently reliable object-centric records to support supervision for spatial grounding, temporal reasoning, and interaction understanding. This claim is tested at three levels: static 3D object perception, dynamic 3D tracking, and downstream VLM reasoning.

HoliWorld ecosystem and data representation

The capture system records synchronized stereo RGB and IMU measurements. HoliWorld-Foundry first estimates camera poses and scene geometry, after which a VLM supplies object categories, attributes, motion cues, and interaction descriptions. These cues determine whether an entity is processed as a persistent static object or as a dynamic tracking target.

Figure 2: The HoliWorld ecosystem connects egocentric capture, Foundry, Data, VL training, and independently annotated evaluation.

The resulting world representation preserves links between observations and derived records. At each time step, a world state contains the RGB and depth observations, camera calibration and poses, scene geometry, 2D detections and masks, static and dynamic 3D instances, semantic labels, interaction annotations, confidence scores, and evidence links. Static instances persist across observations, whereas dynamic instances retain time-indexed states and identities.

This evidence traceability is important for two reasons. First, it permits QA answers to be grounded in particular objects, observations, and time intervals rather than generated as unsupported textual labels. Second, it allows errors in automatically generated supervision to be analyzed against the underlying perception outputs. The representation therefore serves simultaneously as an annotation format, a data lineage mechanism, and an interface between perception and VLM training.

HoliWorld-Data contains approximately 100 scenes and 444 egocentric videos. The release includes approximately 100 3D Gaussian-splatting reconstructions, 1.322 million RGB frames with corresponding depth maps and poses, 90,000 2D boxes, 2,466 static 3D boxes, and 2,095 dynamic tracks. These records yield approximately 4,000 3D grounding annotations and 80,000 QA pairs, divided into 13,647 static and 64,409 dynamic examples.

The supervision covers 30 QA types. Static tasks include depth, height, distance, relative position, and viewpoint-dependent relations. Dynamic tasks cover motion status, direction, duration, event order, path characteristics, interaction phases, state transitions, and temporal grounding. The breadth of the task taxonomy is a strength, although the diversity of approximately 100 scenes remains substantially narrower than the scale suggested by the total number of generated examples.

HoliWorld-Foundry

Geometric preparation

Foundry uses visual-inertial SLAM to estimate camera poses and sparse geometry from stereo RGB and IMU measurements. Depth Anything 3 supplies dense depth estimates for RGB keyframes, and the observations are consolidated into a 3D Gaussian-splatting representation and a reconstructed point cloud. Dynamic regions are supplemented with per-frame depth completion.

This preparation creates a shared world coordinate system in which both persistent objects and moving objects can be represented. The use of common geometry is essential: image-space displacement alone cannot distinguish object motion from camera motion, whereas world-coordinate trajectories can support motion and temporal reasoning after camera motion has been accounted for.

The pipeline subsequently branches into static and dynamic perception, but both branches retain links to the same observations and geometric reconstruction.

Figure 3: HoliWorld-Foundry reconstructs shared geometry and routes entities to complementary static and dynamic perception pipelines before fusing them into a unified world state.

Static object perception

Static perception combines two complementary proposal mechanisms. Boxer operates on the left and right RGB streams, producing open-vocabulary 2D detections that are lifted into metric 3D boxes using depth and camera poses. TR3D operates on the reconstructed scene point cloud and provides scene-level 3D detections, particularly for objects whose full extents are difficult to observe from individual image views.

The two sources are merged using geometric compatibility checks. Proposals with insufficient point-cloud support are removed, while spatially compatible boxes are consolidated. The method retains a current candidate and, where available, one preselected fallback candidate. A VLM then validates each candidate against its own source observations and supplementary views rendered from the Gaussian scene. Validation can retain, refine, relabel, or reject a candidate; a rejected candidate receives at most one fallback retry.

This design makes the VLM a verification and correction module rather than the sole source of 3D localization. That distinction is methodologically significant. The geometric stages constrain candidate extent and support, while the VLM contributes semantic discrimination and evidence-based rejection. The fallback mechanism also limits repeated VLM search, although the paper does not report the computational cost or failure distribution associated with candidate validation.

Figure 4: Static perception fuses image-based lifting and point-cloud detection, applies geometric filtering, and performs evidence-conditioned VLM verification with a bounded fallback retry.

Dynamic object perception

Dynamic perception begins with interaction grounding. Hand regions and VLM-generated interaction descriptions identify the manipulated object in each interaction segment. Descriptions and keyframe evidence are compared across segments to establish identity continuity when an object disappears and later reappears.

SAM-based mask initialization and propagation then provide frame-level object regions within each segment. Valid depth pixels inside the masks are back-projected into world coordinates, filtered, and fit with oriented 3D bounding boxes. Boxes associated with the same identity are linked over time to produce 4D tracks.

The dynamic formulation relies on an explicit simplifying assumption: at most one moving object is present at each time. This assumption materially constrains the scope of the reported dynamic supervision. It simplifies mask selection and association, but it leaves open whether the method can handle simultaneous manipulation, multiple independently moving objects, or interactions involving object-object motion.

Figure 5: Dynamic perception establishes cross-segment identities, propagates masks, lifts valid depth into world coordinates, and links oriented boxes into 4D tracks.

Experimental results

Static 3D object perception

Foundry achieves the best reported AP at both IoU thresholds on ScanNet, ScanNet++, and HoliWorld-Bench. On ScanNet, AP$_{25}$ is 0.68 and AP$_{50}$ is 0.55, compared with 0.40 and 0.12 for the strongest competing results in the evaluation. On ScanNet++, Foundry reaches 0.30 AP$_{25}$ and 0.13 AP$_{50}$, compared with 0.23 and 0.05. On HoliWorld-Bench, it obtains 0.56 AP$_{25}$ and 0.24 AP$_{50}$.

Dataset	AP$_{25}$	AP$_{50}$	mIoU
ScanNet	0.68	0.55	0.72
ScanNet++	0.30	0.13	0.53
HoliWorld-Bench	0.56	0.24	0.53

The gains at AP$_{50}$ are particularly relevant because they indicate improved geometric alignment under stricter matching, rather than only increased proposal recall. On ScanNet, the reported AP$_{50}$ advantage over the strongest baseline is 0.43; on ScanNet++, it is 0.08. The considerably lower absolute performance on ScanNet++ also demonstrates that the method remains sensitive to scene complexity and domain variation.

Matched-box mIoU should be interpreted jointly with AP. It is conditional on successful matching and therefore does not measure missed objects. The paper explicitly notes that the larger improvement on HoliWorld-Bench is driven partly by better detection coverage rather than only by more accurate extents among already matched boxes.

Figure 6: Qualitative static perception results show object coverage and 3D box alignment across representative scenes.

The ablations support the dual-branch design. Image-based lifting is more effective for small objects and view-specific details, whereas point-cloud detection recovers larger scene-level extents. Stereo fusion further improves complementary coverage. Caption guidance recovers missed entities, while VLM verification changes geometric metrics only modestly but corrects semantic labels and rejects unsuitable candidates.

Figure 7: Qualitative ablations illustrate the complementary coverage supplied by image lifting, point-cloud detection, and their fusion.

The implication is that the principal value of VLM verification is not necessarily a large increase in box overlap. Its more direct contribution is semantic reliability and candidate filtering, which are important for downstream QA generation even when AP and mIoU change little.

Dynamic tracking

Dynamic tracking is evaluated on 20 Aria Digital Twin sequences and 20 HoliWorld-Bench sequences. At a 0.1 m world-coordinate tolerance, Foundry obtains HOTA scores of 54.31% on ADT20 and 43.14% on HoliWorld-Bench. These results compare with 36.00% and 15.13% for the strongest baseline values in the respective evaluations.

Dataset	HOTA	AssA	PCL	IoU	GTC	PP
ADT20	54.31%	71.94%	47.95%	30.88%	43.03%	56.25%
HoliWorld-Bench	43.14%	56.54%	57.89%	41.79%	53.18%	76.49%

The ADT20 result is notable because Foundry exceeds IT3DEgo's 36.00% HOTA despite that baseline receiving ground-truth first-visible mask initialization. On HoliWorld-Bench, the gap is larger: Foundry's HOTA is 43.14%, compared with 15.13% for SAM+LMK, the strongest competing 3D pipeline in the reported comparison. Foundry also achieves the highest association accuracy and trajectory coverage on both datasets.

Figure 8: Predicted and ground-truth trajectories across interaction stages demonstrate the recovery of world-coordinate motion over time.

The comparison requires careful qualification. Several baselines receive annotation-derived inputs, including ground-truth first-visible masks or ground-truth category queries, whereas Foundry uses no annotation-derived test-time inputs. This strengthens the practical comparison, but the input adaptations mean that the systems are not evaluated under identical initialization conditions. The paper reports these differences explicitly and uses common trajectory scoring, but performance should still be interpreted in light of the heterogeneous baseline interfaces.

The dynamic ablations isolate two distinct requirements. Cross-interaction label continuity improves identity association, while SAM-based association improves mask temporal consistency and therefore 3D lifting. Enabling both yields HOTA/PCL values of 54.31%/47.95% on ADT20 and 43.14%/57.89% on HoliWorld-Bench. This supports the view that identity continuity and spatially stable masks address different failure modes rather than providing redundant mechanisms.

Figure 9: Qualitative comparisons show the relative trajectory quality and identity continuity of the evaluated tracking methods.

Downstream VLM reasoning

The downstream experiment fine-tunes Qwen3.5-4B on HoliWorld-Data to obtain HoliWorld-VL-4B. On HoliWorld-Bench, Token F1 increases from 16.35% for the original Qwen3.5-4B to 78.02% after training. On EgoDynamic4D, it increases from 18.29% to 66.39%.

Model	HoliWorld-Bench F1	Y/N	MCQ	Num.	EgoDynamic4D F1	Y/N	MCQ	Num.
Qwen3.5-4B	16.35	34.86	62.00	10.80	18.29	59.47	79.16	0.02
Qwen3.8-27B	43.67	82.85	75.00	66.13	42.93	61.07	57.54	0.16
HoliWorld-VL-4B	78.02	93.52	94.00	81.84	66.39	99.67	94.92	16.33

HoliWorld-VL-4B also surpasses zero-shot Qwen3.8-27B on all reported metrics across both benchmarks, despite using a substantially smaller parameter count. The result is strongest for HoliWorld-Bench, where Token F1 improves by 61.67 percentage points over the untuned 4B model. The improvement on EgoDynamic4D is 48.10 points, suggesting transfer beyond the training benchmark.

These results provide evidence that the generated records contain usable spatial and temporal signal. They do not, however, isolate the contributions of individual supervision sources. The reported fine-tuning experiment combines static grounding, dynamic trajectories, captions, interaction semantics, and QA formats. Consequently, it establishes the aggregate utility of HoliWorld-Data but does not determine whether the gains arise primarily from geometric supervision, temporal identity tracking, synthetic QA diversity, or the increased quantity of task-specific examples.

Figure 10: The QA taxonomy spans static mapping, dynamic trajectory reasoning, interaction descriptions, state transitions, and temporal grounding.

Limitations and open questions

The principal limitation is scale and coverage. Although the dataset contains approximately 80,000 QA pairs, the underlying recordings comprise roughly 100 scenes and are collected through a particular capture setup. The number of training examples therefore exceeds the number of independent environments by several orders of magnitude, and the degree to which the learned model generalizes to unseen domains, users, camera configurations, and interaction patterns is not established.

The dynamic pipeline assumes at most one moving object at each time. This assumption excludes multi-object manipulation and concurrent motion, precisely the settings in which identity association and interaction reasoning become more difficult. The method also depends on VLM-generated interaction descriptions and on depth completion in dynamic regions; systematic errors in either component can propagate into identities, 3D boxes, tracks, and QA answers.

The evaluation protocol has additional constraints. Static evaluation uses 20 sequences from each benchmark, while dynamic evaluation uses 20 sequences from ADT and 20 from HoliWorld-Bench. Several baseline systems receive privileged initialization or queries, and the baselines do not all expose the same output types. Although the paper documents these adaptations and applies common trajectory metrics, larger and more homogeneous comparisons would be needed to separate algorithmic gains from interface differences.

Finally, downstream improvements may partly reflect distributional alignment between HoliWorld-Data and HoliWorld-Bench. HoliWorld-Bench is independently annotated and held out at the recording level, which reduces direct annotation leakage, but both resources originate from the same ecosystem and capture process. The cross-dataset gains on EgoDynamic4D are encouraging, yet the broader generality of the learned representation remains an open empirical question.

Conclusion

HoliWorld presents an integrated pipeline for converting egocentric recordings into evidence-linked world records and multimodal supervision. Its technical contribution is the coupling of shared scene geometry, static 3D object perception, dynamic identity-aware tracking, interaction semantics, and QA construction within a single representation.

The reported results show substantial gains in static detection, dynamic tracking, and spatial-temporal VLM reasoning. Foundry reaches 0.55 AP$_{50}$0 on ScanNet, 54.31% HOTA on ADT20, and 43.14% HOTA on HoliWorld-Bench, while HoliWorld-VL-4B attains Token F1 scores of 78.02% and 66.39% on HoliWorld-Bench and EgoDynamic4D, respectively. These findings support the paper's narrower and technically concrete claim: automated, evidence-linked supervision extracted from human-egocentric experience can improve both object-centric world annotation and downstream spatial-temporal reasoning, subject to the current constraints on scene diversity, dynamic complexity, and evaluation scale.