HolyScenes: 3D Landmark Benchmark

• HolyScenes is a large-scale, internet-sourced 3D benchmark dataset for evaluating open-vocabulary semantic segmentation of outdoor architectural landmarks.
• It employs detailed 3D reconstruction with 3D Gaussian Splatting and diverse semantic annotations from real-world images to capture varying conditions.
• Benchmark evaluations report mAP scores and inference times, supporting rigorous cross-method comparisons in language–3D feature distillation research.

HolyScenes is a large-scale, internet-sourced 3D benchmark dataset constructed to evaluate open-vocabulary semantic understanding of outdoor architectural landmarks under real-world conditions. Introduced in HaLo-NeRF (Dudai et al. 2024), and utilized as the primary evaluation benchmark in Lang3D-XL (Krakovsky et al., 8 Dec 2025), it provides challenging testbeds for cross-modal, 3D-consistent semantic segmentation and feature distillation methods at city-block scale.

1. Dataset Construction and Scene Composition

HolyScenes is assembled from internet photo collections of six significant religious and architectural landmarks:

• Milano Cathedral
• St. Paul’s Cathedral
• Blue Mosque
• Badshahi Mosque
• Notre-Dame Cathedral
• Hurba (a smaller site)

Each scene reconstruction uses approximately 2,000+ images, primarily outdoor photographs spanning diverse lighting, weather conditions, crowd densities, and camera viewpoints. Scene reconstructions cover spatial extents up to 25,600 m² (Badshahi Mosque). Images typically range from 1,000 to 2,000 pixels on the long axis and are downsampled to ½ or ¼ resolution for training. Final scene geometry is represented using 3D Gaussian Splatting, with the number of Gaussians ranging from 30,000 to 360,000 depending on scene scale.

2. Annotation Protocol and Semantic Categories

For semantic evaluation, HolyScenes designates six core architectural classes:

• Windows
• Minarets (or, contextually, spires)
• Domes
• Towers
• Spires (if distinct from minarets)
• Portals (encompassing doors, gateways, entrances)

Annotations are provided as binary masks for each category on held-out views, facilitating open-vocabulary protocols where each text prompt is mapped to a class. To increase linguistic coverage and retrieval robustness, category-denoting prompts are automatically expanded with common synonyms via an LLM (e.g., "portals → doors, gateways, entrances, portals"). There are no hierarchical, bounding-box, or relational annotations; objects are labeled independently across categories.

3. Data Splits, Preprocessing, and Augmentation

Splitting and Validation

• Training: All collected images for a scene contribute to 3D reconstruction and feature distillation.
• Testing: Conducted on held-out test masks (∼100–200 images per class as in HaLo-NeRF). These masks are strictly category-specific and unseen during training.
• Validation: A supplementary set of 52 images (approx. 8–11 per scene) maximizes inter-category overlap for novel-view generalization checks.
• Cross-scene/domain: No scene is held out wholesale; evaluations always use unseen masks/views within the trained scenes.

Preprocessing Pipeline

• Geometric registration: COLMAP is applied for camera pose recovery and sparse point triangulation. Outlier points near image boundaries are discarded.
• Semantic feature preprocessing: A physical-scale CLIP pyramid is calculated per image by converting focal lengths and scene distances into meter-based patch sizes. Resulting crops are used for CLIP-based feature extraction.
• Feature map downsampling: During feature training, rendered 3-channel (3×H×W) bottleneck feature maps are randomly resized to 80–120 pixels per axis before hashing to conserve memory.
• Loss focusing: LangSAM-generated building masks modulate loss functions, directing learning toward architectural content.

4. Evaluation Task and Protocols

The core evaluation task is open-vocabulary 3D semantic segmentation: given any text prompt (e.g., "minaret"), the model must output a probabilistic 3D or rendered 2D mask indicating regions relevant to the queried class.

Metrics

• Average Precision (AP): Calculated per class using the standard object detection paradigm by thresholding model-reported relevancy maps against ground-truth masks.
• Mean Average Precision (mAP): Defined as $\text{mAP} = \frac{1}{C} \sum_{c=1}^C AP_c$, where $C$ is the number of categories.
• Reconstruction metrics: For ablations, L₁ and PSNR are reported on held-out RGB images, as well as end-to-end training/inference times and GPU memory usage.

Baseline Methods

Method Category	Models Evaluated	Typical Inference Speed
2D-only segmentation	LSeg, ToB, CLIPSeg (pretrained & FT), LangSAM	Fast
3D (slow)	HaLo-NeRF	∼2 h/prompt
3D feature distillation	DFF, LERF, LangSplat, Feature3DGS, FMGS, Lang3D-XL	Sub-second

All baselines are evaluated using the same set of held-out test masks.

5. Semantic Integration in 3D with Lang3D-XL

In Lang3D-XL, HolyScenes is used to train and evaluate language-embedded 3D Gaussians for open-vocabulary segmentation with the following workflow:

1. 3D Gaussian Splatting: Each scene is reconstructed, with each Gaussian $G_i$ parameterized by mean $\mu_i$, covariance $\Sigma_i$, opacity $\alpha_i$, spherical-harmonic color, and an auxiliary learnable feature vector $f_i \in \mathbb{R}^{d'}$ with $d' = 3$.
2. Rendering to bottleneck: Differentiable rasterization produces per-view, low-dimensional ($3 \times H \times W$) "semantic bottleneck" feature maps $\mathbf{M}$.
3. Hash encoding: Each per-pixel bottleneck vector $m_{u,v} \in \mathbb{R}^3$ is used to index a multi-resolution hash table $H$, generating embeddings at $L$ levels which are concatenated as $E_{u,v} \in \mathbb{R}^{D'}$.
4. MLP-based semantic decoding: $E_{u,v}$ is input to a shared MLP $\mathcal{G}$, which predicts concatenated CLIP (512-D) and DINOv2 (384-D) features $\hat S_{u,v}$.
5. Regularization and Losses: The full objective is

$$L = L_\text{rec} + \lambda_\text{CLIP} L_\text{CLIP} + \lambda_\text{DINO} L_\text{DINO} + \lambda_\text{SAM} L_\text{SAM}$$

• $L_\text{rec}$: 3DGS RGB + D-SSIM
• $L_\text{CLIP}$: $L_1$ on downsampled predicted CLIP features
• $L_\text{DINO}$: $L_1$ on DINOv2 features; shared MLP weights regularize CLIP/DINO coupling
• $L_\text{SAM}$: object-wise feature variance over LangSAM masks
• Attenuated Downsampler: To reconcile high-to-low resolution features, the Attenuated Downsampler learns attention weights $w_i$ over local windows $S$, combining features as $$F_d = (\sum_{i \in S} e^{w_i} f_i ) / (\sum_{i \in S} e^{w_i})$$.

6. Quantitative Performance and Benchmarking

HolyScenes supports robust benchmarking for both accuracy and efficiency. The mAPs reported in Lang3D-XL (Krakovsky et al., 8 Dec 2025) are:

Model/Method	mAP Score	Inference Time (per prompt)
CLIPSeg_FT (2D)	0.66	Fast
HaLo-NeRF	0.68	∼2 h
Lang3D-XL	0.59	<0.1 s
FMGS (fast 3D)	0.51	Fast

Lang3D-XL’s per-category AP: Windows 0.51, Minarets 0.84, Domes 0.70, Towers 0.46, Spires 0.67, Portals 0.38.

7. Significance and Application Scope

HolyScenes was designed to stress-test feature-distillation and language–3D fusion methods at real-world scale, providing a challenging, standardized testbed for future work in 3D open-vocabulary segmentation, scene retrieval, and semantic navigation. Its use of in-the-wild internet photo collections, open-vocabulary protocol, and large-scale, metric-accurate reconstructions distinguishes it from prior 3D semantic datasets. The evaluation infrastructure, consistent with HaLo-NeRF, Lang3D-XL, and related works, enables reproducibility and rigorous cross-method comparison in both accuracy and system efficiency (Krakovsky et al., 8 Dec 2025).