---
title: 'HolyScenes: 3D Landmark Benchmark'
url: https://www.emergentmind.com/topics/holyscenes-dataset
type: topic
---

# HolyScenes: 3D Landmark Benchmark

HolyScenes is a large-scale, internet-sourced 3D benchmark dataset constructed to evaluate open-vocabulary semantic understanding of outdoor architectural landmarks under real-world conditions. Introduced in HaLo-NeRF (Dudai et al. 2024), and utilized as the primary evaluation benchmark in Lang3D-XL [2512.07807], it provides challenging testbeds for cross-modal, 3D-consistent semantic segmentation and feature distillation methods at city-block scale.

## 1. Dataset Construction and Scene Composition

HolyScenes is assembled from internet photo collections of six significant religious and architectural landmarks:

- Milano Cathedral
- St. Paul’s Cathedral
- Blue Mosque
- Badshahi Mosque
- Notre-Dame Cathedral
- Hurba (a smaller site)

Each scene reconstruction uses approximately 2,000+ images, primarily outdoor photographs spanning diverse lighting, weather conditions, crowd densities, and camera viewpoints. Scene reconstructions cover spatial extents up to 25,600 m² (Badshahi Mosque). Images typically range from 1,000 to 2,000 pixels on the long axis and are downsampled to ½ or ¼ resolution for training. Final scene geometry is represented using 3D Gaussian Splatting, with the number of Gaussians ranging from 30,000 to 360,000 depending on scene scale.

## 2. Annotation Protocol and Semantic Categories

For semantic evaluation, HolyScenes designates six core architectural classes:

- Windows
- Minarets (or, contextually, spires)
- Domes
- Towers
- Spires (if distinct from minarets)
- Portals (encompassing doors, gateways, entrances)

Annotations are provided as binary masks for each category on held-out views, facilitating open-vocabulary protocols where each text prompt is mapped to a class. To increase linguistic coverage and retrieval robustness, category-denoting prompts are automatically expanded with common synonyms via an LLM (e.g., "portals → doors, gateways, entrances, portals"). There are no hierarchical, bounding-box, or relational annotations; objects are labeled independently across categories.

## 3. Data Splits, Preprocessing, and Augmentation

### Splitting and Validation

- **Training:** All collected images for a scene contribute to 3D reconstruction and feature distillation.
- **Testing:** Conducted on held-out test masks (∼100–200 images per class as in HaLo-NeRF). These masks are strictly category-specific and unseen during training.
- **Validation:** A supplementary set of 52 images (approx. 8–11 per scene) maximizes inter-category overlap for novel-view generalization checks.
- **Cross-scene/domain:** No scene is held out wholesale; evaluations always use unseen masks/views within the trained scenes.

### Preprocessing Pipeline

- **Geometric registration:** COLMAP is applied for camera pose recovery and sparse point triangulation. Outlier points near image boundaries are discarded.
- **Semantic feature preprocessing:** A physical-scale CLIP pyramid is calculated per image by converting focal lengths and scene distances into meter-based patch sizes. Resulting crops are used for CLIP-based feature extraction.
- **Feature map downsampling:** During feature training, rendered 3-channel (3×H×W) bottleneck feature maps are randomly resized to 80–120 pixels per axis before hashing to conserve memory.
- **Loss focusing:** LangSAM-generated building masks modulate loss functions, directing learning toward architectural content.

## 4. Evaluation Task and Protocols

The core evaluation task is open-vocabulary 3D semantic segmentation: given any text prompt (e.g., "minaret"), the model must output a probabilistic 3D or rendered 2D mask indicating regions relevant to the queried class.

### Metrics

- **Average Precision (AP):** Calculated per class using the standard object detection paradigm by thresholding model-reported relevancy maps against ground-truth masks.
- **Mean Average Precision (mAP):** Defined as $ \text{mAP} = \frac{1}{C} \sum_{c=1}^C AP_c $, where $C$ is the number of categories.
- **Reconstruction metrics:** For ablations, L₁ and PSNR are reported on held-out RGB images, as well as end-to-end training/inference times and GPU memory usage.

### Baseline Methods

| Method Category             | Models Evaluated                                         | Typical Inference Speed |
|-----------------------------|----------------------------------------------------------|------------------------|
| 2D-only segmentation        | LSeg, ToB, CLIPSeg (pretrained & FT), LangSAM            | Fast                   |
| 3D (slow)                   | HaLo-NeRF                                                | ∼2 h/prompt            |
| 3D feature distillation     | DFF, LERF, LangSplat, Feature3DGS, FMGS, Lang3D-XL       | Sub-second             |

All baselines are evaluated using the same set of held-out test masks.

## 5. Semantic Integration in 3D with Lang3D-XL

In Lang3D-XL, HolyScenes is used to train and evaluate language-embedded 3D Gaussians for open-vocabulary segmentation with the following workflow:

1. **3D Gaussian Splatting:** Each scene is reconstructed, with each Gaussian $G_i$ parameterized by mean $\mu_i$, covariance $\Sigma_i$, opacity $\alpha_i$, spherical-harmonic color, and an auxiliary learnable feature vector $f_i \in \mathbb{R}^{d'}$ with $d' = 3$.
2. **Rendering to bottleneck:** Differentiable rasterization produces per-view, low-dimensional ($3 \times H \times W$) "semantic bottleneck" feature maps $\mathbf{M}$.
3. **Hash encoding:** Each per-pixel bottleneck vector $m_{u,v} \in \mathbb{R}^3$ is used to index a multi-resolution hash table $H$, generating embeddings at $L$ levels which are concatenated as $E_{u,v} \in \mathbb{R}^{D'}$.
4. **MLP-based semantic decoding:** $E_{u,v}$ is input to a shared MLP $\mathcal{G}$, which predicts concatenated CLIP (512-D) and DINOv2 (384-D) features $\hat S_{u,v}$.
5. **Regularization and Losses:** The full objective is
   $$
   L = L_\text{rec} + \lambda_\text{CLIP} L_\text{CLIP} + \lambda_\text{DINO} L_\text{DINO} + \lambda_\text{SAM} L_\text{SAM}
   $$
   - $L_\text{rec}$: 3DGS RGB + D-SSIM
   - $L_\text{CLIP}$: $L_1$ on downsampled predicted CLIP features
   - $L_\text{DINO}$: $L_1$ on DINOv2 features; shared MLP weights regularize CLIP/DINO coupling
   - $L_\text{SAM}$: object-wise feature variance over LangSAM masks
   - **Attenuated Downsampler:** To reconcile high-to-low resolution features, the Attenuated Downsampler learns attention weights $w_i$ over local windows $S$, combining features as $F_d = (\sum_{i \in S} e^{w_i} f_i ) / (\sum_{i \in S} e^{w_i})$.

## 6. Quantitative Performance and Benchmarking

HolyScenes supports robust benchmarking for both accuracy and efficiency. The mAPs reported in Lang3D-XL [2512.07807] are:

| Model/Method      | mAP Score | Inference Time (per prompt) |
|-------------------|-----------|-----------------------------|
| CLIPSeg_FT (2D)   | 0.66      | Fast                        |
| HaLo-NeRF         | 0.68      | ∼2 h                        |
| Lang3D-XL         | 0.59      | <0.1 s                      |
| FMGS (fast 3D)    | 0.51      | Fast                        |

Lang3D-XL’s per-category AP: Windows 0.51, Minarets 0.84, Domes 0.70, Towers 0.46, Spires 0.67, Portals 0.38.

## 7. Significance and Application Scope

HolyScenes was designed to stress-test feature-distillation and language–3D fusion methods at real-world scale, providing a challenging, standardized testbed for future work in 3D open-vocabulary segmentation, scene retrieval, and semantic navigation. Its use of in-the-wild internet photo collections, open-vocabulary protocol, and large-scale, metric-accurate reconstructions distinguishes it from prior 3D semantic datasets. The evaluation infrastructure, consistent with HaLo-NeRF, Lang3D-XL, and related works, enables reproducibility and rigorous cross-method comparison in both accuracy and system efficiency [2512.07807].

Source: https://www.emergentmind.com/topics/holyscenes-dataset