---
title: '3DGS-VBench: Compression VQA Benchmark'
url: https://www.emergentmind.com/topics/3dgs-vbench
type: topic
---

# 3DGS-VBench: Compression VQA Benchmark

3DGS-VBench is a large-scale Video Quality Assessment (VQA) dataset and benchmark for compressed 3D Gaussian Splatting (3DGS) content. Introduced in "3DGS-VBench: A Comprehensive Video Quality Evaluation Benchmark for 3DGS Compression" [2508.07038], it contains 660 compressed 3DGS models and corresponding video sequences generated from 11 scenes across 6 state-of-the-art 3DGS compression algorithms with systematically designed parameter levels. The benchmark couples these assets with Mean Opinion Score (MOS) annotations and a comparative evaluation of 15 objective quality metrics. Its motivating observation is that 3DGS compression introduces distortions such as geometry thinning, color quantization artifacts, anchor-level sampling changes, and rate–distortion trade-offs that alter both spatial structure and temporal coherence along camera paths, and that such degradations are not well captured by conventional raster-domain IQA/VQA practice [2508.07038].

## 1. Scope and benchmark identity

3D Gaussian Splatting is treated in the benchmark as an explicit 3D scene representation enabling real-time novel view synthesis with high visual fidelity while imposing a substantial storage footprint. The paper gives representative original-model sizes in the benchmark of 795.1323 MB for Mip-NeRF360, 434.3680 MB for Tanks & Temples, and 665.9170 MB for Deep Blending, which motivates compression-oriented evaluation [2508.07038].

The benchmark is specifically about video quality evaluation for **compressed** 3DGS, rather than a universal benchmark for every 3DGS task. A common source of confusion is terminological: several contemporaneous papers discuss a broader "3DGS-VBench"-style ecosystem covering XR scene creation, authenticity assessment, adverse-weather robustness, compression toolkits, or dynamic synchronized multi-view NVS, but they do not introduce a benchmark with that exact name. The benchmark explicitly titled **3DGS-VBench** is the compression-oriented VQA dataset of [2508.07038], whereas the broader uses are programmatic or hypothetical extensions in adjacent work [2501.09302] [2604.27590] [2512.24742] [2510.17719] [2605.12437].

The paper’s main contributions are threefold. First, it establishes a large-scale VQA dataset tailored to 3DGS compression. Second, it benchmarks 6 compression methods on storage efficiency and visual quality. Third, it evaluates 15 IQA/VQA metrics on 3DGS content and argues that existing metrics, especially traditional full-reference image metrics and standard no-reference IQA baselines, are often poorly aligned with human judgments on this distortion family [2508.07038].

## 2. Dataset construction and rendering protocol

The dataset comprises 11 scenes from four multiview datasets: **Mip-NeRF360** with 6 scenes, **Tanks and Temples** with 2 scenes, **Deep Blending** with 2 scenes, and **PKU-DyMVHumans** with 1 scene [2508.07038]. The Mip-NeRF360 scenes are bicycle at 1237×822, flowers at 1256×828, garden at 1297×840, counter at 1558×1038, kitchen at 1558×1039, and room at 1557×1038. Tanks and Temples contributes train at 980×545 and truck at 979×546. Deep Blending contributes playroom at 1264×832 and drjohnson at 1332×876. PKU-DyMVHumans contributes Dance_Dunhuang_Pair at 1600×876.

The 660 models arise from applying six compression algorithms to these scenes under systematically varied compression settings. Each scene’s trained 3DGS point cloud is imported into Blender, and virtual cameras are assigned calibrated focal length, position, orientation, and resolution matching the original dataset. For each processed visual sequence, 600 continuous viewpoints are arranged on a 360° orbital path with uniform angular spacing. Rendering produces 600 frames per sequence, which are encoded as 20-second videos at 30 fps using FFMPEG `libx265` with constant rate factor \( \mathrm{CRF} = 10 \) [2508.07038]. The paper does not report special temporal stabilization; temporal consistency follows from the continuous orbital camera path and uniform sampling.

The six compression methods and their distortion-level designs are as follows:

| Method | Controlled parameters | DL per scene |
|---|---|---:|
| Compact-3DGS | `hashmap`; `(codebook, rvq_depth)`; `lambda` | 13 |
| CompGS | `g-size`; `c-size`; `reg` | 13 |
| c3dgs | `(c-size, c-include)`; `(g-size, g-include)` | 9 |
| LightGaussian | `prune`; `(c-ratio, c-size)` | 9 |
| Scaffold-GS | `vsize` | 8 |
| HAC | `lambda` | 8 |

For **Compact-3DGS**, the `hashmap` compression levels are \(2^1\), \(2^2\), \(2^7\), \(2^{12}\), with default \(2^{19}\); the \((\text{codebook}, \text{rvq\_depth})\) levels are \((2^3,1)\), \((2^4,1)\), \((2^6,1)\), \((2^3,6)\), with default \((2^6,6)\); and `lambda` levels are 0.014, 0.012, 0.010, 0.006, with default 0.0005. For **CompGS**, `g-size` levels are 1, 2, \(2^3\), \(2^5\), with default \(2^{12}\); `c-size` levels are 1, 2, \(2^2\), \(2^3\), with default \(2^{12}\); and `reg` levels are \(4\times10^{-6}\), \(3\times10^{-6}\), \(2\times10^{-6}\), \(1\times10^{-6}\), with default \(1\times10^{-7}\) [2508.07038].

For **c3dgs**, the \((c\text{-size}, c\text{-include})\) levels are \((1,0.6)\), \((2,0.6)\), \((2^2,0.6)\), \((2^5,0.6)\), with default \((2^{12}, 0.6\times10^{-6})\); the \((g\text{-size}, g\text{-include})\) levels are \((1,0.3)\), \((2^3,0.3)\), \((2^5,0.3)\), \((2^8,0.3)\), with default \((2^{12}, 0.3\times10^{-5})\). For **LightGaussian**, `prune` levels are 0.97, 0.95, 0.90, 0.85, with default 0.66; the \((c\text{-ratio}, c\text{-size})\) levels are \((1,2)\), \((1,2^2)\), \((1,2^3)\), \((1,2^5)\), with default \((1,2^{13})\). For **Scaffold-GS**, `vsize` levels are 0.350, 0.250, 0.200, 0.160, 0.120, 0.080, 0.050, with default 0.001. For **HAC**, `lambda` levels are 0.800, 0.600, 0.400, 0.300, 0.200, 0.120, 0.060, with default 0.004. The total model count is therefore \(11 \times (13 + 13 + 9 + 9 + 8 + 8) = 660\) [2508.07038].

The dataset is hosted at <https://github.com/YukeXing/3DGS-VBench>. The paper does **not** specify a license, file structure or naming convention, or recommended train/val/test splits [2508.07038].

## 3. Subjective annotation protocol and reliability

Subjective evaluation follows an 11-level impairment scale per ITU-T P.910. The display device is a 27-inch AOC Q2790PQ used in an indoor laboratory under standard lighting conditions, and playback is provided through a Python Tkinter interface. To manage fatigue, the 660 processed visual sequences are randomly divided into 8 groups [2508.07038].

The paper contains a notable numerical discrepancy in the description of participants. The abstract states that annotations were obtained from **50 participants**, whereas the subjective experiment section reports **15 annotators × 660 videos = 9,900 annotations**. The benchmark nevertheless reports 660 MOS values after outlier handling, and the score rejection rate is given as 2% following ITU recommendations [2508.07038]. This is best read as a reporting inconsistency rather than a methodological claim about two distinct evaluation rounds.

For a processed visual sequence with valid subject scores \(s_i\) and \(N\) valid ratings after rejection, MOS is defined as

\[
\mathrm{MOS} = \frac{1}{N}\sum_{i=1}^{N} s_i.
\]

The paper states that subject rejection and outlier detection follow ITU guidance, citing ITU-T P.910 and ITU-R BT.500 methodologies, but it does not detail the exact thresholds or criteria used [2508.07038].

The benchmark reports several reliability indicators. Diversity is validated through an SI/TI scatter over the 11 scenes, intended to show coverage of spatial and temporal complexity. The MOS distribution spans 0–10 and is described as approximately Gaussian centered at about 6, with sufficient samples in the low-quality range \(0\text{–}2\), supporting robust training coverage. Inter-rater agreement, confidence intervals, and formal statistical tests are not reported [2508.07038].

## 4. Compression methods and empirical trade-offs

The six benchmarked algorithms span several families of 3DGS compression. **Compact-3DGS** combines learnable volume masking through `lambda`, residual vector quantization for geometry through codebook size and RVQ depth, and hash-grid neural fields replacing spherical harmonics through the `hashmap` parameter. **CompGS** uses K-means vector quantization over Gaussian parameters with geometry and color codebooks and an opacity regularization term `reg`. **c3dgs** combines K-means codebooks with entropy encoding and uses color and geometry codebook sizes together with importance parameters. **LightGaussian** prunes low-significance Gaussians, distills spherical harmonics to lower degrees, and vector-quantizes spherical harmonics. **Scaffold-GS** is anchor-based and controlled by `vsize`. **HAC** uses hash-grid assisted context-aware entropy compression with adaptive quantization through `lambda` [2508.07038].

The benchmark provides quantitative comparisons at default optimal parameters. For the uncompressed reference, the reported averages are: Mip-NeRF360 with PSNR 27.6655, SSIM 0.9141, LPIPS 0.1267, and size 795.1323 MB; Tanks & Temples with PSNR 23.7579, SSIM 0.89339, LPIPS 0.09559, and size 434.3680 MB; and Deep Blending with PSNR 29.6219, SSIM 0.9269, LPIPS 0.1016, and size 665.9170 MB [2508.07038].

Among compressed methods, **HAC** is reported as offering the best quality–efficiency balance. Its default results are 27.5911 PSNR, 0.9112 SSIM, 0.1364 LPIPS, 8.7413 MOS, and 15.8625 MB on Mip-NeRF360; 24.3093 PSNR, 0.9014 SSIM, 0.0928 LPIPS, 8.1786 MOS, and 14.5386 MB on Tanks & Temples; and 30.3207 PSNR, 0.9342 SSIM, 0.0975 LPIPS, 8.8214 MOS, and 7.7961 MB on Deep Blending [2508.07038]. The paper characterizes HAC as achieving approximately \(50\times\) to \(85\times\) compression with competitive quality, including less than 0.3 dB loss versus Scaffold-GS in some cases.

**CompGS** is presented as a balanced trade-off, with 27.1742 PSNR, 0.9042 SSIM, 0.1468 LPIPS, 8.3484 MOS, and 21.9188 MB on Mip-NeRF360; 23.2633 PSNR, 0.8845 SSIM, 0.1132 LPIPS, 7.5524 MOS, and 13.6345 MB on Tanks & Temples; and 30.0751 PSNR, 0.9338 SSIM, 0.0991 LPIPS, 9.0238 MOS, and 14.7798 MB on Deep Blending [2508.07038]. The text summarizes this regime as approximately \(15\times\) to \(45\times\) compression with less than 1 dB quality loss.

**c3dgs** and **Compact-3DGS** occupy intermediate positions. c3dgs reports 28.6562 MB, 17.6700 MB, and 23.8754 MB on Mip-NeRF360, Tanks & Temples, and Deep Blending respectively, with MOS values 8.2960, 7.5619, and 8.6786. Compact-3DGS reports 48.7238 MB, 39.7017 MB, and 43.3126 MB respectively, with MOS values 8.1186, 6.9000, and 8.6786 [2508.07038].

**LightGaussian** shows a different trade-off. On Mip-NeRF360 it records 25.4406 PSNR, 0.8582 SSIM, 0.2009 LPIPS, 8.1333 MOS, and 52.0151 MB; on Tanks & Temples, 22.6449 PSNR, 0.8565 SSIM, 0.1637 LPIPS, 7.4190 MOS, and 28.6682 MB; and on Deep Blending, 25.9883 PSNR, 0.8447 SSIM, 0.1950 LPIPS, 7.9333 MOS, and 43.3730 MB [2508.07038]. The paper describes this pattern as larger quality degradation, approximately 2–4 dB PSNR loss, for moderate compression of about \(12\times\) to \(15\times\).

For **Scaffold-GS**, the text states that it yields the highest or near-highest quality but relatively modest compression, approximately \(4.8\times\) to \(5.9\times\), making it suitable for quality-critical scenarios. The paper also notes that the corresponding table row is incomplete, although the text cites Mip-NeRF360 PSNR 27.8406 as an example of its superior visual quality [2508.07038].

## 5. Objective quality metrics and correlation with human opinion

The benchmark evaluates 15 metrics across three broad paradigms: full-reference IQA, no-reference IQA, and deep no-reference VQA. The full-reference set includes PSNR, SSIM, MS-SSIM, IW-SSIM, VIF, FSIM, LPIPS, and DISTS. The no-reference IQA set includes BRISQUE and CLIP-IQA. The deep no-reference VQA set includes DOVER, FAST-VQA, simpleVQA, VSFA, and Q-Align [2508.07038].

The paper reports standard formulations for several core measures. PSNR is written as

\[
\mathrm{PSNR} = 10 \log_{10}\left(\frac{\mathrm{MAX}^2}{\mathrm{MSE}}\right),
\]

and SSIM as

\[
\mathrm{SSIM}(x,y)=\frac{(2\mu_x\mu_y + C_1)(2\sigma_{xy} + C_2)}{(\mu_x^2 + \mu_y^2 + C_1)(\sigma_x^2 + \sigma_y^2 + C_2)}.
\]

The Spearman rank correlation is

\[
r_s = 1 - \frac{6\sum_{i=1}^{n} d_i^2}{n(n^2-1)},
\]

the Pearson linear correlation is

\[
r_p = \frac{\sum_i (p_i-\bar p)(m_i-\bar m)}{\sqrt{\sum_i (p_i-\bar p)^2}\sqrt{\sum_i (m_i-\bar m)^2}},
\]

and Kendall’s tau is

\[
\tau = \frac{n_c - n_d}{\frac{1}{2}n(n-1)}.
\]

The paper does not report applying a non-linear mapping before PLCC or RMSE calculation [2508.07038].

The reported correlations are:

| Metric | SRCC / PLCC / KRCC | Category |
|---|---|---|
| PSNR | 0.5022 / 0.4976 / 0.3560 | FR-IQA |
| SSIM | 0.5108 / 0.4758 / 0.3684 | FR-IQA |
| LPIPS | 0.5106 / 0.4581 / 0.3619 | FR-IQA |
| DISTS | 0.7317 / 0.7146 / 0.5269 | FR-IQA |
| VIF | 0.4220 / 0.4252 / 0.2917 | FR-IQA |
| FSIM | 0.5866 / 0.5815 / 0.4311 | FR-IQA |
| IW-SSIM | 0.6309 / 0.5962 / 0.4488 | FR-IQA |
| MS-SSIM | 0.5028 / 0.4700 / 0.3602 | FR-IQA |
| CLIP-IQA | 0.3913 / 0.2738 / 0.3216 | NR-IQA |
| BRISQUE | 0.2379 / 0.1819 / 0.1749 | NR-IQA |
| DOVER | 0.9409 / 0.9308 / 0.7901 | Deep NR-VQA |
| FAST-VQA | 0.9314 / 0.9255 / 0.7753 | Deep NR-VQA |
| simpleVQA | 0.9350 / 0.7813 / 0.9314 | Deep NR-VQA |
| VSFA | 0.9392 / 0.9345 / 0.7908 | Deep NR-VQA |
| Q-Align | 0.8485 / 0.6489 / 0.8126 | Deep NR-VQA |

The benchmark’s central empirical conclusion is that deep no-reference VQA models substantially outperform traditional full-reference IQA and standard no-reference IQA methods on compressed 3DGS videos. **DOVER** gives the strongest overall alignment with human opinion, with SRCC 0.9409, while **DISTS** is the strongest full-reference metric at SRCC 0.7317. By contrast, classic full-reference metrics such as SSIM and PSNR are both near 0.51 SRCC, and BRISQUE is particularly weak at 0.2379 [2508.07038]. The paper attributes this gap to a mismatch between conventional pixel-domain assumptions and the structural, spatio-temporal artifacts introduced by generative 3DGS compression.

## 6. Position in the broader 3DGS benchmark landscape

3DGS-VBench occupies a specific niche: human-centered video-quality evaluation for compressed 3DGS along continuous camera paths. It is not a geometry benchmark, an authenticity benchmark, an adverse-condition robustness benchmark, or a dynamic multi-view standardized dataset. Other papers cover those neighboring problem classes with different emphases. **Splatwizard** is a unified toolkit for 3DGS compression benchmarking with PSNR, SSIM, LPIPS, Chamfer distance, FPS, and resource profiling, but it explicitly states that it does not mention any benchmark named "3DGS-VBench" [2512.24742]. **Fake3DGS** proposes an authenticity benchmark for 3DGS manipulation detection and suggests that it could serve an authenticity track within a comprehensive "3DGS-VBench" [2604.27590]. The XR study in "Creating Virtual Environments with 3D Gaussian Splatting: A Comparative Study" recommends axes such as fidelity, efficiency, robustness, editability/interactivity, and scalability for a potential benchmark structure [2501.09302]. **RaindropGS** proposes a domain-specific track for unconstrained raindrop-corrupted inputs [2510.17719], while the standardized dynamic multi-view benchmark built around TA-3DGS addresses synchronized dynamic-scene NVS rather than compression VQA [2605.12437].

This suggests that 3DGS-VBench is best understood as a **specialized compression-VQA component** within a larger potential benchmarking ecosystem. Its contribution is unusually specific: it measures perceptual consequences of 3DGS compression in video form, rather than attempting to unify all evaluation dimensions of 3DGS research.

The paper also states several limitations. Scene diversity is restricted to 11 scenes, with only one dynamic human figure scene. The rendering protocol uses a uniform 360° orbital camera path and a single encoding setting, `libx265` with `CRF=10`. The paper does not specify a license, file structures, or recommended data splits. It does not introduce a new VQA architecture or prescribe official training hyperparameters for specialized models, although it argues that the MOS-labeled videos can support such training [2508.07038].

Within those limits, the benchmark has two broader implications. First, it shows that strong human-aligned quality estimation for compressed 3DGS is currently better served by deep no-reference video models than by traditional full-reference image metrics. Second, it provides a common protocol for comparing storage reduction against perceptual degradation across six widely used compression families. The paper further notes that current 3DGS generation often uses L1 and SSIM losses, and suggests that adopting better perceptual losses, citing DISTS and related work, may improve generation results for compression-aware training [2508.07038].

Source: https://www.emergentmind.com/topics/3dgs-vbench