3DGS-VBench: Compression VQA Benchmark

• The paper establishes a comprehensive VQA dataset for compressed 3DGS, offering detailed MOS annotations across 11 scenes and six state-of-the-art compression algorithms.
• The benchmark comprises 660 models generated via systematically varied parameters and continuous 360° orbital camera rendering, ensuring broad coverage of spatial and temporal artifacts.
• Deep no-reference VQA models, particularly DOVER, outperform traditional metrics in aligning with human judgments, highlighting the need for improved perceptual quality assessment in 3DGS compression.

3DGS-VBench is a large-scale Video Quality Assessment (VQA) dataset and benchmark for compressed 3D Gaussian Splatting (3DGS) content. Introduced in "3DGS-VBench: A Comprehensive Video Quality Evaluation Benchmark for 3DGS Compression" (Xing et al., 9 Aug 2025), it contains 660 compressed 3DGS models and corresponding video sequences generated from 11 scenes across 6 state-of-the-art 3DGS compression algorithms with systematically designed parameter levels. The benchmark couples these assets with Mean Opinion Score (MOS) annotations and a comparative evaluation of 15 objective quality metrics. Its motivating observation is that 3DGS compression introduces distortions such as geometry thinning, color quantization artifacts, anchor-level sampling changes, and rate–distortion trade-offs that alter both spatial structure and temporal coherence along camera paths, and that such degradations are not well captured by conventional raster-domain IQA/VQA practice (Xing et al., 9 Aug 2025).

1. Scope and benchmark identity

3D Gaussian Splatting is treated in the benchmark as an explicit 3D scene representation enabling real-time novel view synthesis with high visual fidelity while imposing a substantial storage footprint. The paper gives representative original-model sizes in the benchmark of 795.1323 MB for Mip-NeRF360, 434.3680 MB for Tanks & Temples, and 665.9170 MB for Deep Blending, which motivates compression-oriented evaluation (Xing et al., 9 Aug 2025).

The benchmark is specifically about video quality evaluation for compressed 3DGS, rather than a universal benchmark for every 3DGS task. A common source of confusion is terminological: several contemporaneous papers discuss a broader "3DGS-VBench"-style ecosystem covering XR scene creation, authenticity assessment, adverse-weather robustness, compression toolkits, or dynamic synchronized multi-view NVS, but they do not introduce a benchmark with that exact name. The benchmark explicitly titled 3DGS-VBench is the compression-oriented VQA dataset of (Xing et al., 9 Aug 2025), whereas the broader uses are programmatic or hypothetical extensions in adjacent work (Qiu et al., 16 Jan 2025, Nucci et al., 30 Apr 2026, Liu et al., 31 Dec 2025, Teng et al., 20 Oct 2025, Zhang et al., 12 May 2026).

The paper’s main contributions are threefold. First, it establishes a large-scale VQA dataset tailored to 3DGS compression. Second, it benchmarks 6 compression methods on storage efficiency and visual quality. Third, it evaluates 15 IQA/VQA metrics on 3DGS content and argues that existing metrics, especially traditional full-reference image metrics and standard no-reference IQA baselines, are often poorly aligned with human judgments on this distortion family (Xing et al., 9 Aug 2025).

2. Dataset construction and rendering protocol

The dataset comprises 11 scenes from four multiview datasets: Mip-NeRF360 with 6 scenes, Tanks and Temples with 2 scenes, Deep Blending with 2 scenes, and PKU-DyMVHumans with 1 scene (Xing et al., 9 Aug 2025). The Mip-NeRF360 scenes are bicycle at 1237×822, flowers at 1256×828, garden at 1297×840, counter at 1558×1038, kitchen at 1558×1039, and room at 1557×1038. Tanks and Temples contributes train at 980×545 and truck at 979×546. Deep Blending contributes playroom at 1264×832 and drjohnson at 1332×876. PKU-DyMVHumans contributes Dance_Dunhuang_Pair at 1600×876.

The 660 models arise from applying six compression algorithms to these scenes under systematically varied compression settings. Each scene’s trained 3DGS point cloud is imported into Blender, and virtual cameras are assigned calibrated focal length, position, orientation, and resolution matching the original dataset. For each processed visual sequence, 600 continuous viewpoints are arranged on a 360° orbital path with uniform angular spacing. Rendering produces 600 frames per sequence, which are encoded as 20-second videos at 30 fps using FFMPEG libx265 with constant rate factor $\mathrm{CRF} = 10$ (Xing et al., 9 Aug 2025). The paper does not report special temporal stabilization; temporal consistency follows from the continuous orbital camera path and uniform sampling.

The six compression methods and their distortion-level designs are as follows:

Method	Controlled parameters	DL per scene
Compact-3DGS	hashmap; (codebook, rvq_depth); lambda	13
CompGS	g-size; c-size; reg	13
c3dgs	(c-size, c-include); (g-size, g-include)	9
LightGaussian	prune; (c-ratio, c-size)	9
Scaffold-GS	vsize	8
HAC	lambda	8

For Compact-3DGS, the hashmap compression levels are $2^1$, $2^2$, $2^7$, $2^{12}$, with default $2^{19}$; the $(\text{codebook}, \text{rvq\_depth})$ levels are $(2^3,1)$, $(2^4,1)$, $(2^6,1)$, $2^1$0, with default $2^1$1; and lambda levels are 0.014, 0.012, 0.010, 0.006, with default 0.0005. For CompGS, g-size levels are 1, 2, $2^1$2, $2^1$3, with default $2^1$4; c-size levels are 1, 2, $2^1$5, $2^1$6, with default $2^1$7; and reg levels are $2^1$8, $2^1$9, $2^2$0, $2^2$1, with default $2^2$2 (Xing et al., 9 Aug 2025).

For c3dgs, the $2^2$3 levels are $2^2$4, $2^2$5, $2^2$6, $2^2$7, with default $2^2$8; the $2^2$9 levels are $2^7$0, $2^7$1, $2^7$2, $2^7$3, with default $2^7$4. For LightGaussian, prune levels are 0.97, 0.95, 0.90, 0.85, with default 0.66; the $2^7$5 levels are $2^7$6, $2^7$7, $2^7$8, $2^7$9, with default $2^{12}$0. For Scaffold-GS, vsize levels are 0.350, 0.250, 0.200, 0.160, 0.120, 0.080, 0.050, with default 0.001. For HAC, lambda levels are 0.800, 0.600, 0.400, 0.300, 0.200, 0.120, 0.060, with default 0.004. The total model count is therefore $2^{12}$1 (Xing et al., 9 Aug 2025).

The dataset is hosted at https://github.com/YukeXing/3DGS-VBench. The paper does not specify a license, file structure or naming convention, or recommended train/val/test splits (Xing et al., 9 Aug 2025).

3. Subjective annotation protocol and reliability

Subjective evaluation follows an 11-level impairment scale per ITU-T P.910. The display device is a 27-inch AOC Q2790PQ used in an indoor laboratory under standard lighting conditions, and playback is provided through a Python Tkinter interface. To manage fatigue, the 660 processed visual sequences are randomly divided into 8 groups (Xing et al., 9 Aug 2025).

The paper contains a notable numerical discrepancy in the description of participants. The abstract states that annotations were obtained from 50 participants, whereas the subjective experiment section reports 15 annotators × 660 videos = 9,900 annotations. The benchmark nevertheless reports 660 MOS values after outlier handling, and the score rejection rate is given as 2% following ITU recommendations (Xing et al., 9 Aug 2025). This is best read as a reporting inconsistency rather than a methodological claim about two distinct evaluation rounds.

For a processed visual sequence with valid subject scores $2^{12}$2 and $2^{12}$3 valid ratings after rejection, MOS is defined as

$2^{12}$4

The paper states that subject rejection and outlier detection follow ITU guidance, citing ITU-T P.910 and ITU-R BT.500 methodologies, but it does not detail the exact thresholds or criteria used (Xing et al., 9 Aug 2025).

The benchmark reports several reliability indicators. Diversity is validated through an SI/TI scatter over the 11 scenes, intended to show coverage of spatial and temporal complexity. The MOS distribution spans 0–10 and is described as approximately Gaussian centered at about 6, with sufficient samples in the low-quality range $2^{12}$5, supporting robust training coverage. Inter-rater agreement, confidence intervals, and formal statistical tests are not reported (Xing et al., 9 Aug 2025).

4. Compression methods and empirical trade-offs

The six benchmarked algorithms span several families of 3DGS compression. Compact-3DGS combines learnable volume masking through lambda, residual vector quantization for geometry through codebook size and RVQ depth, and hash-grid neural fields replacing spherical harmonics through the hashmap parameter. CompGS uses K-means vector quantization over Gaussian parameters with geometry and color codebooks and an opacity regularization term reg. c3dgs combines K-means codebooks with entropy encoding and uses color and geometry codebook sizes together with importance parameters. LightGaussian prunes low-significance Gaussians, distills spherical harmonics to lower degrees, and vector-quantizes spherical harmonics. Scaffold-GS is anchor-based and controlled by vsize. HAC uses hash-grid assisted context-aware entropy compression with adaptive quantization through lambda (Xing et al., 9 Aug 2025).

The benchmark provides quantitative comparisons at default optimal parameters. For the uncompressed reference, the reported averages are: Mip-NeRF360 with PSNR 27.6655, SSIM 0.9141, LPIPS 0.1267, and size 795.1323 MB; Tanks & Temples with PSNR 23.7579, SSIM 0.89339, LPIPS 0.09559, and size 434.3680 MB; and Deep Blending with PSNR 29.6219, SSIM 0.9269, LPIPS 0.1016, and size 665.9170 MB (Xing et al., 9 Aug 2025).

Among compressed methods, HAC is reported as offering the best quality–efficiency balance. Its default results are 27.5911 PSNR, 0.9112 SSIM, 0.1364 LPIPS, 8.7413 MOS, and 15.8625 MB on Mip-NeRF360; 24.3093 PSNR, 0.9014 SSIM, 0.0928 LPIPS, 8.1786 MOS, and 14.5386 MB on Tanks & Temples; and 30.3207 PSNR, 0.9342 SSIM, 0.0975 LPIPS, 8.8214 MOS, and 7.7961 MB on Deep Blending (Xing et al., 9 Aug 2025). The paper characterizes HAC as achieving approximately $2^{12}$6 to $2^{12}$7 compression with competitive quality, including less than 0.3 dB loss versus Scaffold-GS in some cases.

CompGS is presented as a balanced trade-off, with 27.1742 PSNR, 0.9042 SSIM, 0.1468 LPIPS, 8.3484 MOS, and 21.9188 MB on Mip-NeRF360; 23.2633 PSNR, 0.8845 SSIM, 0.1132 LPIPS, 7.5524 MOS, and 13.6345 MB on Tanks & Temples; and 30.0751 PSNR, 0.9338 SSIM, 0.0991 LPIPS, 9.0238 MOS, and 14.7798 MB on Deep Blending (Xing et al., 9 Aug 2025). The text summarizes this regime as approximately $2^{12}$8 to $2^{12}$9 compression with less than 1 dB quality loss.

c3dgs and Compact-3DGS occupy intermediate positions. c3dgs reports 28.6562 MB, 17.6700 MB, and 23.8754 MB on Mip-NeRF360, Tanks & Temples, and Deep Blending respectively, with MOS values 8.2960, 7.5619, and 8.6786. Compact-3DGS reports 48.7238 MB, 39.7017 MB, and 43.3126 MB respectively, with MOS values 8.1186, 6.9000, and 8.6786 (Xing et al., 9 Aug 2025).

LightGaussian shows a different trade-off. On Mip-NeRF360 it records 25.4406 PSNR, 0.8582 SSIM, 0.2009 LPIPS, 8.1333 MOS, and 52.0151 MB; on Tanks & Temples, 22.6449 PSNR, 0.8565 SSIM, 0.1637 LPIPS, 7.4190 MOS, and 28.6682 MB; and on Deep Blending, 25.9883 PSNR, 0.8447 SSIM, 0.1950 LPIPS, 7.9333 MOS, and 43.3730 MB (Xing et al., 9 Aug 2025). The paper describes this pattern as larger quality degradation, approximately 2–4 dB PSNR loss, for moderate compression of about $2^{19}$0 to $2^{19}$1.

For Scaffold-GS, the text states that it yields the highest or near-highest quality but relatively modest compression, approximately $2^{19}$2 to $2^{19}$3, making it suitable for quality-critical scenarios. The paper also notes that the corresponding table row is incomplete, although the text cites Mip-NeRF360 PSNR 27.8406 as an example of its superior visual quality (Xing et al., 9 Aug 2025).

5. Objective quality metrics and correlation with human opinion

The benchmark evaluates 15 metrics across three broad paradigms: full-reference IQA, no-reference IQA, and deep no-reference VQA. The full-reference set includes PSNR, SSIM, MS-SSIM, IW-SSIM, VIF, FSIM, LPIPS, and DISTS. The no-reference IQA set includes BRISQUE and CLIP-IQA. The deep no-reference VQA set includes DOVER, FAST-VQA, simpleVQA, VSFA, and Q-Align (Xing et al., 9 Aug 2025).

The paper reports standard formulations for several core measures. PSNR is written as

$2^{19}$4

and SSIM as

$2^{19}$5

The Spearman rank correlation is

$2^{19}$6

the Pearson linear correlation is

$2^{19}$7

and Kendall’s tau is

$2^{19}$8

The paper does not report applying a non-linear mapping before PLCC or RMSE calculation (Xing et al., 9 Aug 2025).

The reported correlations are:

Metric	SRCC / PLCC / KRCC	Category
PSNR	0.5022 / 0.4976 / 0.3560	FR-IQA
SSIM	0.5108 / 0.4758 / 0.3684	FR-IQA
LPIPS	0.5106 / 0.4581 / 0.3619	FR-IQA
DISTS	0.7317 / 0.7146 / 0.5269	FR-IQA
VIF	0.4220 / 0.4252 / 0.2917	FR-IQA
FSIM	0.5866 / 0.5815 / 0.4311	FR-IQA
IW-SSIM	0.6309 / 0.5962 / 0.4488	FR-IQA
MS-SSIM	0.5028 / 0.4700 / 0.3602	FR-IQA
CLIP-IQA	0.3913 / 0.2738 / 0.3216	NR-IQA
BRISQUE	0.2379 / 0.1819 / 0.1749	NR-IQA
DOVER	0.9409 / 0.9308 / 0.7901	Deep NR-VQA
FAST-VQA	0.9314 / 0.9255 / 0.7753	Deep NR-VQA
simpleVQA	0.9350 / 0.7813 / 0.9314	Deep NR-VQA
VSFA	0.9392 / 0.9345 / 0.7908	Deep NR-VQA
Q-Align	0.8485 / 0.6489 / 0.8126	Deep NR-VQA

The benchmark’s central empirical conclusion is that deep no-reference VQA models substantially outperform traditional full-reference IQA and standard no-reference IQA methods on compressed 3DGS videos. DOVER gives the strongest overall alignment with human opinion, with SRCC 0.9409, while DISTS is the strongest full-reference metric at SRCC 0.7317. By contrast, classic full-reference metrics such as SSIM and PSNR are both near 0.51 SRCC, and BRISQUE is particularly weak at 0.2379 (Xing et al., 9 Aug 2025). The paper attributes this gap to a mismatch between conventional pixel-domain assumptions and the structural, spatio-temporal artifacts introduced by generative 3DGS compression.

6. Position in the broader 3DGS benchmark landscape

3DGS-VBench occupies a specific niche: human-centered video-quality evaluation for compressed 3DGS along continuous camera paths. It is not a geometry benchmark, an authenticity benchmark, an adverse-condition robustness benchmark, or a dynamic multi-view standardized dataset. Other papers cover those neighboring problem classes with different emphases. Splatwizard is a unified toolkit for 3DGS compression benchmarking with PSNR, SSIM, LPIPS, Chamfer distance, FPS, and resource profiling, but it explicitly states that it does not mention any benchmark named "3DGS-VBench" (Liu et al., 31 Dec 2025). Fake3DGS proposes an authenticity benchmark for 3DGS manipulation detection and suggests that it could serve an authenticity track within a comprehensive "3DGS-VBench" (Nucci et al., 30 Apr 2026). The XR study in "Creating Virtual Environments with 3D Gaussian Splatting: A Comparative Study" recommends axes such as fidelity, efficiency, robustness, editability/interactivity, and scalability for a potential benchmark structure (Qiu et al., 16 Jan 2025). RaindropGS proposes a domain-specific track for unconstrained raindrop-corrupted inputs (Teng et al., 20 Oct 2025), while the standardized dynamic multi-view benchmark built around TA-3DGS addresses synchronized dynamic-scene NVS rather than compression VQA (Zhang et al., 12 May 2026).

This suggests that 3DGS-VBench is best understood as a specialized compression-VQA component within a larger potential benchmarking ecosystem. Its contribution is unusually specific: it measures perceptual consequences of 3DGS compression in video form, rather than attempting to unify all evaluation dimensions of 3DGS research.

The paper also states several limitations. Scene diversity is restricted to 11 scenes, with only one dynamic human figure scene. The rendering protocol uses a uniform 360° orbital camera path and a single encoding setting, libx265 with CRF=10. The paper does not specify a license, file structures, or recommended data splits. It does not introduce a new VQA architecture or prescribe official training hyperparameters for specialized models, although it argues that the MOS-labeled videos can support such training (Xing et al., 9 Aug 2025).

Within those limits, the benchmark has two broader implications. First, it shows that strong human-aligned quality estimation for compressed 3DGS is currently better served by deep no-reference video models than by traditional full-reference image metrics. Second, it provides a common protocol for comparing storage reduction against perceptual degradation across six widely used compression families. The paper further notes that current 3DGS generation often uses L1 and SSIM losses, and suggests that adopting better perceptual losses, citing DISTS and related work, may improve generation results for compression-aware training (Xing et al., 9 Aug 2025).