DIQA-5000: Document Image Quality Benchmark
- DIQA-5000 is a benchmark for document image quality assessment comprising 5,000 enhanced images with subjective MOS ratings for overall quality, sharpness, and color fidelity.
- It employs a stochastic enhancement pipeline over five authentic distortion types to generate diverse outputs from 500 real-world distorted captures.
- DocIQ, evaluated on DIQA-5000, achieves high PLCC and SRCC scores, underscoring the importance of feature fusion and multi-dimensional annotation in NR-IQA.
DIQA-5000 is a subjective benchmark for no-reference document image quality assessment introduced in “DocIQ: A Benchmark Dataset and Feature Fusion Network for Document Image Quality Assessment” (Ma et al., 21 Sep 2025). It comprises 5,000 document images generated from 500 real-world distorted document captures through a stochastic enhancement pipeline, and each benchmark image is annotated along three perceptual dimensions—overall quality, sharpness, and color fidelity—using human ratings aggregated as mean opinion scores (MOSs) (Ma et al., 21 Sep 2025). The benchmark is positioned for OCR, document restoration, and evaluation of document image processing systems, with the stated motivation that document images differ materially from natural images in both structure and degradation patterns, especially because they contain text, tables, and layout-sensitive content and are often affected by shadows, blur, creases, partial occlusions, and moiré (Ma et al., 21 Sep 2025).
1. Scope and problem formulation
DIQA-5000 targets document-image-specific NR-IQA rather than generic natural-image IQA. Its intended operating regime is realistic document capture and enhancement, where quality directly affects OCR, restoration, and downstream document processing (Ma et al., 21 Sep 2025). The benchmark is therefore not organized around pristine-reference comparisons or task-agnostic perceptual fidelity alone; instead, it is built to measure the quality of processed document images under document-centric distortions and enhancement outcomes.
The benchmark’s rationale is that existing IQA datasets and methods are dominated by natural images, whereas document images are structurally distinct and often degraded by artifacts that interact strongly with readability and page layout (Ma et al., 21 Sep 2025). This differentiates DIQA-5000 from earlier document DIQA work such as “CG-DIQA: No-reference Document Image Quality Assessment Based on Character Gradient,” which was blur-sensitive, OCR-oriented, and evaluated on a public DIQA dataset containing 175 color images, with OCR accuracy used as the ground-truth quality metric (Li et al., 2018). DIQA-5000 instead uses multi-dimensional subjective annotation and is explicitly framed as a benchmark for modern learning-based DIQA (Ma et al., 21 Sep 2025).
2. Corpus construction and distortion model
The dataset is built from 500 real-world distorted document images curated from publicly accessible PDFs and selected to cover textual, tabular, and mixed-content layouts (Ma et al., 21 Sep 2025). These documents were printed at 300 dpi to create physical paper originals, and the 500 captured images were then produced under five authentic distortion scenarios, with 100 images per distortion type: shadow from uneven illumination during smartphone capture, occlusion from partial obstruction by objects, blurring from motion blur or defocus, creases from physical folds in printed documents, and moiré patterns from photographing screens displaying documents (Ma et al., 21 Sep 2025).
From these 500 distorted source images, the benchmark images were generated by a stochastic document enhancement pipeline containing six processing operations: dewarp, demoiré, occlusion removal, deblur, deshadow, and appearance enhancement (Ma et al., 21 Sep 2025). The paper states that dewarp has 3 options, demoiré 2, occlusion removal 2, deblur 3, deshadow 4, and appearance enhancement 9 (Ma et al., 21 Sep 2025). The pipeline begins with document boundary detection and background removal; thereafter, each stage is either randomly selected from the available methods or skipped entirely, and the execution order of deblurring, deshadowing, and enhancement is randomized (Ma et al., 21 Sep 2025). This process yields 10 distinct enhanced versions per input image, producing enhanced document images total (Ma et al., 21 Sep 2025).
| Element | Specification |
|---|---|
| Source captures | 500 real-world distorted document images |
| Benchmark size | 5,000 enhanced document images |
| Authentic distortion scenarios | shadow, occlusion, blurring, creases, moiré |
The benchmark itself is the 5,000-image enhanced set; the original 500 distorted captures are the sources used to create it (Ma et al., 21 Sep 2025). This design makes DIQA-5000 a processed-image benchmark rather than a raw-capture-only corpus. A plausible implication is that the benchmark evaluates both acquisition quality and enhancement quality, because multiple enhancement outputs are derived from the same distorted source image (Ma et al., 21 Sep 2025).
3. Subjective annotation and label semantics
DIQA-5000 is annotated along three perceptual dimensions: overall quality, sharpness, and color fidelity (Ma et al., 21 Sep 2025). The paper states that 23 experienced subjects participated, that the 5,000 images were divided into five balanced batches, and that 15 raters were assigned to each batch so that each image received 15 independent ratings for each dimension (Ma et al., 21 Sep 2025). Final quality ratings were annotated as MOSs, and the authors performed data cleaning to remove inconsistent or unreliable ratings according to ITU-R BT.500 for each batch (Ma et al., 21 Sep 2025).
The paper does not provide the exact rating scale, nor does it report a formal instruction script, demographic details, display specifications, room illumination, or session duration (Ma et al., 21 Sep 2025). It also does not report inter-rater agreement coefficients, standard deviations, confidence intervals, Cronbach’s alpha, or split-half reliability (Ma et al., 21 Sep 2025). Accordingly, the benchmark’s annotation protocol is clearly subjective and multi-rater, but its reliability characterization is limited in the published text.
The benchmark design implies that overall quality reflects holistic perceptual quality of the processed document image; sharpness captures perceived clarity and legibility, especially relevant for text edges and fine detail; and color fidelity reflects whether the visual appearance resembles a clean scanned or digital document without objectionable color shifts or appearance distortions (Ma et al., 21 Sep 2025). This tri-axis labeling scheme distinguishes DIQA-5000 from OCR-grounded DIQA formulations such as CG-DIQA, where the target quantity was OCR accuracy rather than subjective MOS (Li et al., 2018).
4. Evaluation protocol and associated model
The benchmark methodology evaluates NR-IQA models on DIQA-5000 using PLCC and SRCC (Ma et al., 21 Sep 2025). The exact evaluation protocol is an 80%-20% train-test split (Ma et al., 21 Sep 2025). The paper does not mention a separate validation split, repeated random splits, median-over-runs reporting, or cross-validation, and it does not specify whether splitting is done by enhanced image or by source document (Ma et al., 21 Sep 2025). The text explicitly notes that this omission matters because source-level leakage could otherwise make the task easier if variants of the same original are split across train and test (Ma et al., 21 Sep 2025).
Six baseline methods are compared: DBCNN, HyperIQA, MUSIQ, RichIQA, StairIQA, and TReS (Ma et al., 21 Sep 2025). These are general-purpose NR-IQA models rather than document-specific DIQA baselines, which itself reflects the scarcity of established document IQA benchmarks (Ma et al., 21 Sep 2025). All compared methods were retrained and tested with their original configurations to ensure fair evaluation (Ma et al., 21 Sep 2025).
The benchmark is introduced together with DocIQ, a specialized NR-DIQA network composed of four components: a layout fusion downsampler, a backbone network, a feature fusion module, and parallel quality regressors (Ma et al., 21 Sep 2025). The backbone is ResNet50 pretrained on ImageNet (Ma et al., 21 Sep 2025). The layout fusion downsampler uses a dual-path design: one path performs ordinary spatial downsampling, while the second path takes a concatenated input composed of the original image and a semantic layout mask indicating text regions, tables, and figures, with the masks produced by pretrained document layout detection models (Ma et al., 21 Sep 2025). The feature fusion module progressively merges multi-scale features from different backbone stages through lightweight hyper-structures, and the final component uses independent regression heads for each quality dimension, with two linear layers per head: a shared first layer and dimension-specific second layers that separately predict scores for each rater (Ma et al., 21 Sep 2025).
Training uses a ResNet50 backbone pretrained on ImageNet, input resolution , Adam optimizer, initial learning rate , step-decay schedule with step size 10 epochs and decay factor 0.6, total 60 epochs, batch size 20, on NVIDIA A10 GPUs (Ma et al., 21 Sep 2025). The paper does not specify the actual loss functions or provide training equations, despite emphasizing score-distribution prediction and aggregation into final MOSs (Ma et al., 21 Sep 2025).
5. Quantitative performance and ablation structure
DIQA-5000 is reported as challenging but learnable (Ma et al., 21 Sep 2025). DocIQ achieves the best published performance in the paper on all three subjective dimensions: overall quality with PLCC and SRCC ; sharpness with PLCC and SRCC ; and color fidelity with PLCC and SRCC (Ma et al., 21 Sep 2025). The paper averages these to mean SRCC and mean PLCC 0 (Ma et al., 21 Sep 2025).
| Dimension | DocIQ | Strongest baseline reported |
|---|---|---|
| Overall quality | PLCC 0.9083, SRCC 0.8832 | RichIQA PLCC 0.8660; MUSIQ SRCC 0.8554 |
| Sharpness | PLCC 0.9006, SRCC 0.8615 | TReS PLCC 0.8800; MUSIQ SRCC 0.8460 |
| Color fidelity | PLCC 0.8907, SRCC 0.8666 | StairIQA PLCC 0.8691; RichIQA SRCC 0.8557 |
The benchmark therefore supports the paper’s claim that document-image-specific modeling is necessary, because general-purpose NR-IQA methods remain below the proposed architecture across all three subjective axes (Ma et al., 21 Sep 2025). This suggests that natural-image IQA features transfer only partially to document-image settings.
The ablation study is organized around three design choices: the layout fusion downsampler, the feature fusion module, and the multi-rater strategy (Ma et al., 21 Sep 2025). With all three enabled, the model obtains SRCCs of 0.8832, 0.8615, and 0.8666 for overall quality, sharpness, and color fidelity, respectively (Ma et al., 21 Sep 2025). Replacing the multi-rater strategy with direct MOS regression reduces these to 0.8636, 0.8545, and 0.8553; removing the layout fusion downsampler while keeping feature fusion and multi-rater modeling yields 0.8696, 0.8481, and 0.8589; removing feature fusion while keeping layout fusion and multi-rater modeling yields 0.8448, 0.8293, and 0.8401; and removing both layout fusion and feature fusion while keeping multi-rater strategy yields 0.8162, 0.7901, and 0.8137 (Ma et al., 21 Sep 2025). The paper summarizes the average SRCC losses as 0.0115 for removing layout fusion, 0.0323 for removing feature fusion, and 0.0126 for replacing score-distribution modeling with direct MOS regression (Ma et al., 21 Sep 2025). On that basis, feature fusion is the largest single contributor among the reported ablations.
The same paper also evaluates DocIQ on SmartDoc-QA / SOC, a public set of 2,130 smartphone-captured document images after cleaning, where quality is measured using OCR performance metrics Character ACCuracy and Word ACCuracy (Ma et al., 21 Sep 2025). There, DocIQ again performs best, with CACC PLCC 1, SRCC 2, and WACC PLCC 3, SRCC 4 (Ma et al., 21 Sep 2025). This does not alter the definition of DIQA-5000, but it does indicate that the learned representation correlates with OCR-oriented document utility as well as with subjective MOS (Ma et al., 21 Sep 2025).
6. Related work, ambiguity of the acronym, and benchmark limitations
Within arXiv usage, the acronym “DIQA” is overloaded. DIQA-5000 refers to a document image quality assessment benchmark (Ma et al., 21 Sep 2025), but “Enhanced Dermatology Image Quality Assessment via Cross-Domain Training” introduces Legit.Health-DIQA-Artificial for dermatology NR-IQA and explicitly does not mention DIQA-5000 (Montilla et al., 19 Jun 2025). Likewise, “On Quantum Annealing Without a Physical Quantum Annealer” uses DiQA to mean discretised quantum annealing, and the phrase “DIQA-5000” does not appear in that paper (Bhave et al., 2023). Terminological disambiguation is therefore necessary when the acronym appears without domain context.
In the document-IQA lineage, CG-DIQA is the most direct earlier method-level antecedent in the provided literature (Li et al., 2018). That work proposed a training-free NR-DIQA method based on character gradient, computed on MSER-detected character patches, and evaluated against OCR accuracy on 175 mobile-captured document images (Li et al., 2018). DIQA-5000 differs from that formulation in at least three respects that are explicit in the published descriptions: benchmark scale, distortion breadth, and label semantics, because it comprises 5,000 enhanced images, includes multiple authentic distortion scenarios and enhancement pipelines, and uses subjective MOS along three dimensions rather than OCR accuracy alone (Ma et al., 21 Sep 2025, Li et al., 2018).
Subsequent benchmark-oriented work has also remained adjacent rather than identical. “Q-Doc: Benchmarking Document Image Quality Assessment Capabilities in Multi-modal LLMs” introduces a three-tier evaluation framework built on SmartDoc-QA with 4,260 real-world document images and explicitly does not mention DIQA-5000 (Huang et al., 14 Nov 2025). This indicates that DIQA-5000 occupies one branch of a broader document-IQA benchmark ecosystem rather than serving as the sole standard.
The benchmark’s limitations are material. The dataset includes only five distortion families, the source image count is 500, the published protocol uses a single 80/20 split, and annotation details remain sparse: the exact rating scale, subject demographics, reliability coefficients, and MOS computation formula are not reported (Ma et al., 21 Sep 2025). On the modeling side, explicit fusion equations, loss functions, parameter counts, FLOP analyses, and a quantitative reduced-resolution study are also absent from the published text (Ma et al., 21 Sep 2025). These omissions do not negate DIQA-5000’s benchmark value, but they constrain reproducibility, statistical interpretability, and rigorous comparison under source-level split control.
