---
title: 'DIQA-5000: Document Image Quality Benchmark'
url: https://www.emergentmind.com/topics/diqa-5000
type: topic
---

# DIQA-5000: Document Image Quality Benchmark

DIQA-5000 is a subjective benchmark for no-reference document image quality assessment introduced in “DocIQ: A Benchmark Dataset and Feature Fusion Network for Document Image Quality Assessment” [2509.17012]. It comprises 5,000 document images generated from 500 real-world distorted document captures through a stochastic enhancement pipeline, and each benchmark image is annotated along three perceptual dimensions—overall quality, sharpness, and color fidelity—using human ratings aggregated as mean opinion scores (MOSs) [2509.17012]. The benchmark is positioned for OCR, document restoration, and evaluation of document image processing systems, with the stated motivation that document images differ materially from natural images in both structure and degradation patterns, especially because they contain text, tables, and layout-sensitive content and are often affected by shadows, blur, creases, partial occlusions, and moiré [2509.17012].

## 1. Scope and problem formulation

DIQA-5000 targets document-image-specific NR-IQA rather than generic natural-image IQA. Its intended operating regime is realistic document capture and enhancement, where quality directly affects OCR, restoration, and downstream document processing [2509.17012]. The benchmark is therefore not organized around pristine-reference comparisons or task-agnostic perceptual fidelity alone; instead, it is built to measure the quality of processed document images under document-centric distortions and enhancement outcomes.

The benchmark’s rationale is that existing IQA datasets and methods are dominated by natural images, whereas document images are structurally distinct and often degraded by artifacts that interact strongly with readability and page layout [2509.17012]. This differentiates DIQA-5000 from earlier document DIQA work such as “CG-DIQA: No-reference Document Image Quality Assessment Based on Character Gradient,” which was blur-sensitive, OCR-oriented, and evaluated on a public DIQA dataset containing 175 color images, with OCR accuracy used as the ground-truth quality metric [1807.04047]. DIQA-5000 instead uses multi-dimensional subjective annotation and is explicitly framed as a benchmark for modern learning-based DIQA [2509.17012].

## 2. Corpus construction and distortion model

The dataset is built from 500 real-world distorted document images curated from publicly accessible PDFs and selected to cover textual, tabular, and mixed-content layouts [2509.17012]. These documents were printed at 300 dpi to create physical paper originals, and the 500 captured images were then produced under five authentic distortion scenarios, with 100 images per distortion type: shadow from uneven illumination during smartphone capture, occlusion from partial obstruction by objects, blurring from motion blur or defocus, creases from physical folds in printed documents, and moiré patterns from photographing screens displaying documents [2509.17012].

From these 500 distorted source images, the benchmark images were generated by a stochastic document enhancement pipeline containing six processing operations: dewarp, demoiré, occlusion removal, deblur, deshadow, and appearance enhancement [2509.17012]. The paper states that dewarp has 3 options, demoiré 2, occlusion removal 2, deblur 3, deshadow 4, and appearance enhancement 9 [2509.17012]. The pipeline begins with document boundary detection and background removal; thereafter, each stage is either randomly selected from the available methods or skipped entirely, and the execution order of deblurring, deshadowing, and enhancement is randomized [2509.17012]. This process yields 10 distinct enhanced versions per input image, producing \(500 \times 10 = 5{,}000\) enhanced document images total [2509.17012].

| Element | Specification |
|---|---|
| Source captures | 500 real-world distorted document images |
| Benchmark size | 5,000 enhanced document images |
| Authentic distortion scenarios | shadow, occlusion, blurring, creases, moiré |

The benchmark itself is the 5,000-image enhanced set; the original 500 distorted captures are the sources used to create it [2509.17012]. This design makes DIQA-5000 a processed-image benchmark rather than a raw-capture-only corpus. A plausible implication is that the benchmark evaluates both acquisition quality and enhancement quality, because multiple enhancement outputs are derived from the same distorted source image [2509.17012].

## 3. Subjective annotation and label semantics

DIQA-5000 is annotated along three perceptual dimensions: overall quality, sharpness, and color fidelity [2509.17012]. The paper states that 23 experienced subjects participated, that the 5,000 images were divided into five balanced batches, and that 15 raters were assigned to each batch so that each image received 15 independent ratings for each dimension [2509.17012]. Final quality ratings were annotated as MOSs, and the authors performed data cleaning to remove inconsistent or unreliable ratings according to ITU-R BT.500 for each batch [2509.17012].

The paper does not provide the exact rating scale, nor does it report a formal instruction script, demographic details, display specifications, room illumination, or session duration [2509.17012]. It also does not report inter-rater agreement coefficients, standard deviations, confidence intervals, Cronbach’s alpha, or split-half reliability [2509.17012]. Accordingly, the benchmark’s annotation protocol is clearly subjective and multi-rater, but its reliability characterization is limited in the published text.

The benchmark design implies that overall quality reflects holistic perceptual quality of the processed document image; sharpness captures perceived clarity and legibility, especially relevant for text edges and fine detail; and color fidelity reflects whether the visual appearance resembles a clean scanned or digital document without objectionable color shifts or appearance distortions [2509.17012]. This tri-axis labeling scheme distinguishes DIQA-5000 from OCR-grounded DIQA formulations such as CG-DIQA, where the target quantity was OCR accuracy rather than subjective MOS [1807.04047].

## 4. Evaluation protocol and associated model

The benchmark methodology evaluates NR-IQA models on DIQA-5000 using PLCC and SRCC [2509.17012]. The exact evaluation protocol is an 80%-20% train-test split [2509.17012]. The paper does not mention a separate validation split, repeated random splits, median-over-runs reporting, or cross-validation, and it does not specify whether splitting is done by enhanced image or by source document [2509.17012]. The text explicitly notes that this omission matters because source-level leakage could otherwise make the task easier if variants of the same original are split across train and test [2509.17012].

Six baseline methods are compared: DBCNN, HyperIQA, MUSIQ, RichIQA, StairIQA, and TReS [2509.17012]. These are general-purpose NR-IQA models rather than document-specific DIQA baselines, which itself reflects the scarcity of established document IQA benchmarks [2509.17012]. All compared methods were retrained and tested with their original configurations to ensure fair evaluation [2509.17012].

The benchmark is introduced together with DocIQ, a specialized NR-DIQA network composed of four components: a layout fusion downsampler, a backbone network, a feature fusion module, and parallel quality regressors [2509.17012]. The backbone is ResNet50 pretrained on ImageNet [2509.17012]. The layout fusion downsampler uses a dual-path design: one path performs ordinary spatial downsampling, while the second path takes a concatenated input composed of the original image and a semantic layout mask indicating text regions, tables, and figures, with the masks produced by pretrained document layout detection models [2509.17012]. The feature fusion module progressively merges multi-scale features from different backbone stages through lightweight hyper-structures, and the final component uses independent regression heads for each quality dimension, with two linear layers per head: a shared first layer and dimension-specific second layers that separately predict scores for each rater [2509.17012].

Training uses a ResNet50 backbone pretrained on ImageNet, input resolution \(1600 \times 1600\), Adam optimizer, initial learning rate \(2 \times 10^{-4}\), step-decay schedule with step size 10 epochs and decay factor 0.6, total 60 epochs, batch size 20, on NVIDIA A10 GPUs [2509.17012]. The paper does not specify the actual loss functions or provide training equations, despite emphasizing score-distribution prediction and aggregation into final MOSs [2509.17012].

## 5. Quantitative performance and ablation structure

DIQA-5000 is reported as challenging but learnable [2509.17012]. DocIQ achieves the best published performance in the paper on all three subjective dimensions: overall quality with PLCC \(= 0.9083\) and SRCC \(= 0.8832\); sharpness with PLCC \(= 0.9006\) and SRCC \(= 0.8615\); and color fidelity with PLCC \(= 0.8907\) and SRCC \(= 0.8666\) [2509.17012]. The paper averages these to mean SRCC \(= 0.8704\) and mean PLCC \(= 0.8999\) [2509.17012].

| Dimension | DocIQ | Strongest baseline reported |
|---|---|---|
| Overall quality | PLCC 0.9083, SRCC 0.8832 | RichIQA PLCC 0.8660; MUSIQ SRCC 0.8554 |
| Sharpness | PLCC 0.9006, SRCC 0.8615 | TReS PLCC 0.8800; MUSIQ SRCC 0.8460 |
| Color fidelity | PLCC 0.8907, SRCC 0.8666 | StairIQA PLCC 0.8691; RichIQA SRCC 0.8557 |

The benchmark therefore supports the paper’s claim that document-image-specific modeling is necessary, because general-purpose NR-IQA methods remain below the proposed architecture across all three subjective axes [2509.17012]. This suggests that natural-image IQA features transfer only partially to document-image settings.

The ablation study is organized around three design choices: the layout fusion downsampler, the feature fusion module, and the multi-rater strategy [2509.17012]. With all three enabled, the model obtains SRCCs of 0.8832, 0.8615, and 0.8666 for overall quality, sharpness, and color fidelity, respectively [2509.17012]. Replacing the multi-rater strategy with direct MOS regression reduces these to 0.8636, 0.8545, and 0.8553; removing the layout fusion downsampler while keeping feature fusion and multi-rater modeling yields 0.8696, 0.8481, and 0.8589; removing feature fusion while keeping layout fusion and multi-rater modeling yields 0.8448, 0.8293, and 0.8401; and removing both layout fusion and feature fusion while keeping multi-rater strategy yields 0.8162, 0.7901, and 0.8137 [2509.17012]. The paper summarizes the average SRCC losses as 0.0115 for removing layout fusion, 0.0323 for removing feature fusion, and 0.0126 for replacing score-distribution modeling with direct MOS regression [2509.17012]. On that basis, feature fusion is the largest single contributor among the reported ablations.

The same paper also evaluates DocIQ on SmartDoc-QA / SOC, a public set of 2,130 smartphone-captured document images after cleaning, where quality is measured using OCR performance metrics Character ACCuracy and Word ACCuracy [2509.17012]. There, DocIQ again performs best, with CACC PLCC \(= 0.9218\), SRCC \(= 0.9086\), and WACC PLCC \(= 0.9107\), SRCC \(= 0.8989\) [2509.17012]. This does not alter the definition of DIQA-5000, but it does indicate that the learned representation correlates with OCR-oriented document utility as well as with subjective MOS [2509.17012].

## 6. Related work, ambiguity of the acronym, and benchmark limitations

Within arXiv usage, the acronym “DIQA” is overloaded. DIQA-5000 refers to a document image quality assessment benchmark [2509.17012], but “Enhanced Dermatology Image Quality Assessment via Cross-Domain Training” introduces Legit.Health-DIQA-Artificial for dermatology NR-IQA and explicitly does not mention DIQA-5000 [2506.16116]. Likewise, “On Quantum Annealing Without a Physical Quantum Annealer” uses DiQA to mean discretised quantum annealing, and the phrase “DIQA-5000” does not appear in that paper [2307.09695]. Terminological disambiguation is therefore necessary when the acronym appears without domain context.

In the document-IQA lineage, CG-DIQA is the most direct earlier method-level antecedent in the provided literature [1807.04047]. That work proposed a training-free NR-DIQA method based on character gradient, computed on MSER-detected character patches, and evaluated against OCR accuracy on 175 mobile-captured document images [1807.04047]. DIQA-5000 differs from that formulation in at least three respects that are explicit in the published descriptions: benchmark scale, distortion breadth, and label semantics, because it comprises 5,000 enhanced images, includes multiple authentic distortion scenarios and enhancement pipelines, and uses subjective MOS along three dimensions rather than OCR accuracy alone [2509.17012][1807.04047].

Subsequent benchmark-oriented work has also remained adjacent rather than identical. “Q-Doc: Benchmarking Document Image Quality Assessment Capabilities in Multi-modal Large Language Models” introduces a three-tier evaluation framework built on SmartDoc-QA with 4,260 real-world document images and explicitly does not mention DIQA-5000 [2511.11410]. This indicates that DIQA-5000 occupies one branch of a broader document-IQA benchmark ecosystem rather than serving as the sole standard.

The benchmark’s limitations are material. The dataset includes only five distortion families, the source image count is 500, the published protocol uses a single 80/20 split, and annotation details remain sparse: the exact rating scale, subject demographics, reliability coefficients, and MOS computation formula are not reported [2509.17012]. On the modeling side, explicit fusion equations, loss functions, parameter counts, FLOP analyses, and a quantitative reduced-resolution study are also absent from the published text [2509.17012]. These omissions do not negate DIQA-5000’s benchmark value, but they constrain reproducibility, statistical interpretability, and rigorous comparison under source-level split control.

Source: https://www.emergentmind.com/topics/diqa-5000