Reconstructing Historical Manuscripts through MSI: The Potential of Contrast in Assessing Image Quality and Legibility

Abstract: Digital restoration of historical manuscript images aims to improve readability while preserving the authenticity of cultural heritage documents. However, evaluating quality of restored manuscripts remains challenging, where readability is often subjective and expert annotations are scarce. This study investigates the suitability of contrast-based image quality measures to assess quality and legibility of reconstructed manuscript images from multi-spectral imaging. Two experiments were conducted with publicly-available data sets, facilitating manual quality scores by experts and full-reference image quality measures as reference evaluations. The results show that potential contrast achieves the highest correlation with expert ratings, while contrast-to-noise ratio demonstrates the strongest agreement with full-reference quality measures. Overall, contrast-based measures consistently outperform general image quality measures, demonstrating their potential as objective indicators of manuscript legibility and reconstruction quality.

• The paper finds that Normalized Potential Contrast (NPC) best matches expert legibility judgments on the SALAMI dataset, achieving a mean Spearman correlation of 0.6806 and outperforming RMSC, CNR, entropy, BRISQUE, NIQE, and PIQE.
• Experiments on randomly reconstructed multi-spectral parchment images show that contrast-to-noise ratio (CNR) aligns most closely with full-reference quality measures, reaching mean correlations of up to 0.91, while NPC performs competitively.
• The findings support contrast-based, reference-free evaluation for manuscript restoration, but mask dependence, limited datasets, proxy references, and NPC’s intensity-transformation invariance require validation with broader human assessments and document-specific models.

Motivation and problem statement

Automated evaluation of legibility in digitally restored historical manuscripts remains an open problem in document image analysis. Standard image quality assessment (IQA) measures are designed for natural images and frequently fail to capture whether degraded text is actually readable, while expert annotations of legibility are scarce and expensive to produce. This paper investigates whether contrast-based measures—specifically Normalized Potential Contrast (NPC) and contrast-to-noise ratio (CNR)—can serve as objective indicators of manuscript reconstruction quality and legibility, using two publicly available datasets derived from multi-spectral imaging (MSI). The motivation is practical: task-driven restoration pipelines and large-scale evaluation require reference-free quality signals that correlate with human judgment.

The key methodological premise is that Potential Contrast (PC), introduced by Shaus et al., estimates the maximum achievable contrast between foreground and background pixel classes under arbitrary grayscale intensity transformations. This makes it conceptually distinct from conventional contrast statistics: it accounts for the fact that a viewer may adjust brightness/contrast when inspecting a manuscript, so information invisible at the given intensity scaling may still be recoverable. The paper employs the normalized variant (NPC), which is data-type independent and invariant to linear brightness/contrast changes.

Experimental design

Two experiments are conducted on MSI-derived data.

Experiment 1 (SALAMI dataset). The SALAMI dataset provides 250 grayscale reconstructions ($900 \times 900$ px) from 50 regions of 48 manuscripts, each annotated region-wise by 20 philology/paleography experts. From these, the author selects 11 images (5 versions each; 55 test images) satisfying two criteria: writing covers the whole image, and at least one version permits automated text/background extraction via direct thresholding. For each image, 10,000 random patches of size $200 \times 400$ are sampled to match the region-wise structure of the score maps. NPC is evaluated against mean expert score maps via absolute Spearman Rank Correlation Coefficient (SRCC), alongside CNR, root mean square contrast (RMSC), entropy, and the no-reference IQA measures BRISQUE, NIQE, and PIQE. Notably, PC had been excluded from the original SALAMI baseline study because it requires defined foreground/background samples; this paper supplies those masks via thresholding.

Experiment 2 (Parchment dataset with random reconstructions). Four severely degraded patches (208R, 305R, 309R, 602V) of artificially degraded iron-gall ink parchment are selected, restricted to regions where signal persists in at least one of the 21 spectral bands (400–950 nm). Binary text/background masks are created manually on small regions using a heuristically chosen band, under a strict 15-minute-per-image budget in GIMP to mimic realistic annotation constraints. Since no expert ratings exist for this data, quality references are constructed by generating 100,000 random grayscale reconstructions per patch via uniform random orthogonal projections of the 21-band MSI vectors onto one dimension (equivalently, uniform sampling on the sphere), and correlating tested measures against three full-reference (FR) IQA measures computed against the untreated ground truth: HaarPSI, polarity-invariant Pearson correlation, and MS-SSIM. An auxiliary experiment identifies the best-ranked MSI band per measure.

Results

Experiment 1: NPC correlates best with expert judgment. Over the 10,000 random patches, NPC achieves a mean SRCC of 0.6806 against the mean expert score maps—the highest of all tested measures—with low variance across patches, indicating stability. On full images the correlation drops slightly to 0.6561, which the author attributes to the region-wise construction of the score maps versus whole-image NPC computation. The comparison is stark for general-purpose NR-IQA measures:

IQA measure	SRCC (10k patches)	SRCC (full image)
NPC	0.6806	0.6561
RMSC	0.5896	0.6150
CNR	0.5332	0.4627
Entropy	0.4805	0.5136
NIQE	0.2031	0.2341
PIQE	0.1630	0.1558
BRISQUE	0.0562	0.1709

Three observations carry weight here. First, the three contrast-based measures (NPC, RMSC, CNR) occupy the top three positions, supporting the central claim that contrast notions are informative proxies for restoration quality and legibility. Second, BRISQUE, NIQE, and PIQE—all designed for natural images—correlate poorly with expert legibility scores (SRCC ≤ 0.24), confirming that natural-image IQA transfers badly to this domain. Third, NPC is not infallible: the paper explicitly shows a failure case where NPC rates a patch at 0.13 despite an expert mean score of 188.95/255, and the overall correlation of ~0.68 leaves substantial unexplained disagreement.

Experiment 2: CNR leads against full-reference measures. Averaged over the four images and 100,000 random reconstructions each, CNR achieves the highest mean agreement with the FR reference set (mean SRCC up to 0.91 for image 2), followed by NPC:

IQA measure	Img 1	Img 2	Img 3	Img 4
CNR	0.86	0.91	0.90	0.68
NPC	0.79	0.91	0.78	0.61
Entropy	0.55	0.73	0.59	0.27
NIQE	0.22	0.47	0.61	0.49
PIQE	0.16	0.06	0.39	0.34
BRISQUE	0.10	0.16	0.07	0.07
RMSC	0.03	0.57	0.48	0.26

(Each cell is the mean SRCC over HaarPSI, Pearson correlation, and MS-SSIM.) The reversal of ranking between CNR and NPC relative to Experiment 1 is explained by a structural property rather than noise: NPC is invariant to linear intensity transformations, aligning with human viewing behavior but not with FR measures that operate at fixed intensity scaling. The paper illustrates this with two visually very different renderings of the same reconstruction sharing an identical NPC value of 0.77—a property that is advantageous when users can adjust display contrast, but that depresses correlation with scale-sensitive FR metrics. Image 4 yields uniformly weaker correlations (CNR 0.68, NPC 0.61), suggesting measure reliability degrades on particular degradation patterns.

Limitations

The paper is candid about several constraints. Experiment 2 lacks human annotations entirely, so FR-IQA measures serve as proxy references; these capture structural fidelity but do not necessarily represent expert judgment of legibility, meaning the CNR result should be read as agreement with computational references, not with humans. Both NPC and CNR depend on manually created foreground/background masks, injecting subjectivity into the evaluation—the 15-minute masking budget is a pragmatic mitigation, not a validation. The SALAMI subset is small (11 source images) and filtered by the requirement that thresholding succeeds, potentially biasing toward higher-contrast material. Dataset coverage is narrow relative to the diversity of manuscript degradations, and deep-learning-based or document-specific IQA measures are not yet included in the comparison.

Summary

This study provides the first systematic evaluation of Potential Contrast as an IQA measure for historical manuscript legibility. The evidence supports two complementary conclusions: NPC best tracks expert legibility assessments (SRCC ≈ 0.68 on SALAMI), while CNR best tracks full-reference quality on controlled degradation data (mean SRCC up to 0.91), with both consistently outperforming natural-image no-reference measures such as BRISQUE, NIQE, and PIQE. The open questions left by the paper are concrete: how contrast-based measures behave against direct human judgment in the absence of ground truth, how sensitive they are to mask quality, and whether they retain their advantage against learning-based document-specific quality measures and across broader degradation types.