Uni-DocDiff: Unified Document Restoration

• Uni-DocDiff is a unified model for document restoration that uses a conditional diffusion framework to repair degradations like deblurring, deshadowing, and dewarping in a single architecture.
• It employs learnable task prompts and integrates classical feature-based priors via an adaptive fusion module to effectively manage multi-task interference.
• Extensive empirical results on standard benchmarks demonstrate its superior performance and extensibility, making it a significant advancement in document restoration.

Uni-DocDiff is a unified and scalable model for document restoration based on a conditional diffusion framework. It is designed to address a wide spectrum of document degradations—deblurring, deshadowing, illumination correction, binarization, handwriting removal, and geometric dewarping—within a single architecture. Unlike previous approaches which rely on separate expert models for each restoration task or handcrafted multi-task unification strategies, Uni-DocDiff employs learnable task prompts, a set of classical feature-based priors, and an adaptive fusion mechanism. Extensive empirical results demonstrate state-of-the-art or superior performance compared to both specialized and prior unified models (Zhao et al., 6 Aug 2025).

1. Conditional Diffusion Backbone

Uni-DocDiff’s core is a conditional diffusion model, specifically a Pixel Prediction Branch (PPB) modeled after Denoising Diffusion Probabilistic Models. During training, the clean document image $x_0$ is corrupted through a fixed, progressively noised forward process: $$q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{\alpha_t} x_{t-1}, (1-\alpha_t)I)$$ with the marginal

$$x_t = \sqrt{\bar\alpha_t} x_0 + \sqrt{1-\bar\alpha_t} \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)$$

and $\bar\alpha_t = \prod_{i=1}^t \alpha_i$.

The reverse process reconstructs the original image, conditioned on the degraded input $x_d$ and a learnable task prompt $\tau$: $$p_\theta(x_{t-1}\mid x_t, x_d, \tau) = \mathcal{N}(x_{t-1}; \mu_\theta(x_t, x_d, \tau), \Sigma_\theta(\tau))$$

Uni-DocDiff predicts $\hat x_0$ directly for sharper results: $\hat x_0 = \mathcal{F}(x_t; x_d, \tau, P)$ enabling the reconstruction mean

$$\mu_\theta = \frac{\sqrt{\bar\alpha_{t-1}}(x_t - \sqrt{1-\bar\alpha_t}\epsilon_\theta(x_t)) + \sqrt{1-\bar\alpha_{t-1}}\hat x_0}{\sqrt{\bar\alpha_t}}$$

The loss for training deviates from the standard MSE (“simple” loss), instead using $$q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{\alpha_t} x_{t-1}, (1-\alpha_t)I)$$0 loss and frequency-weighted auxiliary terms according to task frequency (see Section 5).

2. Learnable Task Prompt Encoding

Task specification in Uni-DocDiff is achieved via a learnable, low-dimensional vector $$q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{\alpha_t} x_{t-1}, (1-\alpha_t)I)$$1 that represents the restoration task (for example, deblurring or binarization). At each level $$q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{\alpha_t} x_{t-1}, (1-\alpha_t)I)$$2 in the backbone, this task prompt is processed by a multilayer perceptron (MLP): $$q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{\alpha_t} x_{t-1}, (1-\alpha_t)I)$$3 These weights modulate the gating of prior features via the Prior Fusion Module (PFM), contextualizing U-Net features for the task at hand. This learnable prompt mechanism imparts scalability and extensibility, as new tasks can be supported by embedding new prompts and retraining lightweight modules, without retraining the full model.

3. Prior Pool Construction

The Prior Pool ($$q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{\alpha_t} x_{t-1}, (1-\alpha_t)I)$$4) supplies explicit local high-frequency and global low-frequency cues derived from classical image processing operators:

• High-frequency features: Sobel filters ($$q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{\alpha_t} x_{t-1}, (1-\alpha_t)I)$$5) and Canny edge detection ($$q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{\alpha_t} x_{t-1}, (1-\alpha_t)I)$$6), capturing structural, text, and edge details.
• Low-frequency features: Median filtering ($$q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{\alpha_t} x_{t-1}, (1-\alpha_t)I)$$7), Gaussian blur ($$q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{\alpha_t} x_{t-1}, (1-\alpha_t)I)$$8), and DCT-based low-frequency reconstruction (retaining the top $$q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{\alpha_t} x_{t-1}, (1-\alpha_t)I)$$9 DCT coefficients), emphasizing smooth content and background.

All six maps, resized to match the input resolution, are concatenated along the channel dimension to form $$x_t = \sqrt{\bar\alpha_t} x_0 + \sqrt{1-\bar\alpha_t} \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)$$0. The use of inexpensive, interpretable filters, as opposed to learned feature extractors, provides explicit access to frequency-tailored information for restoration tasks that are sensitive to specific content frequencies.

4. Prior Fusion Module (PFM)

PFM mediates the integration of classical priors and learned features at each U-Net level. The module consumes the U-Net activation map $$x_t = \sqrt{\bar\alpha_t} x_0 + \sqrt{1-\bar\alpha_t} \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)$$1, the Prior Pool $$x_t = \sqrt{\bar\alpha_t} x_0 + \sqrt{1-\bar\alpha_t} \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)$$2, and the task prompt $$x_t = \sqrt{\bar\alpha_t} x_0 + \sqrt{1-\bar\alpha_t} \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)$$3, and fuses them as follows:

1. Stage-specific adaptation: Adapt priors through a learned convolution:

$$x_t = \sqrt{\bar\alpha_t} x_0 + \sqrt{1-\bar\alpha_t} \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)$$4

2. Content-driven weighting: Compute weights from global average-pooled features:

$$x_t = \sqrt{\bar\alpha_t} x_0 + \sqrt{1-\bar\alpha_t} \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)$$5

3. Task-driven weighting: Project the task prompt using an MLP:

$$x_t = \sqrt{\bar\alpha_t} x_0 + \sqrt{1-\bar\alpha_t} \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)$$6

4. Channel-wise weighting and fusion:

$$x_t = \sqrt{\bar\alpha_t} x_0 + \sqrt{1-\bar\alpha_t} \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)$$7

($$x_t = \sqrt{\bar\alpha_t} x_0 + \sqrt{1-\bar\alpha_t} \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)$$8 denotes broadcasted channel-wise multiplication.)

5. Integration with intermediate features:

$$x_t = \sqrt{\bar\alpha_t} x_0 + \sqrt{1-\bar\alpha_t} \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)$$9

This mechanism enables adaptive selection of relevant priors per task and content state, mitigating multi-task interference by disentangling frequency and task-specific features.

5. Training Paradigm and Multi-Task Loss

Uni-DocDiff employs a two-stage training pipeline:

• Stage 1—Pixel prediction: Joint training on five restoration tasks (deblurring, deshadowing, illumination correction, binarization, handwriting removal) via mini-batch sampling across tasks. Losses are task-frequency-tailored:

$\bar\alpha_t = \prod_{i=1}^t \alpha_i$0

for low-frequency tasks, and

$\bar\alpha_t = \prod_{i=1}^t \alpha_i$1

for high-frequency tasks, where $\bar\alpha_t = \prod_{i=1}^t \alpha_i$2 and $\bar\alpha_t = \prod_{i=1}^t \alpha_i$3 are low-pass (Gaussian) and high-pass (Sobel) filters respectively. The overall loss is a weighted sum, normalized by batch size per task type.

• Stage 2—Geometric prediction: With the backbone frozen, a Coordinate Prediction Branch is trained separately on dewarping datasets using a simple $\bar\alpha_t = \prod_{i=1}^t \alpha_i$4 loss on backward deformation maps:

$\bar\alpha_t = \prod_{i=1}^t \alpha_i$5

Optimizer: AdamW with a learning rate of $\bar\alpha_t = \prod_{i=1}^t \alpha_i$6, employing standard augmentations.

6. Quantitative Evaluation and Ablation

Empirical validation spans six standard document restoration benchmarks. Across deblurring (TDD), deshadowing (Jung’s, Kligler’s, OSR), illumination rectification (DocUNet, RealDAE), dewarping (DIR300, DocUNet*), binarization (DIBCO’18), and handwriting removal (EnsExam), Uni-DocDiff attains performance matching or surpassing specialist baselines and the prior unified DocRes architecture. Selected results are summarized below:

Task	Baseline (Metric)	Uni-DocDiff (Metric)
Deblurring (TDD)	DE-GAN: 0.9226/22.24	0.9824/28.77
Deshadowing	BGSNet: 0.9094/17.34	0.9156/23.93
Illum. rect.	GCDRNet: 0.9423/24.42	0.9485/24.97
Dewarping (DIR)	UVDoc: 0.6380/6.40/0.218	0.6573/5.30/0.203
Binarization	DocRes: 89.82/94.33/19.35	90.32/93.84/19.76
Handw. removal	EnsExam: 0.9671/35.68	0.9685/36.23

Ablative studies show that removing either the Prior Pool or PFM degrades multi-task performance, highlighting their role in mitigating task interference. Incorporating frequency-aware loss terms yields a measurable gain (0.5–1 dB PSNR) across low- and high-frequency tasks.

A notable property is the model’s extensibility: when adapting to new tasks (e.g., novel handwriting-removal splits), retraining only lightweight modules like PFM suffices. This contrasts with alternative prompt-based unification systems that require more substantial retraining.

7. Significance and Architectural Implications

Uni-DocDiff advances document restoration by offering a unified, highly scalable architecture capable of expert-level performance across disparate restoration tasks. The integration of interpretable, computationally light priors with frequency- and task-adaptive fusion mechanisms exemplifies a modular, extensible system. The empirical superiority over disjoint and prior unified baselines, coupled with graceful task-adaptability, marks Uni-DocDiff as a substantive development in the progression toward generalized document analysis and restoration pipelines (Zhao et al., 6 Aug 2025).