Progressive Prompt Fusion Network for TIR Enhancement

• PPFN is a modular neural framework for enhancing multi-degraded thermal images using dual prompt embeddings and a specialized training curriculum.
• It employs FiLM-style channel modulation within encoder–decoder architectures to dynamically steer feature extraction for degradation-specific restoration.
• Empirical results show that integrating PPFN with SPT improves PSNR by up to +8.76% and outperforms traditional methods on real and synthetic TIR data.

The Progressive Prompt Fusion Network (PPFN) is a modular neural framework designed to address the enhancement of thermal infrared (TIR) images subject to multiple, often intertwined degradations such as low contrast, blur, and various noise sources. Existing TIR enhancement techniques generally tackle each degradation independently or apply all-in-one approaches developed for RGB imagery, which often underperform due to fundamentally different imaging characteristics. PPFN introduces prompt-guided channel modulation and a specialized training curriculum—Selective Progressive Training (SPT)—to robustly handle multi-degradation infrared scenarios, enabling significant improvements on real and synthetically corrupted data (Liu et al., 10 Oct 2025).

1. Thermal Infrared Image Degradation Model

TIR imaging pipelines introduce sequential degradations that fundamentally alter the signal structure. The composite degradation is formulated as:

$$\mathbf{I}_d = \bigl(\mathbf{n}_s \circ \mathbf{n}_o \circ \mathcal{K} \circ \mathcal{C}_{\mathrm{LC}}\bigr)(\mathbf{I}_c) + \mathbf{n}_r$$

where:

• $\mathbf{I}_c \in \mathbb{R}^{H \times W}$: Clean TIR image.
• $\mathcal{C}_{\mathrm{LC}}$: Low-contrast operator, parameterized as $$\mathcal{C}_{\mathrm{LC}}(\mathbf{I}) = \alpha\,\mathbf{I} + \beta$$ with $0 < \alpha < 1$.
• $\mathcal{K}$: Blur kernel, e.g., convolution $\mathcal{K}(\mathbf{I}) = \mathbf{k} * \mathbf{I}$.
• $\mathbf{n}_o$, $\mathbf{n}_s$: Optics-induced fixed-pattern noise, stripe noise.
• $\mathbf{n}_r$: Additive random noise, modeled as Gaussian, Poisson, or salt-and-pepper.

Each degradation exhibits unique spatial and frequency behaviors, leading to a complex restoration problem, particularly when multiple effects are superimposed.

2. Prompt Pair Construction and Feature Fusion

PPFN utilizes dual sets of learnable prompt embeddings to encode explicit priors about degradation types and scenario composition:

• Degradation-specific prompts: $\mathbf{I}_c \in \mathbb{R}^{H \times W}$0, capturing noise, blur, and contrast degradations.
• Type-specific prompts: $\mathbf{I}_c \in \mathbb{R}^{H \times W}$1, distinguishing single versus hybrid (composite) degradations.

Learnable prompt embeddings $\mathbf{I}_c \in \mathbb{R}^{H \times W}$2 are encoded as:

$\mathbf{I}_c \in \mathbb{R}^{H \times W}$3

$\mathbf{I}_c \in \mathbb{R}^{H \times W}$4

where $\mathbf{I}_c \in \mathbb{R}^{H \times W}$5 and $\mathbf{I}_c \in \mathbb{R}^{H \times W}$6 are lightweight encoders (two $\mathbf{I}_c \in \mathbb{R}^{H \times W}$7 convolutions + ReLU). The resulting feature vectors are concatenated, linearly projected, and nonlinearly activated:

$\mathbf{I}_c \in \mathbb{R}^{H \times W}$8

This fused prompt vector provides adaptive context suitable for modulating feature processing throughout the network.

3. Network Architecture and Prompt-guided Modulation

The PPFN module integrates into encoder–decoder or U-Net architectures, exemplified by its default use with Restormer. The principal mechanisms are:

• Feature Extraction: Stacked downsampling convolution layers yield multiscale representations.
• Backbone Block Modulation: At each relevant layer $\mathbf{I}_c \in \mathbb{R}^{H \times W}$9, a prompt fusion block computes channel-wise modulation parameters via

$\mathcal{C}_{\mathrm{LC}}$0

where $\mathcal{C}_{\mathrm{LC}}$1. The activation is FiLM-style:

$\mathcal{C}_{\mathrm{LC}}$2

This dynamic conditioning steers intermediate representations according to the degradation class and the composite/single scenario, without incurring prohibitive parameter overhead or architectural complexity.

4. Selective Progressive Training (SPT)

SPT implements a curriculum and ordering-aware training regime. For composite degradations, it proceeds in reverse order (contrast, blur, noise), so each decoder step targets the removal of a single corruption, with gradients accumulated for all restoration targets before a single parameter update. For each step $\mathcal{C}_{\mathrm{LC}}$3 in $\mathcal{C}_{\mathrm{LC}}$4 (contrast, blur, noise):

1. The model receives input $\mathcal{C}_{\mathrm{LC}}$5 and attends to corresponding $\mathcal{C}_{\mathrm{LC}}$6, scenario prompt $\mathcal{C}_{\mathrm{LC}}$7.
2. For single degradations, the network is supervised toward $\mathcal{C}_{\mathrm{LC}}$8; for composite, toward $\mathcal{C}_{\mathrm{LC}}$9.
3. For composite cases, the output is subjected to $$\mathcal{C}_{\mathrm{LC}}(\mathbf{I}) = \alpha\,\mathbf{I} + \beta$$0 before propagating to the next step, preventing premature gradient flow.

This approach (Algorithm 1 in the cited paper) prevents interference across restoration sub-tasks and supports progressive, stable convergence in highly degraded settings.

5. Benchmarking, Metrics, and Experimental Performance

The HM-TIR dataset underpins both benchmarking and ablation. It comprises 1,503 images ($$\mathcal{C}_{\mathrm{LC}}(\mathbf{I}) = \alpha\,\mathbf{I} + \beta$$1, 8–14$$\mathcal{C}_{\mathrm{LC}}(\mathbf{I}) = \alpha\,\mathbf{I} + \beta$$2m) spanning diverse open-world scenes and corruption types (low contrast, blur, stripe, optical, Gaussian noise), partitioned 80/20 for training/validation. Evaluation metrics include:

• PSNR: $$\mathcal{C}_{\mathrm{LC}}(\mathbf{I}) = \alpha\,\mathbf{I} + \beta$$3 ($$\mathcal{C}_{\mathrm{LC}}(\mathbf{I}) = \alpha\,\mathbf{I} + \beta$$4).
• SSIM: Standard structural similarity over luminance/contrast.
• No-reference IQA: NIMA, MUSIQ (higher better), NIQE (lower better).

Key empirical findings:

Backbone Configuration	PSNR (dB)	SSIM	Relative Gain
Restormer (baseline)	23.28	0.796	—
Restormer + PPFN + SPT	25.32	0.818	+8.76%

• On Iray real-noise dataset: PPFN + SPT achieved NIMA 3.83, MUSIQ 30.91, NIQE 8.47, outperforming all tested alternatives.
• For all single degradations (denoising, deblurring, contrast): consistently highest PSNR/SSIM.
• Integrating PPFN + SPT into FocalNet, UFormer, NAFNet, XRestormer yields consistent across-the-board PSNR gains, e.g., FocalNet increases from 21.22 to 22.63 dB.

6. Functional Insights and Ablation Findings

Explicit injection of dual prompt vectors (degradation and scenario) effectively guides intermediate network states to focus on the appropriate enhancement subtask, allowing for scenario-adaptive restoration. The lightweight FiLM-style modulation:

$$\mathcal{C}_{\mathrm{LC}}(\mathbf{I}) = \alpha\,\mathbf{I} + \beta$$5

affords high-efficiency plug-and-play adaptability across backbone architectures. The SPT regimen's use of reverse-order training and $$\mathcal{C}_{\mathrm{LC}}(\mathbf{I}) = \alpha\,\mathbf{I} + \beta$$6 prevents target interference, crucial for stability on multi-degraded scenarios.

Ablation studies in the original paper show both dual-prompt fusion and SPT are essential; omitting non-linearity in fusion, using only a single prompt, or skipping SPT degrades PSNR by 0.2–0.5 dB, emphasizing the necessity of each innovation (Liu et al., 10 Oct 2025).

7. Current Limitations and Prospective Developments

PPFN currently assumes a fixed three-step degradation order (noise→blur→contrast); real-world TIR imaging may present more complex, non-canonical degradation patterns. The loss function is restricted to per-pixel $$\mathcal{C}_{\mathrm{LC}}(\mathbf{I}) = \alpha\,\mathbf{I} + \beta$$7; perceptual or adversarial losses could plausibly improve realism. Extending prompt quantization to continuous, data-driven estimation of degradation-specific parameters (such as blur kernel shape or noise magnitude) would increase flexibility and further generalizability.

PPFN with SPT establishes a state-of-the-art approach for multi-degradation TIR restoration, outperforming both specialized and all-in-one restorers developed for visible-spectrum imagery, with up to +8.76% improvement in PSNR under complex composite degradations (Liu et al., 10 Oct 2025).