---
title: 'CoTIR-Bench: Chain-of-Thought Restoration Benchmark'
url: https://www.emergentmind.com/topics/cotir-bench
type: topic
---

# CoTIR-Bench: Chain-of-Thought Restoration Benchmark

CoTIR-Bench (Chain-of-Thought Image Restoration Benchmark) is a large-scale, richly supervised dataset designed for the training, validation, and benchmarking of universal image restoration models that leverage explicit chain-of-thought (CoT) reasoning. Introduced in "Universal Image Restoration via Internalized Chain-of-Thought Reasoning," CoTIR-Bench provides end-to-end paired data with comprehensive intermediate reasoning traces, supporting both classic and composite image degradations at scale [2606.17557].

## 1. Scope, Purpose, and Design Objectives

CoTIR-Bench was constructed to address the limitations of traditional and multi-round restoration benchmarks, particularly with respect to weak modeling of interactions between multiple degradations and the computational costs of chaining specialized models. The benchmark aims to:

- Enable large-scale fine-tuning of generative restoration models using rich, intermediate reasoning supervision.
- Span both single degradation types (e.g., deblurring, denoising) and composite synthetic or real-world degradations.
- Provide a unified, standardized test set for consistent comparison across restoration methods regarding perceptual quality, fidelity, and reasoning coherence.

## 2. Dataset Composition and Degradation Types

CoTIR-Bench comprises approximately 5.20 million training image pairs, a 100,000-sample validation split, and a 2,000-image representative test split uniformly drawn from diverse degradation categories. The benchmark encompasses the following degradation types:

- **Single degradations:** Low-Light Enhancement (LLE), Light Adjustment (LA), Dehazing (DH, NHDH), Deraining (DR, DRD), Desnowing (DS), Reflection Removal (RR), Deshadowing (DSH), Deblurring (DB), Super-Resolution (SR), Denoising (DN), Underwater Enhancement (UE), Deflaring (DF), Dewatermarking (DW), DSLR Conversion (DSLR), Coloring (C), JPEG Artifact Repair (JPEG).
- **Composite degradations:** Randomly synthesized multi-degradation scenarios (e.g., LLE + DN, DR + JPEG, DH + SR + DB), reflecting practical restoration challenges.

| Split        | Number of Samples          | Coverage              |
|--------------|---------------------------|-----------------------|
| Training     | ≈5,200,000                | Single/composite      |
| Validation   | 100,000                   | Hyperparameter tuning |
| Testing      | 2,000                     | Uniform by category   |

The degradation distribution in the training split includes Deraining (DR, DRD) at 17%, Deblurring (DB) at 12%, Dehazing (DH, NHDH) at 11%, Denoising (DN) at 10%, with all other types between 2–8%, and composite mixtures accounting for 25%. Severity levels for each degradation are uniformly sampled from 1–5 ($\text{mean} \approx 3.0$, $\sigma \approx 1.2$).

## 3. Construction Methodology and Annotation Process

The dataset synthesizes its training pairs by re-processing over 60 publicly available restoration datasets, cropping/resizing all images to $512 \times 512$ pixels. 

For each $(x, y, d_1, ..., d_k)$ sample (with $x$ as degraded input, $y$ as ground-truth, and $d_i$ as text tokens indicating degradations), structured CoT traces are generated as follows:

**Automated CoT Trace Generation:**
1. **Instruction Preparation:** A "precise prompt" ($\varphi_\mathrm{pre}$) specifies the degradations and instructs on restoration, feature description, and planning.
2. **Chain-of-Thought Generation:** The triplet $C = \{c_s, c_d, c_p\}$ is output by Qwen2.5-VL, where $c_s$ describes sharp features of $y$, $c_d$ identifies and localizes degradations in $x$, and $c_p$ describes an ordered restoration plan.
3. **Prompt Augmentation:** A "vague prompt" ($\varphi_\mathrm{gen}$) is generated via templates or GPT-5 Codex for diversity.
4. **Filtering:** CLIP-based consistency checks (score threshold 0.25) enforce sample relevance.

The dataset synthesis logic is:

```python
for each dataset D in public_collections:
    for each (x,y) in D:
        d_types = identify_degradations(D)
        φ_pre = format_precise_prompt(d_types)
        C = Qwen2.5_VL.generate_CoT(x,y,φ_pre)
        if clip_score(x,y,φ_pre) >= τ:
            φ_gen = sample_vague_prompt()
            store {
                x, y,
                d_types, severity_levels,
                prompts: {pre:φ_pre, gen:φ_gen},
                CoT: {c_s, c_d, c_p}
            }
split_data(train=5.2M-100K, val=100K, test=2000)
```

## 4. Data Format, Annotation, and Tokenization

Each sample is provided as:

- `images/ID_x.png`: degraded image (PNG, uint8)
- `images/ID_y.png`: ground-truth clean image (PNG, uint8)
- `meta/ID.json`: UTF-8 JSON metadata, containing:
  - `"id"`
  - `"degradation_types"` (e.g., `["DR", "JPEG"]`)
  - `"severity"` (per-type, 1–5)
  - `"prompt_precise"` and `"prompt_vague"` (text)
  - `"cot"` with fields `"c_s"`, `"c_d"`, `"c_p"`

Tokenization for all text fields (prompts and chain-of-thought substeps) uses T5's SentencePiece vocabulary (32,000 subword units). During model training, substeps are processed by the CoT Adapter, projecting them into three separate embedding heads.

**Mathematical encoding of CoT:** For each triplet $C = \{c_s, c_d, c_p\} \in (\mathbb{R}^d)^3$ stored with tolerance bounds $\delta_s, \delta_d, \delta_p$, the restoration model predicts $\hat{C}$ and enforces
$$
\|\hat{c}_i - c_i\|_2^2 \leq \delta_i, \quad i \in \{s, d, p\}
$$

**Trace lengths:** Mean tokens per sub-step: $c_s$: $\mu = 12$ ($\sigma = 5$), $c_d$: $\mu = 15$ ($\sigma = 6$), $c_p$: $\mu = 18$ ($\sigma = 7$). 90% of traces contain 5–30 tokens per sub-step, maximum observed is 60.

## 5. Evaluation Protocols and Benchmarking

CoTIR-Bench defines a rigorous benchmarking procedure, supporting both full-reference and no-reference evaluation metrics.

**Full-Reference Metrics:**
- PSNR (dB)
- SSIM
- LPIPS (lower is better)

**No-Reference Metrics:**
- CLIP-IQA+
- Q-Align (PSNR-like alignment in CLIP space)
- LIQE
- MACLIP

**Benchmarking Procedure:**
- Test set of 2,000 images, covering all degradation categories.
- All metrics reported on the entire set.
- CPU+GPU runtime measured on 512×512 inputs and averaged.
- Baselines include: PromptIR, OneRestore, InstructIR, DA-CLIP, HYPIR, LucidFlux, AutoDIR, DiffUIR, UniRestore, AgenticIR, Q-Agent (either with public checkpoints or retrained on CoTIR-Bench).

Submissions to the leaderboard are made by uploading restored images that match test filenames; a blind re-evaluation script outputs a JSON with all metric scores. The code repository provides all scripts and documentation.

## 6. Statistical Distribution and Data Characteristics

- **Degradation Distribution:** Deraining and derain-drop 17%, deblurring 12%, dehazing/non-heterogeneous dehazing 11%, denoising 10%; all other classes 2–8%; 25% of samples have composite mixtures (two or more degradations).
- **Severity Levels:** Uniform sampling from 1–5 for every degradation; average severity is approximately 3.0 (standard deviation 1.2).
- **CoT Trace Lengths:** Tokens per sub-step (mean ± standard deviation) — $c_s$: 12 ± 5; $c_d$: 15 ± 6; $c_p$: 18 ± 7.
- **File Organization:** Image data in PNG format, metadata in UTF-8 JSON; split files for `train`, `val`, and `test` enumerate all sample IDs.

## 7. Access, Licensing, and Usage Terms

CoTIR-Bench is distributed under a CC BY-NC-SA 4.0 license, limiting use to non-commercial, share-alike applications. The dataset and codebase are available at https://github.com/gy65896/CoTIR. Split files (`train.txt`, `val.txt`, `test.txt`) support reproducibility and controlled evaluation.

The recommended citation is:
> Guo et al., “Universal Image Restoration via Internalized Chain-of-Thought Reasoning,” IEEE TPAMI, 2025.

Researchers are encouraged to use CoTIR-Bench as a standard for training, validating, and benchmarking universal CoT-driven restoration models under consistent and richly supervised settings [2606.17557].

Source: https://www.emergentmind.com/topics/cotir-bench