---
title: 'CAAAv2: Category-Aware Auto-Annotation v2'
url: https://www.emergentmind.com/topics/caaav2
type: topic
---

# CAAAv2: Category-Aware Auto-Annotation v2

Searching arXiv for the main paper and directly related works mentioned in the provided data.
CAAAv2, short for **Category-Aware Auto-Annotation v2**, is a paradigm for **pixel-level auto-annotation of manipulated regions** in web-harvested image forgery data. It was introduced to address a central bottleneck in image manipulation localization (IML): the high cost of obtaining pixel-precise masks and the resulting scarcity of high-quality annotated datasets. The method operates on pairs of manually forged images and their authentic originals, decomposes constrained image manipulation localization (CIML) into **Shared Probe Group (SPG)** and **Shared Donor Group (SDG)** sub-tasks, applies a specialized model to each sub-task, and filters unreliable SDG masks with **Quality of Example Selection (QES)**. In the reported pipeline, CAAAv2 produces the annotations used to construct **MIMLv2**, a large-scale dataset with **246,212 manually forged images** and pixel-level masks, and serves as the data-generation substrate for webly supervised IML training [2508.20987].

## 1. Problem setting and design rationale

CAAAv2 is situated in the context of **image manipulation localization**, where the objective is to identify manipulated regions at the pixel level. The motivating observation is that manually forged images are abundant on the web, whereas dense ground truth annotations are not. The framework therefore treats web data as a supervision source and uses CIML as an auxiliary mechanism for annotation generation rather than relying exclusively on handcrafted mask datasets [2508.20987].

The specific diagnosis underlying CAAAv2 is that prior CIML approaches suffer from three limitations. First, they **treat all forged/original pairs uniformly**, despite substantive differences between **copy-move-like object transfer** and **in-place edits**. Second, they **over-fit small CIML training sets**, especially **SDG data**. Third, they **under-utilize the absolute pixel-difference cue** that is especially informative in **SPG** settings. CAAAv2 responds by explicitly splitting CIML into SPG and SDG and assigning each branch a tailored architecture.

In this formulation, SPG denotes pairs that **share their main content in-place**, while SDG denotes pairs related **via object copy**. This categorical split is not merely taxonomic; it determines which cues are considered reliable. In SPG, direct appearance differencing is highly informative. In SDG, by contrast, localization depends on cross-image feature correspondence and denoising of spurious matches. A plausible implication is that CAAAv2 treats CIML less as a single segmentation problem and more as a conditional family of problems whose optimal inductive biases differ by edit category.

## 2. End-to-end auto-annotation pipeline

CAAAv2 takes as input a large collection of **manually forged images and their authentic originals**, harvested from the web, including sources such as **imgur**. The pipeline begins with **self-supervised SPG/SDG classification**, continues with branch-specific mask prediction, and ends with selective filtering of SDG outputs using QES [2508.20987].

The first stage trains a **lightweight binary classifier** using synthetic SPG and SDG pairs generated by random augmentations of unlabeled images. Its role is to determine whether an image pair shares its main content in-place or through copied objects. The description emphasizes that this classifier is “simple and reliable” because it only needs to detect whether **large areas are identical**.

After routing, SPG pairs are processed by **Difference-Aware Semantic Segmentation (DASS)**, while SDG pairs are processed by **Correlation DINO (Corr-DINO)**. The output is a pixelwise manipulation mask $M$ for each pair. Only the SDG branch is passed through the QES filter; **SPG branch masks are inherently high quality—no QES needed**.

The paper also provides a compact algorithmic summary:

```text
Input:  {(I_a^k,I_b^k)}_{k=1…N}   // N web-harvested forged/original pairs
Output: {(I_a^k,I_b^k, M^k)} filtered high-quality annotations

// 1. Train SPG/SDG classifier via self-supervision (offline).
for each k=1…N:
  if classifier(I_a^k,I_b^k)=='SPG':
    M^k = DASS(I_a^k, I_b^k)                // Sec. III-B
  else:
    M^k = CorrDINO(I_a^k, I_b^k)            // Sec. III-C

  if pair belongs to SDG:
    score = QES(M^k)
    if score <= 0.5:   discard example k
    else               keep M^k
  // SPG branch masks are inherently high quality—no QES needed.

Return filtered {(I_a,I_b,M)} as MIMLv2.
```

This pipeline is category-aware in a strict operational sense: the category decision changes both the architecture and the quality-control path.

## 3. SPG branch: Difference-Aware Semantic Segmentation

The SPG branch is implemented as **DASS**, a semantic segmentation pipeline centered on the explicit use of the absolute difference map between the forged image and its authentic counterpart. For a pair $(I_a, I_b)$, CAAAv2 computes

$$
D(x,y) = |I_a(x,y) - I_b(x,y)|.
$$

The tensors are then concatenated as $[I_a, I_b, D]$ and fed into a semantic-segmentation network with a **VAN backbone** and a **Multi-Aspect Denoiser** decoder, producing the per-pixel probability map $M_{\text{spg}}$ [2508.20987].

The design premise is that SPG pairs preserve scene structure while modifying content in place, making the difference map a strong localization cue. The reported best-practice guidance is explicit: **always leverage the absolute difference map in SPG; without it, segmentation IoU drops by more than 20 points**. This is one of the clearest methodological claims in the description and frames DASS as a branch whose performance depends on exploiting low-level discrepancies rather than primarily on high-level matching.

Empirically, on the **IMD20–SPG test**, DASS substantially exceeds the listed baselines. The reported values are **IoU = 0.835, F1 = 0.889** for DASS, compared with **IoU = 0.497** for OTSU binarization, **IoU = 0.578** for DMVN, and **IoU = 0.573** for DMAC [2508.20987]. Within the boundaries of the provided data, this establishes that SPG-specific modeling is a major contributor to CAAAv2’s annotation fidelity.

A plausible implication is that SPG localization is dominated by **difference denoising** rather than **semantic retrieval**, which helps explain why a segmentation architecture with explicit differencing is preferred over a correspondence-heavy formulation in this branch.

## 4. SDG branch: Corr-DINO, aggregation, and denoising

The SDG branch addresses the case in which manipulation involves **object copy**, making raw pixel differencing less reliable. CAAAv2 therefore uses **Corr-DINO**, which is built around frozen **ViT (DINOv2)** features, explicit correlation construction, learnable aggregation, **Feature Super-Resolution (FSR)**, and a **Multi-Aspect Denoiser** [2508.20987].

Feature extraction is performed from the first four DINOv2 layers:

$$
F_{a_i}, F_{b_i} \leftarrow \text{ViT (DINOv2)} \text{ layers } i=1\ldots4.
$$

Correlation features are then formed as

$$
F_{a\_\text{corr}} = [ \text{Corr}(F_{a1},F_{b1}),\ \text{Corr}(F_{a1},F_{a1}) ],
$$

$$
F_{b\_\text{corr}} = [ \text{Corr}(F_{b1},F_{a1}),\ \text{Corr}(F_{b1},F_{b1}) ].
$$

Here, $\text{Corr}(f,g)$ is the **spatial correlation (dot-product)** of two feature maps. The branch then applies learnable aggregation:

$$
F_{\text{aggr}} =
\left[
C_{1\times1}(\text{ReLU}(C_{1\times1}(F_{\text{corr}})))\ ;\
\text{AvgPool}(F_{\text{corr}})\ ;\
\text{MaxPool}(F_{\text{corr}})
\right].
$$

FSR combines aggregated correlation with multi-layer ViT features. Let

$$
F_{\text{concat}} = \text{Concat}(F_1,F_2,F_3,F_4).
$$

Then for scale $i$,

$$
F_{\text{asr}_i} =
\text{Conv}_{1\times1}
\bigl(
\text{Conv}_{1\times1}(F_{\text{concat}})
+
\text{Conv}_{1\times1}(\text{Upsample}(F_{\text{aggr}}, \text{scale}=i))
\bigr).
$$

These representations are fused by the **Multi-Aspect Denoiser** in a top-down manner:

$$
F'_4 = \text{Conv}_{3\times3}([F_{\text{asr}_4}, \text{AvgPool}(F_{\text{asr}_4})]),
$$

$$
F'_n = \text{Conv}_{3\times3}( F'_{n+1} + \text{Conv}_{1\times1}(F_{\text{asr}_n}) ), \quad n=3\to1.
$$

This is followed by **dilated $3 \times 3$ convolutions** and a final **$3 \times 3 \to 1 \times 1$ convolution** for mask logits.

The description identifies one implementation choice as crucial: the **frozen ViT backbone in Corr-DINO**. The stated reason is to reduce overfitting, since **trainable CNN backbones severely overfit SDG data**. On the **IMD20–SDG test**, the reported values are **IoU = 0.702** for SACM (conference), **IoU = 0.744** for Corr-DINO without QES, and **IoU = 0.912** for Corr-DINO with QES, corresponding to an increase of **16.8 points** after filtering [2508.20987]. These results indicate that the SDG branch is effective not only because of correspondence modeling but also because its predictions are subjected to post hoc quality control.

## 5. Quality of Example Selection and dataset distillation

**QES** is the mechanism used to score SDG auto-annotations and discard masks with spurious regions. Given normalized mask probabilities $p_{i,j}$ and two thresholds $T_h$ and $T_l$, CAAAv2 defines

$$
A_{\text{high}} = |\{(i,j) : p_{i,j} > (1-T_h)\}|,
$$

$$
A_{\text{total}} = |\{(i,j) : p_{i,j} > T_l\}|,
$$

and

$$
QES = \frac{A_{\text{high}}}{A_{\text{total}}}.
$$

In practice, the method uses **$T_h = T_l = 1/16$** and retains only examples with **QES $>$ 0.5** [2508.20987].

Conceptually, QES acts as a **no-ground-truth quality filter**. It measures how concentrated the predicted mask mass is in highly confident regions relative to all pixels above a lower threshold. This suggests that masks dominated by diffuse or weak activations are treated as unreliable. The paper explicitly notes a **precision/coverage trade-off**: higher QES yields better masks but fewer retained samples.

The reported threshold study on **SDG on IMD20** is as follows:

| QES thr | kept ratio | resulting IoU |
|---|---:|---:|
| 0.0 | 245% | 0.744 |
| 0.3 | 172% | 0.865 |
| 0.5 | 100% | 0.912 |
| 0.7 | 50% | 0.957 |
| 0.9 | 6% | 0.968 |

The table makes the trade-off explicit: moving from **0.5** to **0.9** raises IoU from **0.912** to **0.968**, but reduces the kept ratio from **100%** to **6%**. Conversely, lowering the threshold preserves many more examples but at sharply reduced quality. Within CAAAv2, the selected operating point is **QES $>$ 0.5**, which balances dataset scale and annotation fidelity [2508.20987].

## 6. Role in MIMLv2 and downstream webly supervised IML

CAAAv2 is the annotation engine used to construct **MIMLv2**. The reported workflow is: **collect approximately 310 K forged/original pairs from imgur**, excluding duplicates and evaluation-set images; **auto-annotate via CAAAv2** to obtain **approximately 370 K raw masks**; apply the **QES filter on the SDG branch**; and obtain a final dataset with **246,212 forged images** and **63,847 authentic originals**, each with a pixel-accurate mask [2508.20987].

The dataset is described as **about 120× larger than IMD20** and as more diverse and modern. The abstract further characterizes MIMLv2 as a **large-scale, diverse, and high-quality dataset**. In the broader system, these annotations are used together with **Object Jitter**, described as a technique that further enhances model training by generating **high-quality manipulation artifacts**, to train **Web-IML**, a model designed to leverage web-scale supervision for the IML task.

The downstream impact reported for the full webly supervised pipeline is substantial. Training **Web-IML** on **MIMLv2 (+Object Jitter)** yields an **overall +31% average IoU gain across 8 benchmarks** and **outperforms prior best by 24.1 IoU points**. The abstract also states that Web-IML **surpasses previous SOTA TruFor by 24.1 average IoU points** [2508.20987]. While these gains belong to the end-to-end framework rather than to CAAAv2 in isolation, they identify the practical significance of the auto-annotation paradigm: CAAAv2 converts abundant but unlabeled web forgeries into training-grade supervision at scale.

## 7. Operational guidance, scope, and limitations

Several implementation recommendations are given explicitly. The **frozen ViT backbone in Corr-DINO** is described as crucial for preventing SDG overfitting. The **self-supervised SPG/SDG classifier** is described as sufficient because it only needs to identify whether large image areas are identical. For SPG, the **absolute difference map** is essential; omitting it reduces segmentation IoU by **more than 20 points**. For SDG, **QES** is effective but should be tuned with awareness of the precision/coverage trade-off [2508.20987].

The stated limitations define the current scope of CAAAv2. **Extremely small or heavily compressed forgeries** can still evade both **DASS** and **Corr-DINO**. In addition, content outside the **SPG/SDG taxonomy**, such as **adversarial manipulations**, may require additional modeling. These limitations are structural rather than incidental: the system is optimized for the two CIML categories it models explicitly, and its quality filter assumes that reliable SDG masks exhibit a characteristic confidence concentration.

A common misconception would be to treat CAAAv2 as a generic weak-supervision recipe for all manipulation types. The provided description does not support that conclusion. Instead, CAAAv2 is specifically a **category-aware** auto-annotation framework whose effectiveness depends on a decomposition into **SPG** and **SDG**, on branch-specific architectural choices, and on post hoc filtering of SDG predictions. A plausible implication is that extending the method to new manipulation regimes would require either a richer taxonomy or additional branch-specific models rather than only more web data.

Taken together, CAAAv2 defines a technically specific route from **web-harvested forged/original pairs** to **pixel-accurate masks**: self-supervised category assignment, **difference-aware segmentation** for in-place edits, **correlation-based frozen-feature localization** for copied-content edits, and **quality-aware filtering** for noisy outputs. In the reported system, this combination is what enables MIMLv2 and the subsequent performance gains in webly supervised IML [2508.20987].

Source: https://www.emergentmind.com/topics/caaav2