---
title: 'BrushData-v2: Object-Driven Inpainting Dataset'
url: https://www.emergentmind.com/topics/brushdata-v2
type: topic
---

# BrushData-v2: Object-Driven Inpainting Dataset

BrushData-v2 is a large-scale, object-driven image inpainting dataset introduced to support segmentation-based training and evaluation of diffusion models for image restoration tasks. Constructed as part of the BrushNet framework, BrushData-v2 provides high-quality, open-world semantic masks derived from natural images, enabling rigorous inpainting benchmarks and methodological advances in generative modeling [2403.06976].

## 1. Construction Methodology

BrushData-v2 utilizes the LAION-Aesthetic subset of the LAION-5B corpus, emphasizing high-quality (“aesthetic”) image-text pairs as input. Semantic mask generation is performed using Grounded-SAM (Grounded segment-anything), which processes each image and associated LAION caption to produce open-world object segments. Each mask includes a confidence score, enabling automated filtering: only masks with confidence ≥0.8 are retained. Additional rejection criteria include exclusion of masks occupying less than 1% of the image area and masks exhibiting disconnected “islands.” The retained masks are resized via cubic interpolation to match the latent resolution used in Stable Diffusion (typically $64\times64$). RGB images are center-cropped (or padded) with long edges capped at $1024$ px and subsequently encoded into $4\times$ downsampled VAE latents ($256\times256$ to $64\times64$). Random subsets undergo manual spot checks to further improve dataset quality.

## 2. Dataset Scale, Annotation, and Versioning

BrushData-v2 contains 200,000 image/mask pairs, with original image resolutions distributed approximately uniformly across $256\times256$, $512\times512$, $768\times768$, and $1024\times1024$ pixels, ultimately mapped to $64\times64$ VAE latent grids. The semantic categories, derived from LAION captions and Grounded-SAM outputs, include approximately 25,000 humans/faces, 50,000 animals, 60,000 indoor scenes, and 65,000 outdoor scenes. Each image typically contains 1–3 single-channel binary PNG masks, distinguishing object interiors, exteriors, and multi-part segmentations.

Compared to BrushData-v1—which used synthetic, randomly drawn brush masks over LAION images—v2 introduces realistic object masks and expands coverage to thousands of object classes. Statistical comparisons are summarized below:

|          | #Images | Avg. mask area | #Categories | Avg. mask islands |
|----------|---------|----------------|-------------|------------------|
| v1       | 100,000 | 23%            | 1           | 1                |
| v2       | 200,000 | 29%            | >1,000      | 1.1              |

BrushData-v2 employs automated confidence filtering, area/continuity-based rejection, and spot manual annotation to achieve higher annotation accuracy.

## 3. Data Organization and Access

The dataset is organized for efficient usage and compatibility with generative inpainting workflows:

- **Folder structure:**
  - `images/`: Cropped RGB images in .jpg or .png format.
  - `masks/`: Downsampled single-channel PNG masks.
  - `latents/`: $4\times$ VAE-encoded image latents as .pt or .npy files.
  - `annotations.jsonl`: JSON Lines file, each object encapsulating metadata.

- **Example schema:**
  ```json
  {
    "image_id": "00012345",
    "file_name": "images/00012345.jpg",
    "latent_file": "latents/00012345.npy",
    "mask_file": "masks/00012345.png",
    "caption": "a tabby cat sitting on a wooden floor",
    "objects": [
      { "label": "cat", "confidence": 0.92 },
      { "label": "floor", "confidence": 0.85 }
    ]
  }
  ```
Each record includes LAION captions, object labels, and confidence scores, facilitating fine-grained retrieval and analysis.

## 4. Segmentation-Guided Inpainting Training Protocols

BrushData-v2 is optimized for segmentation-driven inpainting using dual-branch diffusion models. Model input includes the VAE latent of the masked image $z_0^{mask}$ (masked-out pixels zeroed), the binary mask resized to the latent grid $m^{resize}$, and the noise latent $z_t$, constructed as $[z_t ; z_0^{mask} ; m^{resize}] \in \mathbb{R}^{C\times H \times W}$. Training employs feature injection via dual branching [Eq. 6 in the cited paper]:

$$
\epsilon_{t\theta}^{main}(z_t, t, C)_i \leftarrow \epsilon_{t\theta}^{main}(z_t, t, C)_i + w \cdot \mathcal{Z}( \epsilon_{t\theta}^{BrushNet}([z_t, z_0^{mask}, m^{resize}], t)_i )
$$

where $\mathcal{Z}$ denotes a zero-initialized $1 \times 1$ convolution and $w$ is the preservation scale. The training objective follows the standard diffusion reconstruction loss:

$$
L(\theta) = \mathbb{E}_{z_0, \epsilon \sim \mathcal{N}(0, I), t \sim U[1, T]} \left\|\epsilon - \epsilon_{t\theta}(z_t, t, C)\right\|_2^2
$$

Data augmentation includes random horizontal flips, color jittering pre-encoding, and random mask dilation/erosion (±2 pixels) during early epochs.

## 5. Evaluation Protocol on BrushBench

Performance is quantitatively evaluated using BrushBench, which employs seven complementary metrics:

- **PSNR (unmasked region):** $PSNR = 10 \cdot \log_{10}( MAX_I^2 / MSE )$, $MSE = \frac{1}{N} \sum_n (I_n - \hat{Y}_n)^2$
- **LPIPS:** $LPIPS(x, y) = \sum_l \| \phi_l(x) - \phi_l(y) \|_2$ with pretrained VGG features
- **CLIP Similarity:** $\mathrm{CLIPSim}(I, T) = \frac{\langle f_{img}(I), f_{text}(T) \rangle}{\|f_{img}(I)\| \cdot \|f_{text}(T)\|}$
- **ImageReward (IR), Human Preference Score (HPS), Aesthetic Score (AS):** learned perceptual metrics

Average performance (Stable Diffusion v1.5 backbone):

| Inpainting Mode | IR   | HPS  | AS    | PSNR  | LPIPS   | MSE   | CLIP   |
|-----------------|------|------|-------|-------|---------|-------|--------|
| Inside          | 12.64↑|27.78↑| 6.51↑|31.94↑|0.0080↓  |18.67↓ |26.39↑  |
| Outside         | 10.88↑|28.09↑| 6.64↑|27.82↑|0.0225↓  | 4.63↓ |27.22↑  |

Head-to-head comparisons with baselines (e.g., BLD, SDI, HDP, PP, CNI) affirm that models trained on BrushData-v2 consistently rank first or second across all seven metrics.

## 6. Practical Guidance and Best Practices

Recommended training regime comprises 430,000 iterations on $8 \times$ V100 GPUs (~3 days), batch size of 16, learning rate $10^{-5}$ (no weight decay), text guidance scale of 7.5 at inference with 50 DDIM steps, and preservation scale $w$ initiated at 1.0 and reduced to 0.2–0.5 for coarser style control.

Common pitfalls include mask resizing artifacts (necessitating cubic interpolation and post-blur), loss of high-frequency detail in VAE reconstruction (mitigated by blurred-mask pixel-space copy-paste), and text-image mismatch (text cross-attention disabled in the BrushNet branch for purity of masked feature embedding). Domain-shift issues arise if switching the base model checkpoint; for example, domain adaptation toward anime can affect unmasked region style.

A plausible implication is that BrushData-v2, through its emphasis on real-object segmentation and rigorous annotation protocols, addresses critical shortcomings of synthetic mask datasets, catalyzing robust inpainting research and more reliable evaluation [2403.06976].

Source: https://www.emergentmind.com/topics/brushdata-v2