---
title: 'LayerBench: RGBA Inpainting Benchmark'
url: https://www.emergentmind.com/topics/layerbench
type: topic
---

# LayerBench: RGBA Inpainting Benchmark

LayerBench is a standardized benchmark dataset and evaluation protocol designed to rigorously assess transparent RGBA image inpainting methods, where both color (RGB) and transparency (alpha channel) restoration are required within user-defined image masks. Originating alongside the Trans-Adapter framework, LayerBench addresses the deficiencies of prior benchmarks, which were restricted to fully opaque RGB images, by introducing tasks and metrics specific to layered image workflows ubiquitous in game assets, compositing, animation, presentations, and related contexts. LayerBench is notable for introducing the first non-reference metric dedicated to quantifying the quality of reconstructed transparency (alpha) edges, and for providing a well-curated set of RGBA images where inpainting masks frequently cross transparency boundaries [2508.01098].

## 1. Objective and Scope

LayerBench was developed to support the evaluation of inpainting approaches on transparent images, a regime where models must simultaneously predict plausible RGB content and spatially coherent alpha mattes. This requirement is critical in workflows where alignment between RGB and alpha determines whether composited results retain crisp, artifact-free boundaries. Traditional inpainting datasets such as EditBench and BrushBench are restricted to fully opaque RGB images and do not stress test reconstruction fidelity at semi-transparent boundaries or in compositional contexts where alpha edge quality is paramount. LayerBench systematically addresses this gap by including inpainting masks that specifically cross transparency boundaries, thereby challenging models to restore both appearance and transparency consistency [2508.01098].

## 2. Dataset Structure and Composition

LayerBench consists of 800 high-resolution (1024×1024) RGBA samples partitioned equally into two subsets:

- **LayerBench-Natural (400 samples):** Curated from PNG stock repositories and established matting datasets (DIM, Semantic Matting, Distinctions-646), encompassing real-world objects, line drawings, cartoons, and artwork elements.
- **LayerBench-Generated (400 samples):** Split between 200 held-out images from the MAGICK synthetic dataset (selected for aesthetic quality) and 200 synthetic images generated by LayerDiffusion+SDXL pipelines using ChatGPT-driven prompts.

Each sample includes:
- Ground-truth RGBA image.
- User-defined binary inpainting mask (varied: freehand, geometric, Bézier, object-shaped).
- “Mask-simple” semantic annotation: concise human label of the masked region.
- Full descriptive caption for the scene (LLaVA-generated).
- Implicit compatibility for compositing onto arbitrary backgrounds; white and black background composites are provided or easily derived.

No additional augmentation is applied; all images are standardized to 1024×1024 resolution and accompanied by their respective semantic and geometric annotations [2508.01098].

## 3. Data Acquisition and Preprocessing

Natural subset images are sourced through purchase or download from high-resolution PNG repositories, filtered for quality (minimum long side ≥600px, absence of jagged edge artifacts), and center-cropped or padded for uniformity. The generated subset leverages chroma-keyed alpha channels from MAGICK and new data synthesized with LayerDiffusion (prompted via ChatGPT, ensuring diversity and complexity). Each image is paired with a semantic prompt generated by LLaVA and a separate “mask-simple” label. As part of the focus on evaluation reproducibility, no augmentation or resolution change is performed beyond initial curation; this ensures consistent, high-fidelity test conditions [2508.01098].

## 4. Evaluation Methodology

### Standard Image-Quality and Semantic Metrics

Perceptual and semantic metrics are systematically collected on composited outputs for both white and black backgrounds, using:

- **Aesthetic Score (AS):** Predicted by a CLIP-based aesthetic scoring model (~1–10, higher is better).
- **LPIPS (Learned Perceptual Image Patch Similarity):** Measures perceptual distance on the masked region (lower is better).
- **CLIP Similarity:** Computes text-image alignment between the inpainted region and the provided prompt (higher is better).

The evaluation workflow composites each RGBA result onto both white and black backgrounds because RGB values in zero-alpha regions can be arbitrary; metrics are averaged across both composites for robustness. Unmasked (input-preserved) areas are retained in composites to directly isolate the inpainting region [2508.01098].

### Alpha Edge Quality (AEQ) Metric

LayerBench introduces a novel non-reference Alpha Edge Quality (AEQ) metric to objectively assess sharpness and alignment at reconstructed RGB/alpha boundaries:

1. **Concatenated Input:** An 8-channel tensor $I_{\text{concat}} = [I_w; I_b; \alpha; M_e]$ comprising RGB composites on white ($I_w$) and black ($I_b$), predicted alpha channel ($\alpha$), and computed binary edge mask $M_e$.
2. **CNN-Based Classifier ($\mathcal{F}$):** Lightweight CNN outputs per-pixel likelihood $p_{\text{low}}(x,y)$ of low-quality edge.
3. **Evaluation Region:** AEQ is computed over edge pixels within the inpainting mask ($\mathcal{M}_e$), extracted by Canny($\alpha$, threshold=20) and dilated.
4. **Formula:**
   $$
   AEQ = 1 - \frac{1}{|\mathcal{M}_e|} \sum_{(x, y) \in \mathcal{M}_e} \mathcal{F}(I_w, I_b, \alpha, \mathcal{M}_e)_{x, y}
   $$
   AEQ ranges from 0 to 1, with higher values indicating sharper and better-aligned boundaries. The CNN classifier $\mathcal{F}$ is trained using synthetic degradations and cross-entropy loss with class-imbalance weights prioritizing edge fidelity [2508.01098].

## 5. Baseline Methods and Benchmark Results

LayerBench reports benchmarking results across single-pass and two-stage inpainting pipelines:

- **Single-Pass RGBA Inpainting:** Exemplified by Trans-Adapter, which directly predicts the inpainted RGBA patch using cross-domain self-attention and spatial alignment.
- **Two-Stage Pipelines:** Conventional approaches applying diffusion model-based inpainting on RGB, followed by separate matting (ZIM) or dichotomous segmentation (U²-Net, BiRefNet) for alpha estimation.

Evaluation regimes cover both “pure noise” initialization (mask fully randomized) and “blended noise” (mask blended with mostly noise, minor original input). Summary statistics across LayerBench indicate:

- **Aesthetic Score (AS):** All methods achieve around 6.0; Trans-Adapter is often marginally higher.
- **LPIPS:** Trans-Adapter consistently attains lower scores (e.g., 0.0591 vs. 0.0697 for ZIM+SD1.5).
- **CLIP Similarity:** Approximately 27.0 across methods, denoting robust prompt adherence.
- **AEQ:** Trans-Adapter reaches ≥0.985, outperforming two-stage pipelines (down to 0.950 for U²-Net).

Observed failure modes for two-stage systems include jagged alpha boundaries (causing compositing artifacts), RGB/alpha misalignments (introducing color fringes), and lost fine structure in semi-transparent regions. The single-pass approach using Trans-Adapter demonstrates superior RGBA coherence [2508.01098].

## 6. Usage Protocol and Reproducibility Guidelines

LayerBench, along with its complete codebase, annotations, and evaluation tools—including the AEQ classifier—is publicly available for research utilization. A recommended evaluation protocol includes:

1. Load the RGBA sample and corresponding mask.
2. Apply the inpainting method to produce $(R', G', B', \alpha')$.
3. Composite results on white and black backgrounds, yielding two RGB outputs.
4. Replace unmasked (unmodified) regions with the original to isolate reconstruction error.
5. Compute AS, LPIPS, and CLIP Similarity across both composites and average results.
6. Calculate AEQ using the provided CNN classifier and derived edge mask.

For best practice, maintain evaluation resolution at 1024×1024, apply both initialization regimes ("pure noise" and "blended noise"), report all four key metrics, and design masks that explicitly overlap alpha boundaries for stress-testing edge alignment [2508.01098].

## 7. Impact and Research Significance

LayerBench establishes a new evaluation paradigm for transparent inpainting, enabling meticulous measurement of semantic fidelity, perceptual quality, and, distinctively, alpha edge alignment. By introducing challenging benchmarks and a transparent, reproducible evaluation standard, LayerBench facilitates equitable comparison between approaches, drives innovation in spatially coherent inpainting models, and addresses a longstanding evaluation gap in image layer and compositing workflows [2508.01098].

Source: https://www.emergentmind.com/topics/layerbench