OABench: Object Addition Benchmark

• OABench is a synthetic dataset and evaluation platform for text-guided object addition that integrates multi-modal inputs (image, mask, text) for seamless compositing.
• It leverages Diffree, a latent diffusion model enhanced with an Object Mask Predictor, to automatically determine object placement and maintain background consistency.
• The benchmark rigorously assesses added objects using metrics like LPIPS, Local CLIP Score, and unified performance scores, driving improvements in semantic image editing.

The Object Addition Benchmark (OABench) is a purpose-built synthetic dataset and evaluation platform for text-guided object addition in images. It addresses the problem of seamlessly integrating new objects into real-world scenes using only text-based control, minimizing manual intervention and maximizing fidelity in the composited result. OABench enables quantitative and qualitative assessment of models tasked with inserting objects such that background consistency, spatial appropriateness, and semantic relevance to provided textual descriptions are maintained.

1. Dataset Construction and Structure

OABench comprises approximately 74,000 tuples rooted in real-world imagery from established instance segmentation benchmarks such as COCO and LVIS. The construction pipeline removes select objects from original images using PowerPaint, an advanced inpainting model, to create backgrounds that preserve original lighting, texture, and context. Each tuple consists of:

• The original image containing the target object
• The inpainted image where the object is removed
• A binary mask indicating the precise region of the removed object
• A text description specifying the object

This design ensures high-quality, contextually plausible backgrounds and provides standardized inputs for both ground-truth and model-generated additions. The multi-modal nature (image, mask, text) supports training and evaluation of models that must infer object placement, appearance, and integration from minimal guidance.

2. Model Architecture: Diffree

Diffree is a latent diffusion-based model trained on OABench, distinguished by its ability to add objects to images guided solely by textual descriptions. Its architecture is an augmentation of the Stable Diffusion pipeline with an Object Mask Predictor (OMP):

• The Stable Diffusion network operates in the latent space of a pre-trained VAE, with noise progressively reduced through conditional denoising steps informed by the input image and object description (via CLIP encoding).
• The OMP is a deep convolutional module (with ResBlocks and attention mechanisms) that predicts the mask for the new object's location, based on the text and latent features. This mask enables “shape-free” object addition, obviating user-drawn masks.

During training, paired data from OABench allow the joint optimization of both image synthesis and mask prediction objectives:

$$L_{\text{total}} = L_{\mathrm{DM}} + \lambda L_{\mathrm{OMP}}$$

where $L_{\mathrm{DM}}$ is the diffusion model loss and $L_{\mathrm{OMP}}$ is the mask prediction loss.

3. Methodology and Inference Workflow

Diffree's inference process integrates several technical steps:

• Encoding of the target image into a latent feature representation ($z = \mathcal{E}(x)$), followed by the addition of Gaussian noise using the forward SDE process:

$$\tilde{z}_t = \sqrt{\bar{\alpha}_t} \tilde{z} + \sqrt{1 - \bar{\alpha}_t} \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)$$

• The denoising network ($\epsilon_{t,\theta}$) reconstructs the latent to match both the original image and the text guidance, simultaneously predicting the mask via the OMP, which leverages noise-free features:

$$\tilde{o}_t = \frac{\tilde{z}_t - \sqrt{1-\bar{\alpha}_t} \epsilon_{t,\theta}(\ldots)}{\sqrt{\bar{\alpha}_t}}$$

• Integration of classifier-free guidance enhances alignment with the textual object specifications.

This pipeline eliminates the need for manual mask or bounding box input, instead relying on model-inferred placement that adapts to scene context, lighting, and semantic relationships.

4. Evaluation Protocols and Metrics

OABench enables rigorous evaluation of object addition models across multiple criteria:

Metric Category	Concrete Metric/Protocol	Evaluates
Success Rate	Percentage of samples with correct addition	Placement, integration, and relevance
Background Consistency	LPIPS (Learned Perceptual Image Patch Similarity)	Fidelity of inpainted non-object region
Object Appropriateness	GPT-4V spatial and context appropriateness	Sensibility of placement
Object Relevance	Local CLIP Score	Semantic correspondence to text prompt
Quality and Diversity	Local FID	Appearance realism and diversity
Unified Metric	Sum of above (e.g., 35.92 on COCO)	Aggregate performance

Diffree achieves a success rate exceeding 98% (compared to 17–19% for InstructPix2Pix), achieves a 36% reduction in LPIPS (indicating better background preservation), and produces placements evaluated by GPT-4V as highly contextually sensible. A plausible implication is that classifier-free guidance and robust mask prediction are key factors in this performance.

5. Comparative Analysis with Existing Methods

Diffree's “shape-free” approach distinguishes it from mask-guided methods (e.g., PowerPaint) and previous text-guided inpainting systems (e.g., InstructPix2Pix):

• No manual mask or region annotation is required—Diffree predicts placement automatically from text and scene context.
• Background consistency is maintained to a higher degree due to advanced inpainting techniques in the original dataset construction, together with diffusion-based compositing.
• Superior quantitative metrics: Achieving a unified score of 35.92 on COCO (compared to 4.48 for InstructPix2Pix) underlines the efficacy of combined mask and image synthesis training.
• Diffree is robust to iterative additions, avoiding the compounding errors that degrade image quality in alternative pipelines.

A noted limitation is that while Local CLIP Score may be marginally lower on some examples versus mask-guided baselines, overall performance across unified metrics and real-world applicability is higher.

6. Relation to Object Concept Learning and Future Perspectives

The conceptual foundation of OABench is closely related to Object Concept Learning (OCL) benchmarks, as explored in “Beyond Object Recognition: A New Benchmark towards Object Concept Learning” (Li et al., 2022), where multi-level annotations (category, attribute, affordance) and explicit causal structures provide training signals for explainable object reasoning. While OABench targets visual compositionality via image synthesis and mask prediction, a plausible implication is that integrating OCL-style causal explanations and reasoning architectures (such as Object Concept Reasoning Networks) could further enhance semantic plausibility and explanatory transparency in future object addition systems.

Planned directions include expanding the dataset, refining mask prediction modules for greater realism and diversity, leveraging vision-LLMs (e.g., GPT-4V, AnyDoor), and enabling complex iterative editing workflows in domains such as interior or product design.

7. Significance and Research Impact

OABench facilitates the scalable evaluation and training of object addition models that move beyond simple inpainting or category classification, towards contextual, semantically consistent, and text-guided compositional image editing. By enabling automated mask and placement prediction, and rigorous multi-criteria evaluation, OABench has established itself as a reference testbed for research in text-to-image object composition and guided visual editing. The interplay between OABench and emerging models such as Diffree presages further advances in embodied AI and semantic image manipulation.