HiddenObjects: Scalable Diffusion-Distilled Spatial Priors for Object Placement

Abstract: We propose a method to learn explicit, class-conditioned spatial priors for object placement in natural scenes by distilling the implicit placement knowledge encoded in text-conditioned diffusion models. Prior work relies either on manually annotated data, which is inherently limited in scale, or on inpainting-based object-removal pipelines, whose artifacts promote shortcut learning. To address these limitations, we introduce a fully automated and scalable framework that evaluates dense object placements on high-quality real backgrounds using a diffusion-based inpainting pipeline. With this pipeline, we construct HiddenObjects, a large-scale dataset comprising 27M placement annotations, evaluated across 27k distinct scenes, with ranked bounding box insertions for different images and object categories. Experimental results show that our spatial priors outperform sparse human annotations on a downstream image editing task (3.90 vs. 2.68 VLM-Judge), and significantly surpass existing placement baselines and zero-shot Vision-LLMs for object placement. Furthermore, we distill these priors into a lightweight model for fast practical inference (230,000x faster).

• The paper introduces a scalable framework that extracts explicit spatial priors from diffusion models using iterative inpainting, verification, and preference ranking.
• The methodology leverages the HiddenObjects dataset with 27 million annotations, outperforming human and baseline methods in spatial accuracy and realism.
• It distills these priors into a DETR-style transformer model, achieving a 230,000× speedup and significant GPU memory reduction for real-time applications.

HiddenObjects: Scalable Diffusion-Distilled Spatial Priors for Object Placement

Motivation and Background

Object placement in natural scenes is conditioned by strong spatial priors that reflect semantic and compositional regularities—pizzas are placed on tables, benches commonly appear at ground level, and kites are positioned near the sky. Existing datasets for learning spatial priors are either manually annotated (limited scalability and diversity) or constructed via object removal pipelines, which introduce background artifacts that compromise model generalization and facilitate shortcut learning. Recent diffusion models trained on internet-scale data have demonstrated implicit spatial prior knowledge, but this has not been distilled into explicit, scalable datasets or models for practical insertion tasks.

Figure 1: Background artifacts in PIPE object removal pipelines frequently induce visual traces, facilitating shortcut learning and damaging downstream placement generalization.

Proposed Framework: Extraction of Explicit Spatial Priors

The paper introduces an automated framework to distill class-conditioned spatial priors from text-conditioned diffusion models via exhaustive inpainting and verification. Candidate bounding boxes on real background images are iteratively inpainted with target objects using diffusion pipelines, then validated for class-consistent insertions, and ranked via reward models trained for human preference alignment.

Figure 2: The spatial prior extraction pipeline, combining diffusion-based inpainting, verification, and preference ranking to produce dense spatial prior heatmaps.

For each background/object pair:

• The inpainter (ControlNet-adapted Qwen-Image) synthesizes candidate insertions.
• The verifier (Grounded-SAM-2) detects and confirms presence/class-consistency.
• The ranker (ImageReward) measures human-aligned plausibility.
• Valid placements are aggregated into explicit spatial prior distributions.

HiddenObjects Dataset Construction

Leveraging the pipeline, the authors construct the HiddenObjects dataset with 27 million placement annotations across 27k diverse scenes. Positive (plausible) and negative (implausible/failure) placements are densely evaluated and ranked, capturing a spectrum of spatial preferences per class and scene. The dataset surpasses previous benchmarks in terms of spatial density, background quality, scalability, and human-aligned annotation.

Figure 3: Aggregated spatial priors (heatmaps) for multiple classes, visualizing natural semantic and photographic biases such as “bench” at the bottom or “kite” at the top of images.

Component Analysis: Inpainter and Ranker Selection

Through empirical analysis, ControlNet-based inpainters demonstrate superior enforcement of scene context, refusing to produce objects in implausible regions or respecting physical constraints (depth, perspective), while U-Net and cross-attention inpainters frequently generate semantically inconsistent artifacts. For ranking, ImageReward yields highest average precision and human alignment, outperforming CLIP Score and aesthetic predictors.

Figure 4: Verifier comparison shows ImageReward and VisionReward select placements consistent with scene context, unlike CLIP Score which often ignores global realism.

Figure 5: Inpainter comparison reveals ControlNet variants reliably enforce scene constraints, rejecting implausible insertions or producing context-aware composites.

Quantitative and Qualitative Evaluation

The spatial priors distilled with this pipeline dramatically outperform human annotations and established baselines for object placement in downstream image editing tasks, as measured by ImgEdit-Judge (Qwen2.5-VL-based evaluation). The method achieves average scores of 3.90 (vs. 2.68 for sparse human annotation and comparable performance for random bounding boxes), indicating greater realism and semantic plausibility.

Figure 6: Comparison of object insertion methods on OPA backgrounds, highlighting superior placement and photorealism using HiddenObjects spatial priors.

Analysis of Systematic Dataset Biases

Spatial prior distributions from HiddenObjects are markedly less center-biased than classic object-centric datasets (COCO, VOC, PIPE), providing broader spatial coverage and distribution variance. However, due to generative model limitations, placements on very small regions (relative area $<1\%$) are underrepresented, reflecting the pixel resolution bottleneck and perspective constraints of current diffusion backbones.

Figure 7: Distribution of object centers shows reduced center-crop bias in HiddenObjects relative to COCO, VOC, ImageNet, and PIPE.

Distillation: Fast Model for Placement Proposal

To enable real-time inference, the spatial priors are distilled into a DETR-style transformer model conditioned on scene and object classes, producing ranked bounding box proposals. The distilled model achieves a 230,000$\times$ speedup and 300$\times$ reduction in GPU memory (3.77 ms/image, 188.9 MB RAM vs. 14.49 minutes/image, 65.9 GB RAM for full pipeline), making it practical for downstream vision tasks.

On benchmark evaluation, the distilled model outperforms zero-shot VLMs (Qwen2.5-VL, InternVL3, LLaVA) and profiled object placement models (PlaceNet, GracoNet, BootPlace, TerseNet), achieving 56.6% mAP and 62.9% IoU50@1 on the HiddenObjects test set, with robust generalization to OPA backgrounds.

Figure 8: Evaluation of object placement strategies on OPA backgrounds, demonstrating superior composite realism for the top-ranked proposals from HiddenObjects.

Practical and Theoretical Implications

Explicit spatial priors, densely distilled from diffusion models, present foundational advances for controllable image editing, scene understanding, and robust data augmentation. The framework circumvents manual annotation bottlenecks and artifact-laden synthetic pipelines, aligning model placement proposals with human spatial expectations and compositional semantics. The distillation yields tractable models for deployment in interactive editing, context-aware compositionality, and outlier detection. The observed scale and bias limitations highlight avenues for advancing generative resolution and perspective fidelity.

Future Directions

Enlarging the dataset to further diminish compositional and photographic biases, advancing generative fidelity for minuscule object insertions, and leveraging multimodal constructs (text, depth, segmentation) will improve robust spatial prior modeling. Integrating explicit spatial prior models with vision-language systems may unlock zero-shot placement prediction and context-conditioned scene augmentation.

Figure 9: Inpainting speedup analysis quantifies computational savings by early detection of failed placements in diffusion denoising, achieving $2.4\times$ reduction for 81% recall.

Figure 10: Bounding box proposal grid visualizes spatial coverage and pruning for efficient placement sampling.

Conclusion

The HiddenObjects framework and dataset establish a scalable methodology for learning explicit, human-aligned spatial priors for object placement in natural scenes by distilling knowledge from large-scale diffusion models. Robust evaluation shows significant gains in downstream image editing quality, spatial diversity, and inference efficiency over manual annotations, legacy datasets, and zero-shot VLM systems. The approach has far-reaching implications for scene compositionality, image synthesis, and the practical deployment of vision models demanding explicit spatial context.