HomeDiffusion: Zero-Shot Object Customization with Multi-View Representation Learning for Indoor Scenes

Abstract: Recently, zero-shot object customization generation methods have rapidly developed and shown tremendous potential for applications. For instance, in the e-commerce domain, consumers can observe the visual effect of furniture placed within their personal living spaces or clothes worn on their own bodies. Many existing approaches perform object customization generation based on diffusion models and extracted reference object features. However, the generated object significantly diverges from the original reference object in details such as patterns and curves. Particularly for asymmetrical reference objects, the absence of comprehensive multi-viewpoint information prevents the generation of object poses that harmonize with the background scene. To address these shortcomings, we have constructed a novel dataset comprising multi-angle images of furniture and indoor scenes. Based on diffusion models, we introduce HomeDiffusion, which can leverage multi-viewpoint images of the same reference object to accurately generate visually harmonious object poses within specified areas of the background scene. During the diffusion process, we further extract high-fidelity details of the reference object and perform cross-attention with the noise latents in the latent space, thereby ensuring the preservation of details in the customized object generation. Extensive qualitative and quantitative experiments demonstrate that our method achieves superior performance over other existing zero-shot as well as few-shot object customization approaches.

• The paper introduces HomeDiffusion, a zero-shot framework that combines multi-view representation learning with pixel-aligned latent cross-attention to preserve object identity while generating viewpoint-consistent indoor scenes.
• The method achieves 89.4 CLIP and 86.2 DINO scores on ZOC-Indoor-Eval, outperforming AnyDoor and Paint-by-Example while also surpassing few-shot DreamBooth across one, three, and five reference views.
• The results show that additional reference views improve fidelity with only about 5% inference overhead, but real-world validation remains limited by synthetic training data, small user studies, and reliance on author-created benchmarks.

HomeDiffusion is a zero-shot object customization framework for indoor scenes that addresses two persistent failure modes in exemplar-based image editing: the inability to synthesize object viewpoints that are geometrically consistent with the background when the object is asymmetric, and the loss of fine-grained identity details during diffusion-based inpainting. Built on Stable Diffusion V2.1, the method conditions generation on multi-view reference images of a furniture item, a masked background, and a composited image, and it introduces pixel-aligned cross-attention to transfer high-fidelity details in latent space (2606.29828).

Motivation and problem setting

The task is defined as inserting a reference object into a user-specified region (a binary mask) of a background image such that the object harmonizes in layout, lighting, perspective, and spatial relationships while preserving texture, shape, and distinctive features. The authors identify two challenges in prior work. First, single-view representations are fundamentally insufficient for asymmetric objects: when the background orientation differs from the reference viewpoint, methods such as AnyDoor tend to "paste" the object without adjusting its pose, producing discordant perspective. Second, identity encoders trade off fidelity and detail: Paint-by-Example's CLIP-based conditioning, which uses only the class token, "tends to ignore the high-frequency details" (a concession quoted from that paper itself), while DINO-V2-based conditioning improves fidelity but still lacks viewpoint reasoning. The authors argue indoor scenes are the natural testbed because walls and furniture layouts encode explicit depth and viewpoint cues.

Data construction

Training data is derived from 3D-FRONT, a corpus of professionally designed synthetic indoor scenes. The authors render scene images from multiple camera angles, segment the target furniture to obtain background-removed multi-view object images, and render furniture-free backgrounds from the same cameras to recover lighting and shadow effects. The resulting dataset contains over 180k scene images spanning eight furniture categories (bed, chair, coffee table, floor lamp, nightstand, sofa, TV cabinet, wardrobe), with more than 60% of furniture items having five or more distinct viewpoints. A key limitation is that all data is synthetic; real-world domain gap is not explicitly evaluated, though generalization results on Viton-HD and outdoor scenes partially mitigate this concern.

Method

HD visual encoder

To encode a $448\times448$ reference image without the token explosion of naively raising DINO-V2 resolution, the HD visual encoder processes the image in two paths: a global path that resizes to $224\times224$ and extracts class plus patch tokens, and a local path that splits the image into four non-overlapping $224\times224$ patches and encodes each. The five sets of patch tokens ($5\times256\times1536$) are concatenated and projected via an MLP back to $256\times1536$, retaining only the global class token. This yields high-resolution detail at a fixed token budget, an approach inspired by high-resolution VLM designs such as InternLM-XComposer2-4KHD.

Stage 1: Multi-view Object Representation Learning (MORL)

MORL trains the model to predict a held-out viewpoint $x_r^j$ of a furniture item given the remaining views $\{x_r^i\}_{i\neq j}$. Each condition view is encoded by the HD visual encoder, stacked, and mapped through an MLP to a $257\times1024$ multi-view token $C_o$ injected into the U-Net cross-attention layers. Because the target viewpoint is otherwise unconstrained, a pre-trained ControlNet supplies the Canny edge map of the target view as layout guidance. Two training strategies improve robustness: sampling with replacement (allowing duplicate views) and, with 10% probability, collapsing the condition set to a single view — the latter enabling inference with one or many reference images. An additional 10% probability of zeroing the condition set supports classifier-free guidance. Ablations attribute a DINO Score gain of +1.6 (one view) and +1.2 (five views) to this stage, and qualitative results on a complex furniture item show that single-view conditioning produces color changes and hollow panels under large viewpoint shifts, while three-view conditioning resolves them.

Stage 2: Background-driven Object Customization Learning (BOCL)

In BOCL, the MLP is frozen to preserve multi-view representations and reduce training difficulty. The model receives (1) the masked background $x_b \odot m$ and (2) a composited image $224\times224$0 formed by pasting a reference view into the editable region, which carries high-fidelity object detail. The composite is encoded by the VAE so that it is pixel-aligned with the generation target in latent space. A new U-Net encoder extracts 13 levels of feature maps $224\times224$1 from $224\times224$2, which are fused with the corresponding generator feature maps $224\times224$3 via level-wise pixel-aligned cross-attention rather than direct addition — the authors note that direct addition would be harmful because the pasted object's viewpoint deviates from the target. A "Background Drop" strategy (discarding background loss with 50% probability) focuses learning on the editable region. This pixel-aligned cross-attention contributes +0.6 (one view) and +1.2 (five views) DINO Score over composited-image guidance alone.

Experimental results

The authors construct the ZOC-Indoor-Eval benchmark (160 scenes: 8 categories × 20 scenes, each with five-view references and a scene image whose furniture viewpoint differs from all references) and evaluate with CLIP Score and DINO Score following DreamBooth and AnyDoor protocol.

Method	CLIP Score	DINO Score
Paint-by-Example	83.4	78.5
AnyDoor	87.1	83.3
HomeDiffusion	89.4	86.2

On ZOC-Indoor-Eval with a single-view reference, HomeDiffusion exceeds AnyDoor by 2.3 CLIP and 2.9 DINO points. Against the few-shot DreamBooth baseline, which requires per-object test-time fine-tuning, HomeDiffusion achieves DINO Scores of 86.2, 87.5, and 88.1 with one, three, and five views respectively — surpassing DreamBooth (69.8, 81.3, 85.5) at every view count, with monotonically increasing gains from additional views confirming effective multi-view information utilization. A small-scale user study (6 raters, 900 images) reports HomeDiffusion at 3.45 Fidelity / 3.51 Harmony versus AnyDoor's 2.70 / 2.46 and DreamBooth's 2.18 / 2.57, though six raters is a small sample and no statistical significance testing is reported. On the Viton-HD virtual try-on benchmark, HomeDiffusion again leads (CLIP 82.2, DINO 60.3 vs. AnyDoor's 81.8 / 59.3), though the margin is narrow. Inference overhead is modest: the extra encoder requires only a single forward pass, adding roughly 5% to a 20-step, 512-resolution sampling run.

Limitations and open questions

Several caveats bear on the reported results. The training corpus is entirely synthetic (3D-FRONT renders), and while the authors demonstrate qualitative generalization to outdoor scenes and quantitative transfer to virtual try-on, the effect of the sim-to-real gap on furniture fidelity in real photographs is not quantified. The user study involves only six participants, and no variance or significance analysis accompanies the Fidelity/Harmony scores. The evaluation metrics (CLIP-I and DINO) measure global image similarity and may not fully capture fine-grained detail preservation or geometric plausibility of synthesized viewpoints. The benchmark itself is introduced by the same authors, so independent evaluation on third-party indoor customization benchmarks remains an open question. Finally, the method relies on ControlNet edge maps during MORL training; how the learned multi-view representation behaves when the target viewpoint's layout is only loosely specified at inference (e.g., from a freehand mask alone) is not systematically studied.

Conclusion

HomeDiffusion combines a token-efficient HD visual encoder, a self-generative multi-view representation learning stage, and a background-driven customization stage with pixel-aligned latent cross-attention to achieve zero-shot object insertion that is both detail-preserving and perspective-harmonious. It outperforms zero-shot baselines (Paint-by-Example, AnyDoor) and even few-shot test-time-tuned DreamBooth on the authors' ZOC-Indoor-Eval benchmark, with modest inference overhead. The principal open questions concern validation on real-world imagery and independent benchmarks, and the robustness of viewpoint reasoning when target-view layout guidance is weak.