SceneDesigner: Controllable Multi-Object Image Generation with 9-DoF Pose Manipulation (2511.16666v1)

Abstract: Controllable image generation has attracted increasing attention in recent years, enabling users to manipulate visual content such as identity and style. However, achieving simultaneous control over the 9D poses (location, size, and orientation) of multiple objects remains an open challenge. Despite recent progress, existing methods often suffer from limited controllability and degraded quality, falling short of comprehensive multi-object 9D pose control. To address these limitations, we propose SceneDesigner, a method for accurate and flexible multi-object 9-DoF pose manipulation. SceneDesigner incorporates a branched network to the pre-trained base model and leverages a new representation, CNOCS map, which encodes 9D pose information from the camera view. This representation exhibits strong geometric interpretation properties, leading to more efficient and stable training. To support training, we construct a new dataset, ObjectPose9D, which aggregates images from diverse sources along with 9D pose annotations. To further address data imbalance issues, particularly performance degradation on low-frequency poses, we introduce a two-stage training strategy with reinforcement learning, where the second stage fine-tunes the model using a reward-based objective on rebalanced data. At inference time, we propose Disentangled Object Sampling, a technique that mitigates insufficient object generation and concept confusion in complex multi-object scenes. Moreover, by integrating user-specific personalization weights, SceneDesigner enables customized pose control for reference subjects. Extensive qualitative and quantitative experiments demonstrate that SceneDesigner significantly outperforms existing approaches in both controllability and quality. Code is publicly available at https://github.com/FudanCVL/SceneDesigner.

• The paper introduces CNOCS, a novel 9D pose representation that uses a canonical cuboid abstraction to enhance multi-object image generation.
• The paper employs a two-stage RL training pipeline along with a large-scale pose-labeled dataset to improve spatial alignment and orientation fidelity.
• The paper demonstrates significant gains in realism, semantic alignment, and pose control, outperforming previous methods on key quantitative metrics.

SceneDesigner: Enabling Controllable Multi-Object Image Generation with 9-DoF Pose Manipulation

Overview and Motivation

High-level manipulation of spatial properties in synthesized imagery remains a central problem in T2I frameworks. While methods such as ControlNet and its variants introduce structure-aware control for object layout, precise 3D spatial conditioning (incorporating location, size, and orientation—collectively, 9 degrees of freedom) is highly restrictive or absent in existing approaches. The lack of efficient pose conditioning for multi-object scenarios fundamentally limits the applicability of current models in design, content creation, and simulation.

SceneDesigner addresses these limitations by introducing 3DSceneDesigner, a branched architecture augmenting advanced T2I backbones, and a new pose encoding scheme (CNOCS) for geometric conditioning. This system achieves accurate and flexible multi-object image control, supporting diverse scenarios and overcoming prior generalization and quality bottlenecks.

Methodological Contributions

CNOCS: An Efficient 9D Pose Representation

Previous object pose conditioning methods either projected pose vectors into text spaces or used 3D bounding boxes, the latter missing orientation cues critical for 3D fidelity. Building on the Normalized Object Coordinate System (NOCS), SceneDesigner introduces CNOCS, replacing full 3D shape input with a canonical cuboid abstraction, thus remaining category-agnostic and compositional. CNOCS not only preserves camera-view-dependent geometric attributes but allows several embedding variants (constant Euler, identity mapping, spherical harmonics), with identity mapping (I-CNOCS) empirically selected for optimal generalization.

This representation is formalized by mapping each pixel to its normalized object's local cuboid coordinates, efficiently encoding the location, scale, and orientation within a coarse geometric prior.

ObjectPose9D: Large-Scale 9D Pose-Labeled Dataset

For training pose-conditioning models and evaluating their generalization, ObjectPose9D is constructed by aggregating annotated images from OmniNOCS and MS-COCO, with pose labels refined by human oversight. The annotation pipeline leverages 3D reconstruction (MoGe) and orientation estimation (Orient Anything), filtering for mask size and orientation unambiguity. Comprehensive captions are generated by multimodal LLMs (Qwen-VL), yielding over 120,000 images with rich visual and pose diversity.

Figure 1: Annotation pipeline for 9D pose in MS-COCO, integrating multi-source inference and manual refinement.

Two-Stage RL Training

Pose distribution in natural images is imbalanced, leading standard maximum likelihood training to poor performance for rare poses (e.g., back-facing viewpoints). The two-stage pipeline first performs MLE-based learning of pose control; then, RL fine-tuning on a balanced distribution maximizes a reward combining spatial alignment (IoU-based), orientation alignment (KL divergence), and prior loss stabilization. Efficient training strategies reduce GPU memory usage by leveraging truncated backpropagation.

Disentangled Object Sampling (DOS)

At inference, objects compete or leak attributes in multi-object scenes. DOS mitigates these effects by parallel sampling of object-specific latents, composited via region masks derived from the CNOCS map at each denoising step, robustly associating objects to their conditioning context.

Figure 2: Ablation studies quantify the impact of CNOCS, RL finetuning, and DOS on multi-object generation fidelity and disentanglement.

Experimental Analysis

Single and Multi-Object Pose Control

Quantitative and qualitative results demonstrate substantial improvements over LOOSECONTROL and Continuous 3D Words in spatial and orientation alignment. Notably, SceneDesigner achieves $\mathrm{Acc}_{ls}$ of 50.2% and mIoU of 57.21% in front-facing cases, well above the best baselines (24% and 27%, respectively), and $\mathrm{Abs.Err}$ in orientation control drops to 13.23 vs. 50+ for previous methods. SceneDesigner maintains fidelity for both single- and multi-object scenarios.

Figure 3: Evaluation of 9D pose control; SceneDesigner yields superior results in both fidelity and quality across pose settings.

Visual Quality and Alignment

SceneDesigner substantially improves FID (24.91, lowest among compared models) and CLIP-based text-image alignment (0.345, highest), confirming improvements in both realism and semantic consistency.

User Study

Human evaluators confirm SceneDesigner’s superiority in image quality (0.96), location/size fidelity (>0.96), orientation fidelity (0.91), and alignment (0.94), all normalized scores well above those of the best competitors.

Figure 4: 9D pose control examples in single-object, multi-object, and customization settings, highlighting flexibility and fidelity.

Ablation Results

Removing RL finetuning or dataset diversity dramatically degrades back-view pose generation and generalization. Alternative conditioning schemes (constant Euler, embeddings, prompt-only) are outperformed by CNOCS.

Figure 5: 9D pose control under diverse conditions.

Figure 6: SceneDesigner reliably controls pose for single-object settings.

Figure 7: SceneDesigner maintains disentanglement and pose fidelity in multi-object scenarios.

Implications and Future Directions

Practically, SceneDesigner establishes a new standard for fine-grained 3D-aware control across multiple objects, with potential for VR/AR, interactive design, robotics, and customized content creation. Theoretically, CNOCS provides a category-agnostic basis for geometric conditioning in generative models, enabling further extensions to complex shape and material properties.

However, SceneDesigner still cannot directly control explicit object shapes, and multi-object generation remains constrained by base model capacity and computational overhead introduced by DOS. Future work should target hierarchical shape control, improve compositional reasoning, and reduce inference cost—possibly by more scalable disentanglement or parameter-efficient adaptation.

Conclusion

SceneDesigner (3DSceneDesigner) delivers robust, flexible multi-object image synthesis with explicit 9D pose control using the CNOCS representation, a curated ObjectPose9D dataset, two-stage RL training, and disentangled object sampling. The system achieves high fidelity, pose alignment, and customization across scenarios, outperforming previous approaches across all major metrics, while providing a foundation for further research in interpretable, controllable, and scalable scene generation (2511.16666).