LooseControl: Relaxed Geometric Conditioning

• LooseControl is a generative modeling framework that relaxes strict depth conditioning using coarse proxies like scene boundaries and 3D box layouts.
• It extends into video through LooseControlVideo, enabling high-level choreography via sparse oriented 3D boxes to guide temporal scene layouts.
• The approach reduces authoring burdens while preserving spatial precision, resulting in improved occlusion handling, natural object placement, and enhanced scene realism.

Searching arXiv for LooseControl and closely related control-conditioned generation papers. LooseControl denotes a line of controllable generative modeling methods that relax strict structural conditioning in diffusion-based generation by replacing dense, frame-accurate, or exact geometric supervision with coarse but geometrically meaningful proxies. In image generation, the term refers primarily to a generalized depth-conditioning framework built on Stable Diffusion and ControlNet that replaces exact depth matching with looser Boolean geometric conditions such as scene boundaries and approximate 3D boxes (Bhat et al., 2023). In video generation, the concept is extended by LooseControlVideo, which uses sparse oriented 3D boxes as a “blocking” proxy for multi-object choreography and temporal layout, rendered into a DNOCS control video and consumed by a Wan 2.2/VACE backbone (Bhat et al., 17 Jun 2026). Across this lineage, the central idea is that high-level spatial intent should be specified explicitly, while low-level shape deformation, occlusion realization, and fine appearance formation remain delegated to the generative model.

1. Conceptual definition and scope

LooseControl was proposed against the premise that conventional structural conditioning is often too rigid for realistic authoring workflows. Standard depth-conditioned ControlNet is formulated as

$$\text{Generate } I_{gen} \text{ such that } f_D(I_{gen}) = D_c,$$

where $f_D$ is an off-the-shelf monocular depth estimator and $D_c$ is the conditioning depth map (Bhat et al., 2023). LooseControl generalizes this to

$$\text{Generate } I_{gen} \text{ such that } \phi(f_D(I_{gen}), D_c)\text{ is true},$$

so that the conditioning imposes a broader geometric relationship rather than strict equality (Bhat et al., 2023).

Within this framework, “loose” does not mean unstructured. Rather, the control remains spatially meaningful while becoming easier to author and less entangled with low-level appearance. The image-generation formulation emphasizes two generalized forms. In scene boundary control,

$\phi : f_D(I_{gen}) \leq D_c,$

so the boundary depth map acts as an upper bound on scene depth, allowing objects to appear in front of enclosing surfaces. In 3D box control,

$$\phi : B_i \sim \text{3DBox}(O_{gen}^i) \quad \forall i,$$

so generated objects should roughly fit the specified boxes in position, size, and orientation (Bhat et al., 2023).

LooseControlVideo preserves the same philosophy but targets temporally coherent video authoring. It defines a control signal $b := \{b_t\}_{t=1}^{T}$, where each $b_t$ is a set of time-varying oriented 3D boxes, and models

$p_\theta(y \mid p, b, c)$

for generation, or more generally

$p_\theta(y \mid p, b, c, v)$

for editing, where $f_D$0 is the text prompt, $f_D$1 are camera parameters, and $f_D$2 may be an input source video (Bhat et al., 17 Jun 2026). This formulation separates high-level choreography from low-level execution. The paper explicitly describes the control as geometrically meaningful and spatially precise, but not so rigid that the user must author the exact final appearance or deformation of each object (Bhat et al., 17 Jun 2026).

A plausible implication is that LooseControl occupies a middle point between weak prompt-only conditioning and labor-intensive dense supervision. That interpretation is directly supported by the repeated contrast with exact depth maps, dense flow maps, depth sequences, and per-frame masks in the source papers (Bhat et al., 2023, Bhat et al., 17 Jun 2026).

2. Image-generation formulation: generalized depth conditioning

The original LooseControl method is a generalized depth-conditioning framework for diffusion-based image generation built on top of Stable Diffusion and ControlNet (Bhat et al., 2023). It keeps the ControlNet backbone and the frozen Stable Diffusion U-Net, but replaces exact depth maps with proxy depth maps representing either scene boundaries or 3D box layouts (Bhat et al., 2023). This is not a new generator from scratch; it is a ControlNet fine-tuning strategy that changes the meaning of the depth input (Bhat et al., 2023).

Three design requirements are stated for the proxy condition $f_D$3: it must remain compatible with depth-conditioned ControlNet, it must be extractable automatically from images without manual annotation, and it must also be easy for users to construct manually (Bhat et al., 2023). These constraints explain why the method uses depth-like proxy images rather than abstract geometric tokens.

For scene boundary control, the training pipeline is defined as follows: estimate depth using ZoeDepth, back-project the image into a 3D mesh, project the mesh onto a horizontal plane, approximate the 2D footprint with a polygon, extrude the polygon vertically into boundary planes, and render the depth of that boundary surface from the original camera view (Bhat et al., 2023). The result is a proxy depth map encoding room or scene boundaries rather than precise object geometry.

For 3D box control, the paper extracts object-level spatial proxies by depth estimation with ZoeDepth, segmentation with SAM, back-projection of each segment into 3D point clouds, fitting an oriented minimum-volume 3D bounding box to each segment, and rendering those 3D boxes back into a depth map (Bhat et al., 2023). This depth-like representation specifies approximate object layout without fixing exact shape or appearance.

The paper positions this generalization as a practical answer to the limitations of standard depth-conditioned generation. Users no longer need exact depth for every object; boundaries or rough object locations are sufficient (Bhat et al., 2023). This suggests that LooseControl is less a weakening of geometry than a remapping of geometry into authorable constraints.

3. Video extension: directorial control through spatial blocking

LooseControlVideo extends the LooseControl principle from images to text-to-video generation by borrowing the “blocking” metaphor from filmmaking and animation (Bhat et al., 17 Jun 2026). Instead of requiring frame-accurate supervision, the user provides sparse oriented 3D boxes that describe intended scene layout, trajectories, and approximate timing, while the generative model infers fine-grained execution, including shape deformation, physics-like effects, and realistic occlusions (Bhat et al., 17 Jun 2026).

Each 3D box is parameterized by center $f_D$4, scale $f_D$5, and rotation $f_D$6 (Bhat et al., 17 Jun 2026). Rather than encoding these parameters directly with an MLP, the method renders them from the camera view into a control image $f_D$7, because the base video model operates in image space and the conditioning should expose the relevant geometry in a form the model can readily consume (Bhat et al., 17 Jun 2026).

A central design goal is localized refinement without global disruption. Because the control is rendered into image space and composited frame by frame, the user can change only a subset of objects or a portion of a trajectory while leaving the rest of the scene untouched (Bhat et al., 17 Jun 2026). The paper highlights edits such as turning a horse jump into a different trajectory, modifying a jeep’s path into a drift, or weaving a football through occluding foliage while preserving identity and scene coherence (Bhat et al., 17 Jun 2026). These examples are presented not as separate algorithms but as consequences of the same blocking representation.

For editing, the representation is extended by compositing rendered DNOCS into the input video in an occlusion-aware way. The control input can be mixed on a per-frame basis from DNOCS-only frames, preserved source-video frames, black “fill-in” frames, and spatially composited mixtures, plus gray masks for areas to remove (Bhat et al., 17 Jun 2026). This unified representation allows the same model to handle pure generation, motion retargeting, interpolation, and editing while keeping visible parts of the source video stable (Bhat et al., 17 Jun 2026).

A plausible implication is that LooseControlVideo reframes video control as directorial specification rather than per-frame supervision. That interpretation follows directly from the paper’s contrast between “high-level choreography” and “low-level object deformation and pose changes” (Bhat et al., 17 Jun 2026).

4. Geometric representations: proxy depth maps and DNOCS

The image and video variants of LooseControl differ primarily in how they encode coarse geometry. In the image setting, the proxy is a rendered depth-like map derived from scene boundaries or 3D boxes (Bhat et al., 2023). In the video setting, the key representation is DNOCS, or Depth-modulated Normalized Object Coordinate Space (Bhat et al., 17 Jun 2026).

DNOCS is a 3-channel rendering of each oriented 3D box that jointly encodes local orientation and global depth (Bhat et al., 17 Jun 2026). It is inspired by NOCS, but extends it with depth modulation so that the model can disambiguate which surfaces are closer or farther, which is essential for occlusion reasoning and depth-ordered interactions (Bhat et al., 17 Jun 2026).

The DNOCS pipeline starts from ray-box intersection in the local box frame. Given camera intrinsics $f_D$8 and extrinsics $f_D$9, and a box defined by center $D_c$0, box-to-world rotation $D_c$1, and half-extents $D_c$2, the camera-space ray for pixel $D_c$3 is

$D_c$4

Transforming to world space: $D_c$5 with $D_c$6. Then in the box-local frame,

$D_c$7

where $D_c$8 (Bhat et al., 17 Jun 2026).

Ray-box intersection is performed with the standard slab test for the local AABB $D_c$9, yielding a hit mask $$\text{Generate } I_{gen} \text{ such that } \phi(f_D(I_{gen}), D_c)\text{ is true},$$0, intersection parameter $$\text{Generate } I_{gen} \text{ such that } \phi(f_D(I_{gen}), D_c)\text{ is true},$$1, and local surface point

$$\text{Generate } I_{gen} \text{ such that } \phi(f_D(I_{gen}), D_c)\text{ is true},$$2

The corresponding camera-space depth is

$$\text{Generate } I_{gen} \text{ such that } \phi(f_D(I_{gen}), D_c)\text{ is true},$$3

with background pixels assigned $$\text{Generate } I_{gen} \text{ such that } \phi(f_D(I_{gen}), D_c)\text{ is true},$$4 (Bhat et al., 17 Jun 2026).

Orientation is then encoded by mapping the normalized local surface direction to color: $$\text{Generate } I_{gen} \text{ such that } \phi(f_D(I_{gen}), D_c)\text{ is true},$$5 followed by

$$\text{Generate } I_{gen} \text{ such that } \phi(f_D(I_{gen}), D_c)\text{ is true},$$6

with fixed luminance $$\text{Generate } I_{gen} \text{ such that } \phi(f_D(I_{gen}), D_c)\text{ is true},$$7 and chroma strength $$\text{Generate } I_{gen} \text{ such that } \phi(f_D(I_{gen}), D_c)\text{ is true},$$8 (Bhat et al., 17 Jun 2026). Depth is converted into a normalized inverse-depth signal,

$$\text{Generate } I_{gen} \text{ such that } \phi(f_D(I_{gen}), D_c)\text{ is true},$$9

where $\phi : f_D(I_{gen}) \leq D_c,$0 and $\phi : f_D(I_{gen}) \leq D_c,$1 are the 2nd and 98th percentile depths over valid box pixels. A multiplicative brightness term is

$\phi : f_D(I_{gen}) \leq D_c,$2

with $\phi : f_D(I_{gen}) \leq D_c,$3 and $\phi : f_D(I_{gen}) \leq D_c,$4. The final rendering is

$\phi : f_D(I_{gen}) \leq D_c,$5

with background pixels black (Bhat et al., 17 Jun 2026).

The paper summarizes the resulting semantics succinctly: hue encodes local box orientation, while brightness encodes depth ordering (Bhat et al., 17 Jun 2026). This makes occlusion structure and 6-DOF pose cues visually explicit in the conditioning video. A plausible implication is that DNOCS functions as an image-space surrogate for structured 3D reasoning, which is why the authors render boxes rather than inject raw 3D parameters.

5. Architectures, adaptation strategy, and editing mechanisms

The original LooseControl is implemented by fine-tuning a pretrained depth ControlNet on top of Stable Diffusion v1.5 while keeping the Stable Diffusion U-Net frozen (Bhat et al., 2023). The paper studies two adaptation strategies: naive full fine-tuning of ControlNet and LoRA fine-tuning of ControlNet attention blocks (Bhat et al., 2023). The LoRA update is

$\phi : f_D(I_{gen}) \leq D_c,$6

with $\phi : f_D(I_{gen}) \leq D_c,$7, $\phi : f_D(I_{gen}) \leq D_c,$8, $\phi : f_D(I_{gen}) \leq D_c,$9, and $$\phi : B_i \sim \text{3DBox}(O_{gen}^i) \quad \forall i,$$0 (Bhat et al., 2023). At inference, a scale factor $$\phi : B_i \sim \text{3DBox}(O_{gen}^i) \quad \forall i,$$1 is introduced,

$$\phi : B_i \sim \text{3DBox}(O_{gen}^i) \quad \forall i,$$2

and the paper reports $$\phi : B_i \sim \text{3DBox}(O_{gen}^i) \quad \forall i,$$3 as a good default (Bhat et al., 2023).

Naive fine-tuning caused saturated generations and color artifacts, whereas LoRA-based fine-tuning was more stable and avoided these color issues (Bhat et al., 2023). Training uses Stable Diffusion v1.5, a pretrained ControlNet depth checkpoint, NYU-Depth-v2 for both scene boundary and box control fine-tuning, captions generated using BLIP-2, and proxy depth maps synthesized automatically from the training images (Bhat et al., 2023). Supplementary implementation details include LoRA rank $$\phi : B_i \sim \text{3DBox}(O_{gen}^i) \quad \forall i,$$4, 200 fine-tuning steps, learning rate $$\phi : B_i \sim \text{3DBox}(O_{gen}^i) \quad \forall i,$$5, batch size 12, Adam optimizer, default ControlNet conditioning scale $$\phi : B_i \sim \text{3DBox}(O_{gen}^i) \quad \forall i,$$6, and default LoRA scale $$\phi : B_i \sim \text{3DBox}(O_{gen}^i) \quad \forall i,$$7 (Bhat et al., 2023).

LooseControlVideo adopts a parallel adaptation principle in a different backbone. It is built on the Wan 2.2/VACE video diffusion backbone, and the base model is left unchanged (Bhat et al., 17 Jun 2026). The control video $$\phi : B_i \sim \text{3DBox}(O_{gen}^i) \quad \forall i,$$8 is passed through the standard VACE conditioning pathway, which injects control residuals into the backbone DiT (Bhat et al., 17 Jun 2026). The authors fine-tune only the control modules with LoRA, using rank 64, for 10k iterations on 4 H100 80GB GPUs, while freezing the underlying pretrained video model (Bhat et al., 17 Jun 2026). The paper characterizes this as a ControlNet-style strategy in which the model learns to interpret the DNOCS-rendered control video as a strong geometric prior (Bhat et al., 17 Jun 2026).

The image model introduces two editing mechanisms. “3D box editing” enables users to change, add, remove, move, resize, or rotate boxes while freezing the style of the image (Bhat et al., 2023). Style preservation is implemented by reusing attention keys and values from the source image, but only from the last two decoder layers; sharing keys and values across all layers would make outputs nearly identical to the source (Bhat et al., 2023). “Attribute editing” computes the Jacobian of the bottleneck residual with respect to the latent input,

$$\phi : B_i \sim \text{3DBox}(O_{gen}^i) \quad \forall i,$$9

then performs SVD on this Jacobian to identify influential directions $b := \{b_t\}_{t=1}^{T}$0, and edits by perturbing the bottleneck residual,

$b := \{b_t\}_{t=1}^{T}$1

where $b := \{b_t\}_{t=1}^{T}$2 controls edit strength (Bhat et al., 2023). The paper reports semantic changes such as furniture density, object type, object material, object shape or thickness, and in some cases number of objects (Bhat et al., 2023).

The video model does not present an analogous latent semantic editing mechanism, but its compositional control video serves a similar role at the structural level by mixing DNOCS frames, preserved source content, fill-in frames, and removal masks (Bhat et al., 17 Jun 2026).

6. Empirical performance and comparative position

LooseControl’s image paper emphasizes qualitative comparisons and a user study rather than dense numeric benchmarking. The reported evaluation covers indoor rooms, streets, partially bounded outdoor scenes, and underwater-like or other structured environments (Bhat et al., 2023). Qualitatively, scene boundary control produces fuller, more realistic scenes instead of empty interiors, and 3D box control produces more natural object arrangements rather than unnatural box-shaped objects (Bhat et al., 2023). A user study with 41 participants used a 2-alternative forced choice setup in which participants saw a prompt and condition and chose between ControlNet and LooseControl outputs; more than 95% of responses favored LooseControl (Bhat et al., 2023).

LooseControlVideo, by contrast, reports explicit benchmark results on nuScenes, HO-3D, and BEHAVE (Bhat et al., 17 Jun 2026). The main baselines are a control-free model using only ground-truth first/last frames, VACE with 2D flow, finetuned VACE with 2D flow, and finetuned VACE with 2D boxes (Bhat et al., 17 Jun 2026). The metrics include Containment, Trajectory Error (TrajErr), Occlusion Accuracy (OcclAcc), Global Motion Field Agreement (GMFA), Rigid Motion Consistency (RMC), Global Overlap Winner (GOW), and VBench quality (Bhat et al., 17 Jun 2026). The paper defines

$b := \{b_t\}_{t=1}^{T}$3

where $b := \{b_t\}_{t=1}^{T}$4 is the generated centroid and $b := \{b_t\}_{t=1}^{T}$5 is the projected control center (Bhat et al., 17 Jun 2026).

The reported quantitative pattern is consistent across datasets.

Dataset	LooseControlVideo	Finetuned VACE with 2D flow	Finetuned VACE with 2D boxes
nuScenes TrajErr	5.79	6.78	16.66
nuScenes OcclAcc	92.69	79.32	42.45
nuScenes RMC	0.318	0.528	0.735
HO-3D TrajErr	3.9	5.4	9.7
HO-3D OcclAcc	94.1	84.2	55.1
BEHAVE TrajErr	5.8	7.6	14.9
BEHAVE OcclAcc	90.2	78.5	49.7

On nuScenes, LooseControlVideo also reports GMFA of 0.066 versus 0.093 for finetuned 2D flow and 0.232 for 2D boxes, and GOW of 97.32 (Bhat et al., 17 Jun 2026). On HO-3D it reports RMC 0.122 and GMFA 0.045, and on BEHAVE RMC 0.207 and GMFA 0.062 (Bhat et al., 17 Jun 2026). The abstract summarizes the overall gain as a 1.2x to 3x improvement in Trajectory Error, 2x improvement in Rigid Motion Consistency, and a 1.5x to 2x increase in Occlusion Accuracy over current state-of-the-art layout-conditioned models (Bhat et al., 17 Jun 2026).

The authors interpret the comparative pattern as follows: 2D boxes can yield high containment but poor 3D motion fidelity; 2D flow captures motion but cannot reliably express occlusion and 3D orientation; DNOCS-oriented 3D blocking gives the best balance of trajectory adherence, occlusion correctness, and rigid-motion consistency while keeping visual quality competitive (Bhat et al., 17 Jun 2026). This suggests that LooseControl’s main empirical contribution is not merely relaxed conditioning, but a better factorization of control variables.

7. Limitations, misconceptions, and relation to adjacent work

A common misconception is that LooseControl removes geometric structure. The papers state the opposite. The conditioning remains explicitly geometric, but it is specified at a higher level than exact depth or dense per-frame control (Bhat et al., 2023, Bhat et al., 17 Jun 2026). Another misconception is that “loose” implies imprecision. In the source usage, it instead denotes control that is coarse in authoring burden while remaining spatially precise in the variables it does represent (Bhat et al., 17 Jun 2026).

The limitations are also explicit. In the image framework, control over secondary objects is harder than for primary objects, likely because such objects are less frequent in training data; like ControlNet, too many constraints can reduce diversity; and the method is currently depth-centric, with other guidance channels suggested as future work (Bhat et al., 2023). In the video framework, the system does not explicitly bind identities to individual boxes, so identity-to-box assignment can be ambiguous in multi-subject scenes, and the model still relies on the user to specify timing and rhythm of motion rather than inferring temporal choreography from sparse spatial intent alone (Bhat et al., 17 Jun 2026).

LooseControl has also become a reference point for later geometry-conditioned generation methods. SceneDesigner explicitly contrasts itself with LOOSECONTROL, stating that LOOSECONTROL can use 3D bounding boxes for location and size, but it does not encode orientation, so the same box could correspond to a front-facing or back-facing object (Qin et al., 20 Nov 2025). SceneDesigner’s CNOCS representation and multi-object 9-DoF pose manipulation therefore extend the same general design space toward stronger pose control (Qin et al., 20 Nov 2025). This suggests that LooseControl is best understood not as an isolated model, but as a foundational relaxation strategy in the broader evolution from strict depth conditioning toward authorable 3D scene control.

Outside this generative-model lineage, the phrase “LooseControl” appears in unrelated literatures concerning approximate statistical risk control, partial control in chaotic systems, or loss-of-control analysis in robotics and AI governance (Blot et al., 2024, Das et al., 2015, Beers et al., 2024, Carlucci et al., 11 Jun 2026). Those usages share only a broad intuition of relaxed or indirect control. In the computer-vision generative-model literature, the term is specific to generalized depth conditioning and its extension to directorial video control via spatial blocking (Bhat et al., 2023, Bhat et al., 17 Jun 2026).