FoR-SALE: Frame of Reference Diffusion Editing

• The paper introduces FoR-SALE, a paradigm that integrates multiple frames of reference to enhance spatial alignment in text-to-image diffusion.
• It employs LLM-driven parsing and latent-space editing to precisely adjust object orientation and depth based on explicit spatial cues.
• Empirical benchmarks show up to 9.9% improvement in spatial corrections, addressing limitations in conventional generative models.

Frame of Reference-guided Spatial Adjustment in LLM-based Diffusion Editing (FoR-SALE) is a paradigm for controlling spatial reasoning in text-to-image (T2I) generation by explicitly modeling and integrating diverse spatial perspectives—frames of reference (FoR)—within both language and vision modules. The approach extends traditional LLM-guided diffusion frameworks to directly interpret spatial expressions (e.g., “left of the chair, from the chair’s point of view”) and align the generated image with the intended viewpoint, even when it differs from the default camera perspective (Premsri et al., 27 Sep 2025). By incorporating specialized interpreters and spatially conditioned image editing modules, FoR-SALE addresses critical shortcomings in conventional generative models, resulting in enhanced spatial alignment, orientation handling, and depth modification across synthetic benchmarks.

1. Foundational Concepts

The frame of reference (FoR) in spatial reasoning denotes the perspective from which spatial expressions are interpreted, typically including camera/external, object/intrinsic, and mixed viewpoints (Premsri et al., 25 Feb 2025). In human cognition, accurately interpreting expressions like “the red chicken is left of the chair from the chair’s view” requires mapping object positions, orientations, and relations to the correct spatial frame.

Prior T2I diffusion models (e.g., Stable Diffusion, SD 1.5/2.1, FLUX.1) often assume a camera-relative FoR, which introduces persistent errors for non-camera instructions (intrinsic/object-centric). FoR-SALE remedies this by explicitly modeling multiple frames of reference: the method parses input prompts to extract the spatial frame, converts instructions to the camera’s viewpoint, then applies corrections that adjust object orientation and depth within the latent space (Premsri et al., 27 Sep 2025).

FoR-SALE builds conceptually on frameworks such as Self-correcting LLM-controlled Diffusion (SLD), incorporating explicit FoR interpretation modules and new latent editing operations for spatial adjustment. This multi-perspective alignment distinguishes FoR-SALE from conventional attribute-only or rigid/non-rigid editing approaches (Wang et al., 4 Jan 2024).

2. System Architecture and Process

The FoR-SALE pipeline consists of several interlinked modules:

1. LLM-Driven Visual Perception:
   • An LLM parser analyzes the textual prompt, extracting object mentions, attributes, and spatial relations. Example output from the parser: [("chicken", ["red"]), ("chair", [None])].
   • Vision modules predict object locations (bounding boxes), categorical orientation (one of eight facing directions), and depth estimates from segmentation masks and depth maps:

   $D_i = \frac{1}{|R|} \sum_{j \in R} d_j$

   where $d_j$ is the pixel-wise depth and $R$ is the object’s segmented region.

2. Frame of Reference Interpreter:

   • The system leverages an LLM with a set of 32 rules mapping spatial expressions (eight directions × four relations) to camera-relative coordinates.
   • Input expressions (e.g., “left from chair’s view”) are rewritten from the camera’s frame, enabling cross-perspective alignment.
3. Layout Interpretation, Error Detection and Correction:
   • The system generates an initial image $I_1$ and layout $V_L$ using the base diffusion model, then constructs a revised layout $\tilde{V}_L$ based on the unified perspective.
   • Direct comparison (exact matching) between $V_L$ and $\tilde{V}_L$ highlights misalignments in object positioning, facing direction, and depth.
4. Editing Action Generation:
   • FoR-SALE synthesizes spatially targeted correction actions, which include both existing SLD operations (addition, deletion, reposition, attribute modification) and two new latent editing operations:
     • Facing Direction Modification: Uses segmentation and backward diffusion (e.g., DiffEdit) to rotate objects to the correct orientation.
     • Depth Modification: Adjusts object depth within the latent space:

   $d_{j'} = \min(1, \max(0, d_j - D_i + D_{i'}))$

   for pixel $j$ where $D_i$ is current object depth, $D_{i'}$ is target depth.

5. Iterative Forward-Backward Correction:
   • Corrections are incorporated in one to three rounds via backward diffusion and synthesis with the base T2I model, achieving stepwise spatial alignment between text and image.

3. Latent-Space Editing Operations

FoR-SALE introduces a suite of latent-space operations to address spatial misalignment. Unlike conventional pixel-space edits, these operations leverage the generative model's latent code—facilitating deeper changes in object placement, orientation, and structure.

• Facing Direction Correction:
  • Utilizes segmentation modules to isolate objects, applies orientation modification in the latent space, and re-synthesizes the image via backward diffusion.
• Depth Adjustment:
  • The depth formula above aligns average object depth values, shifting regions of interest closer or farther within the image's geometric structure.

These latent operations are stacked with classical attribute edits such as addition, deletion, reposition, and modification, supporting a full spectrum of spatial changes aligned to any reference frame.

4. Benchmarks and Empirical Evaluation

FoR-SALE is evaluated on two synthetic spatial benchmarks:

• FoR-LMD:
  • Extends the Language-model-driven Diffusion (LMD) benchmark by adding explicit relative/intrinsic perspective cues.
• FoREST (Premsri et al., 25 Feb 2025):
  • Specially designed to probe frame-of-reference comprehension. Features ambiguous (“A-split”) and disambiguated (“C-split”) instances, with fine-grained annotations of object placement, orientation, and perspective.

Performance metrics report significant improvements:

• Initial GPT-4o baseline accuracy: 56.6% overall, lower for intrinsic FoR.
• After one round of FoR-SALE correction: Up to 5.3% gain (single round), up to 9.9% after multiple rounds.
• Error analysis demonstrates improved left/right relational accuracy, facing direction corrections, and partial improvement in depth handling (notably, explicit camera perspective conversion mitigates 2D spatial error, but 3D error persists).

This suggests FoR-SALE robustly bridges the gap in spatial alignment when editing from arbitrary reference frames in T2I applications.

5. Integration with Multimodal and Instructional Editing Paradigms

FoR-SALE builds upon multistep advances in LLM-based and multimodal editing:

• Instruction-Guided Editing (Nguyen et al., 15 Nov 2024):
  • Demonstrates the efficacy of natural language instructions for flexible, fine-grained control over image generation and editing.
• Multimodal Integration (Buburuzan, 30 Jul 2025):
  • Systems such as MObI and AnydoorMed incorporate reference images and explicit spatial conditioning (bounding boxes, camera/lidar fusion) to enforce geometric realism. FoR-SALE extends this by treating the reference frame itself as a first-class conditioning signal.
• Robust Editing Controls and Datasets (Wang et al., 4 Jan 2024):
  • Dual-path injection and unified self-attention schemes enable independent and fused control over appearance and structure; adaptation to FoR-SALE consists of including FoR as an additional guidance stream in the editing pipeline.

A plausible implication is that future extensions may combine SG prompting (Premsri et al., 25 Feb 2025), multimodal spatial controls, and iterative latent refinement to further reduce semantic and geometric deployment errors.

6. Limitations and Future Research

Empirical findings detail several limitations:

• 3D depth and orientation editing remain imperfect, with pronounced errors for occluded or complex object geometries.
• Current LLMs exhibit FoR selection bias and struggle with ambiguous perspectives unless assisted by spatial-guided prompting.
• Few-step inference remains computationally expensive; new model distillation techniques are needed for efficiency in interactive applications (Huang et al., 27 Feb 2024).

Open research issues include:

• Developing improved LLM architectures with explicit spatial reasoning faculties.
• Augmenting training datasets with richer, multi-perspective annotations.
• Extending evaluation metrics (e.g., EditEval and LMM Score (Huang et al., 27 Feb 2024)) to assess frame-of-reference adherence.
• Incorporating multi-round iterative correction and more powerful vision-language fusion modules.

7. Practical Implications and Domain Applications

FoR-SALE advances spatial alignment in creative and professional domains:

• Design and VR:
  • Art and simulation systems benefit from accurate reference-frame handling, ensuring that generated scenes correspond faithfully to arbitrary user-defined perspectives.
• Medical and Safety-Critical Fields (Buburuzan, 30 Jul 2025):
  • Inpainting frameworks (e.g., AnydoorMed) can be extended with FoR-SALE principles to ensure spatially precise anomaly placement and contextual realism in counterfactual scenario generation.
• Digital Forensics (Nguyen et al., 5 Dec 2024):
  • LLM-driven frameworks for forgery detection and localization may integrate FoR-aware reasoning to pinpoint edits performed from non-default viewpoints.

This suggests FoR-SALE defines a technical foundation for generative editing systems capable of robust spatial reasoning, multi-perspective correction, and context-relevant content synthesis.

---

Key Mathematical Formulas from FoR-SALE (Premsri et al., 27 Sep 2025, Wang et al., 4 Jan 2024):

• Average object depth:

$D_i = \frac{1}{|R|} \sum_{j \in R} d_j$

• Latent depth modification:

$d_{j'} = \min(1, \max(0, d_j - D_i + D_{i'}))$

• Unified self-attention in related editing frameworks:

$$\text{Attn}_{\text{uni}} = \mathrm{Softmax}\left( \frac{\text{Attn}_{\text{struct}} \cdot (Q_{\text{out}} K_{\text{app}}^T)}{\sqrt{d}} \right)$$

Summary Table—FoR-SALE System Modules

Module	Function	Spatial Reasoning Approach
LLM Visual Perception	Extract objects, attributes, spatial relations	Parses prompt, segments, estimates
Frame of Reference Interpreter	Convert spatial expression to camera perspective	32-rule mapping/perspective switch
Layout Interpreter & Correction	Detect and correct misalignments	Exact match, iterative refinement
Latent Editing Operations	Modify position, orientation, depth	Uses backward diffusion, formulas

FoR-SALE is distinguished by its systematic, rule-based conversion of spatial prompts, modular correction architecture, and iterative latent-space editing tailored for high-fidelity alignment to arbitrary frames of reference in LLM-based diffusion editing.