Group Relative Attention Guidance for Image Editing (2510.24657v1)

Abstract: Recently, image editing based on Diffusion-in-Transformer models has undergone rapid development. However, existing editing methods often lack effective control over the degree of editing, limiting their ability to achieve more customized results. To address this limitation, we investigate the MM-Attention mechanism within the DiT model and observe that the Query and Key tokens share a bias vector that is only layer-dependent. We interpret this bias as representing the model's inherent editing behavior, while the delta between each token and its corresponding bias encodes the content-specific editing signals. Based on this insight, we propose Group Relative Attention Guidance, a simple yet effective method that reweights the delta values of different tokens to modulate the focus of the model on the input image relative to the editing instruction, enabling continuous and fine-grained control over editing intensity without any tuning. Extensive experiments conducted on existing image editing frameworks demonstrate that GRAG can be integrated with as few as four lines of code, consistently enhancing editing quality. Moreover, compared to the commonly used Classifier-Free Guidance, GRAG achieves smoother and more precise control over the degree of editing. Our code will be released at https://github.com/little-misfit/GRAG-Image-Editing.

• The paper demonstrates that a dominant bias vector in MM-Attention layers can be modulated to precisely control editing strength in DiT models.
• GRAG employs a bias-deviation decomposition with scaling factors to outperform standard Classifier-Free Guidance across qualitative and quantitative benchmarks.
• The method is computationally efficient, architecture-agnostic, and easily integrable into both training-based and training-free image editing pipelines.

Group Relative Attention Guidance for Image Editing: A Technical Analysis

Introduction

The paper "Group Relative Attention Guidance for Image Editing" (2510.24657) addresses the persistent challenge of achieving fine-grained, continuous, and user-aligned control over editing strength in Diffusion-in-Transformer (DiT) based image editing models. The authors identify a previously underexplored phenomenon: a significant, layer-dependent bias vector in the query and key embeddings of the multi-modal attention (MM-Attention) mechanism. This bias, they argue, encodes the model's inherent editing behavior, while the deviation of each token from this bias encodes content-specific editing signals. Leveraging this insight, the authors propose Group Relative Attention Guidance (GRAG), a lightweight, architecture-agnostic method that enables precise modulation of editing intensity by reweighting these deviations. GRAG is shown to be effective across both training-based and training-free DiT-based editors, outperforming standard Classifier-Free Guidance (CFG) in terms of controllability and edit quality.

Analysis of Embedding Bias in MM-Attention

The core technical insight of the paper is the empirical and theoretical identification of a dominant bias vector in the query and key embeddings within each MM-Attention layer of DiT models. Visualization of the embedding features reveals that token distributions are not isotropic but instead cluster around a shared, layer-specific bias.

Figure 1: Visualization of embedding features input to the attention layer, showing a significant bias across different tokens.

Further analysis demonstrates that this bias is modality-dependent: visual features are concentrated at positions corresponding to high RoPE frequencies, while textual features are associated with low frequencies, indicating incomplete alignment in the shared embedding space.

Figure 2: Embedding features aggregated across the sequence and head axes, highlighting the separation of visual and textual features by RoPE frequency.

Statistical analysis across attention heads confirms the presence of a strong bias vector, with mean vector magnitudes and standard deviations indicating that most tokens are close to this bias in the embedding space.

Figure 3: Mean vector magnitudes and standard deviations across attention heads, revealing a significant bias vector in the embedding space.

The authors formalize the decomposition of each embedding as $q_i = q_{\text{bias}} + \Delta q_i$ and $k_i = k_{\text{bias}} + \Delta k_i$, and show that the attention score can be factorized such that the influence of $\Delta k$ (the deviation from the bias) is diluted by the magnitude of the bias. This suggests that direct modulation of $\Delta k$ provides a mechanism for controlling the influence of editing instructions.

Group Relative Attention Guidance (GRAG): Methodology

GRAG operates by explicitly decomposing the key embeddings of a selected token group (e.g., source image tokens) into their mean (the group bias) and their deviations. The method then applies two scaling factors: $\lambda$ for the bias and $\delta$ for the deviations, yielding updated key embeddings:

$$\hat{k}_s^i = \lambda \cdot k_{\text{bias}} + \delta \cdot (k_s^i - k_{\text{bias}})$$

This reparameterization is applied in-place to the relevant slice of the key tensor before the attention computation. The process is highly efficient, requiring only a few lines of code and no additional training.

Figure 4: Illustration of GRAG in the MM-DiT image editing model, showing the application of relative modulation to source image key embeddings.

A PyTorch implementation is provided, demonstrating the minimal code changes required to integrate GRAG into existing MM-Attention blocks:

s_idx, e_idx, bias_scale, delta_scale = 4096, 8192, 1.0, 1.05 group_bias = img_key[:, s_idx:e_idx, :, :].mean(dim=1) img_key[:, s_idx:e_idx, :, :] = bias_scale * group_bias + \ delta_scale * (img_key[:, s_idx:e_idx, :, :] - group_bias)

The $\delta$ parameter is empirically shown to provide the most continuous and effective control over editing strength, while $\lambda$ has a less pronounced effect.

Experimental Results

Qualitative Evaluation

GRAG is evaluated on both training-based (Kontext, Step1X-Edit, Qwen-Edit) and training-free (Flowedit, StableFlow, StableFlow$+$) MM-DiT-based image editing models. Qualitative results demonstrate that GRAG enables smooth, progressive, and user-aligned control over editing strength, outperforming CFG in both responsiveness to instructions and preservation of source image fidelity.

Figure 5: Visualization results on training-based image editing methods, showing improved consistency and edit quality with GRAG.

Figure 6: Visualization results on training-free image editing methods, demonstrating the generalization of GRAG to different architectures.

Quantitative Evaluation

On the PIE benchmark, GRAG consistently improves LPIPS, SSIM, and EditScore metrics compared to baselines. Notably, in Qwen-Edit, GRAG reduces LPIPS from 0.3428 to 0.3042 and increases SSIM from 0.8506 to 0.9263, indicating better content preservation and perceptual quality. EditScore improvements are observed across all models.

Ablation studies confirm that adjusting $\delta$ alone yields the most continuous and effective editing control, while simultaneous adjustment of $\lambda$ and $\delta$ can degrade visual fidelity. In contrast, CFG provides only coarse, stepwise control and is less effective at balancing instruction following and image consistency.

Implementation and Deployment Considerations

GRAG is model-agnostic and can be integrated into any MM-DiT-based image editing pipeline with minimal code changes. The method is computationally negligible, as it involves only mean and difference operations on the key embeddings prior to the attention computation. No additional memory or inference time overhead is introduced.

For deployment, the primary consideration is the selection of the token group to which GRAG is applied (e.g., source image tokens, text tokens) and the tuning of the $\delta$ parameter to achieve the desired editing strength. The method is robust in training-based settings but may exhibit reduced stability in training-free scenarios, particularly when the architecture does not inject source image features via cross-attention.

Theoretical and Practical Implications

The identification of a dominant bias vector in MM-Attention layers has implications for the interpretability and controllability of transformer-based generative models. By exposing and modulating the structure of the embedding space, GRAG provides a principled mechanism for disentangling model-inherent editing behavior from content-specific signals. This insight could inform future architectural designs that explicitly separate or align these components for improved controllability.

Practically, GRAG enables continuous, user-aligned control over editing strength, addressing a key limitation of existing prompt-based and CFG-based methods. The approach is compatible with both open-source and proprietary MM-DiT architectures, facilitating its adoption in real-world creative workflows.

Future Directions

Potential avenues for future research include:

• Extending the bias-deviation decomposition to other modalities (e.g., video, 3D).
• Investigating the interaction between GRAG and other attention guidance or negative prompting techniques.
• Exploring adaptive or learned schedules for $\delta$ based on user feedback or downstream task requirements.
• Integrating bias analysis into model pretraining or fine-tuning to enhance alignment and controllability.

Conclusion

This work provides a rigorous analysis of the internal structure of MM-Attention in DiT-based image editing models, revealing a dominant bias vector that governs editing behavior. The proposed Group Relative Attention Guidance method leverages this insight to enable fine-grained, continuous, and efficient control over editing strength, outperforming standard guidance techniques in both qualitative and quantitative evaluations. GRAG is lightweight, model-agnostic, and readily deployable, offering a practical solution for controllable image editing and a foundation for further research into interpretable and steerable generative models.