SAEdit: Token-level control for continuous image editing via Sparse AutoEncoder (2510.05081v1)

Abstract: Large-scale text-to-image diffusion models have become the backbone of modern image editing, yet text prompts alone do not offer adequate control over the editing process. Two properties are especially desirable: disentanglement, where changing one attribute does not unintentionally alter others, and continuous control, where the strength of an edit can be smoothly adjusted. We introduce a method for disentangled and continuous editing through token-level manipulation of text embeddings. The edits are applied by manipulating the embeddings along carefully chosen directions, which control the strength of the target attribute. To identify such directions, we employ a Sparse Autoencoder (SAE), whose sparse latent space exposes semantically isolated dimensions. Our method operates directly on text embeddings without modifying the diffusion process, making it model agnostic and broadly applicable to various image synthesis backbones. Experiments show that it enables intuitive and efficient manipulations with continuous control across diverse attributes and domains.

• The paper introduces a novel framework that uses a Sparse AutoEncoder for token-level manipulation to enable continuous and disentangled image editing.
• It demonstrates how sparse embeddings derived from a T5 encoder facilitate fine-grained semantic control with minimal interference across attributes.
• Quantitative results and user studies show that SAEdit outperforms baselines in preserving image identity and prompt fidelity during varied edits.

SAEdit: Token-level Control for Continuous Image Editing via Sparse AutoEncoder

Introduction and Motivation

SAEdit introduces a framework for continuous and disentangled image editing in text-to-image diffusion models by leveraging token-level manipulation of text embeddings via a Sparse AutoEncoder (SAE). The method addresses two critical limitations in prompt-based editing: the lack of disentanglement (where modifying one attribute inadvertently alters others) and the absence of continuous control (where edit magnitude cannot be smoothly adjusted). By operating directly on text embeddings and not modifying the diffusion process, SAEdit achieves model-agnostic, scalable, and fine-grained semantic control.

Sparse AutoEncoder Training and Embedding Space Lifting

The core of SAEdit is the SAE, trained on token embeddings from a frozen T5 encoder. The SAE consists of a linear encoder and decoder, with a sparsity-inducing regularization term in the loss function:

$$\mathcal{L} = \mathcal{L}_{\text{rec}} + \alpha \cdot \mathcal{L}_{\text{sparse}}$$

where $\mathcal{L}_{\text{rec}}$ is the reconstruction loss and $\mathcal{L}_{\text{sparse}}$ enforces sparsity in the latent space. The SAE is trained on the final output of the T5 encoder, ensuring compatibility with downstream diffusion models.

Figure 1: SAE training on token embeddings from a frozen T5 encoder, using reconstruction and sparsity losses.

The sparsity constraint yields a high-dimensional latent space where individual dimensions correspond to semantically isolated attributes, facilitating the identification of disentangled edit directions.

Extraction of Disentangled Edit Directions

Edit directions are derived by comparing the SAE-encoded representations of a source prompt and a target prompt that differ by a single attribute (e.g., "a man" vs. "a smiling man"). Token representations are aggregated via max-pooling, and an entry-wise ratio is computed to identify the dimensions most correlated with the desired edit. The edit direction is constructed as a sparse vector, zero everywhere except at the identified indices, where it takes values from the target prompt's representation.

Figure 2: Extraction of edit directions by comparing SAE-encoded prompt pairs and isolating key features via max-pooling and thresholding.

To improve robustness, multiple prompt pairs are generated using an LLM, and the resulting direction vectors are aggregated via SVD to obtain a consistent edit direction.

Application of Edit Directions and Injection Schedule

The edit direction is applied to the SAE-encoded representation of the target token in the source prompt, scaled by a factor $\omega$ for continuous control:

$$e'_{token} = \mathcal{S}_{dec}(\mathcal{S}_{enc}(e_{tgt}) + \omega \cdot d_{edit})$$

This enables fine-grained adjustment of the attribute's intensity. The modified embedding is then used to condition the diffusion model, which acts as a renderer.

Figure 3: Application of the edit direction to the sparse token representation, followed by decoding and conditioning the text-to-image model.

An exponential injection schedule is introduced to align the edit's influence with the hierarchical denoising process of diffusion models, applying the edit gently in early timesteps and increasing its strength in later steps:

$$\omega_t = \min\left(e^{t \cdot \omega} - 1, \tau \right)$$

Qualitative and Quantitative Results

SAEdit demonstrates a diverse range of continuous and disentangled semantic edits across various image styles and domains. Edits such as adding a mustache, changing expressions, and modifying age are highly localized and preserve the subject's identity. The method generalizes to non-human subjects, enabling control over attributes like seasonal appearance and object shape.

Figure 4: Diverse continuous and disentangled semantic edits, including attribute addition, expression change, and localized modifications.

SAEdit integrates seamlessly with different T5-based architectures, such as Stable Diffusion 3.5, yielding consistent and faithful edits.

Figure 5: Results with SD3.5, demonstrating model-agnostic integration and consistent edits.

Quantitative evaluation using LPIPS and VQA-Score shows that SAEdit outperforms both training-free and optimization-based baselines in image preservation and prompt fidelity, even in zero-shot settings.

Figure 6: Quantitative comparison of image preservation and prompt fidelity; SAEdit achieves superior results.

User studies corroborate these findings, with participants preferring SAEdit's edits for both preservation and prompt alignment.

Compositionality and Localization

SAEdit supports compositionality, allowing independent control of multiple attributes and targeted edits on different subjects within the same image. The learned directions are generalizable and can be composed without interference.

Figure 7: Compositionality of learned edit directions, enabling independent control of multiple attributes.

Figure 8: Composition of edits on multiple subjects, demonstrating high localization and disentanglement.

Real Image Editing

SAEdit integrates with inversion techniques (e.g., UniFlow) to enable high-fidelity edits on real-world images. Edits maintain subject identity and background fidelity across all intensity levels.

Figure 9: Real image editing with inversion, showing continuous control over expressions and attributes.

Limitations

SAEdit's disentanglement is constrained by the priors of the underlying text-to-image model. Out-of-distribution edits (e.g., adding a beard to a woman or making a dog green) may result in entangled changes, reflecting the model's inherent biases.

Figure 10: Failure cases for OOD edits, where strong model priors prevent disentangled modifications.

Implementation Details

• SAE latent dimension: 65,536; target active entries per token: 300.
• Training corpus: 12M prompts, 800M tokens (DiffusionDB + HumanCaption-10M).
• SAE trained for 200,000 steps with Adam optimizer, learning rate 0.003, sparsity weight $\alpha = 1/32$.
• Edit directions derived from $n=100$ prompt pairs generated by GPT-5.
• Exponential injection parameter $\tau = 15 \cdot \omega$.

Implications and Future Directions

SAEdit demonstrates that sparse, interpretable representations can be leveraged for fine-grained, continuous, and disentangled control in text-to-image editing. The decoupling of editing from rendering enables broad applicability across architectures. The approach suggests that further advances in sparse representation learning and model interpretability could yield even more powerful editing tools. Future work may focus on improving disentanglement for OOD concepts, extending compositionality, and integrating with multimodal editing frameworks.

Conclusion

SAEdit provides a scalable, model-agnostic framework for continuous and disentangled image editing via token-level manipulation of text embeddings using a Sparse AutoEncoder. The method achieves state-of-the-art results in both qualitative and quantitative evaluations, supports compositional and localized edits, and integrates with real image inversion techniques. The approach highlights the utility of sparse, interpretable representations for semantic control in generative models and opens avenues for further research in fine-grained, controllable image synthesis.