Personalize Anything for Free with Diffusion Transformer (2503.12590v1)

Abstract: Personalized image generation aims to produce images of user-specified concepts while enabling flexible editing. Recent training-free approaches, while exhibit higher computational efficiency than training-based methods, struggle with identity preservation, applicability, and compatibility with diffusion transformers (DiTs). In this paper, we uncover the untapped potential of DiT, where simply replacing denoising tokens with those of a reference subject achieves zero-shot subject reconstruction. This simple yet effective feature injection technique unlocks diverse scenarios, from personalization to image editing. Building upon this observation, we propose \textbf{Personalize Anything}, a training-free framework that achieves personalized image generation in DiT through: 1) timestep-adaptive token replacement that enforces subject consistency via early-stage injection and enhances flexibility through late-stage regularization, and 2) patch perturbation strategies to boost structural diversity. Our method seamlessly supports layout-guided generation, multi-subject personalization, and mask-controlled editing. Evaluations demonstrate state-of-the-art performance in identity preservation and versatility. Our work establishes new insights into DiTs while delivering a practical paradigm for efficient personalization.

• The paper introduces a novel training-free framework for personalized image generation using Diffusion Transformers, addressing limitations of previous methods and achieving state-of-the-art performance.
• The framework employs timestep-adaptive token replacement and patch perturbation strategies to ensure subject consistency and boost structural diversity in generated images.
• The efficient training-free approach supports complex generation tasks and scalability, applicable to digital art and interactive media.

The paper "Personalize Anything for Free with Diffusion Transformer" (2503.12590) introduces a novel training-free framework for personalized image generation using Diffusion Transformers (DiTs), aiming to generate images of user-specified concepts without requiring specific training.

Problem Solved

• Addresses the limitations of existing training-based and training-free methods in personalized image generation, which often suffer from computational inefficiency and inconsistent identity preservation.
• Aims to improve the applicability and compatibility of personalized image generation with diffusion transformers.

Methodology

• Timestep-adaptive token replacement: Enforces subject consistency through early-stage injection of reference subject tokens and enhances flexibility through late-stage regularization.
• Patch perturbation strategies: Boosts structural diversity using techniques like local token shuffling and mask morphology operations.

Evaluation

• Evaluated on personalization tasks, demonstrating state-of-the-art performance in identity preservation and versatility.
• Benchmarks included DreamBench, with performance measured using metrics like FID and CLIP for image quality and image-text alignment.
• User studies corroborated quantitative results, showing preference for the proposed method in textual alignment, identity retention, and image quality.

Implications and Applications

• Efficient personalization framework offers an alternative to traditional approaches, eliminating the need for extensive fine-tuning and pre-training.
• Supports layout-guided generation and interoperability with multiple subjects, implying scalability and practical applications in advertising, digital art, and interactive media content.
• Potential for extension to video or three-dimensional object generation.

In summary, the paper presents a method for personalized image generation using DiTs without requiring training. The approach leverages token replacement and patch perturbation strategies to achieve high-quality results with broad applicability.