FreeCus: Inference-Time Image Customization

• FreeCus is a subject-driven image customization framework that leverages pivotal attention sharing, noise shifting, and semantic feature compensation.
• It operates at inference time on diffusion transformers to preserve subject identity while altering scene, style, and pose without additional training.
• Empirical evaluations show that FreeCus balances identity retention with prompt fidelity by targeting critical DiT layers for effective subject transfer.

FreeCus is a subject-driven image customization framework for diffusion transformers (DiTs) that generates new images of a user-specified subject in novel contexts from a single reference image and a text prompt, without per-subject training, embedding optimization, or a learned personalization encoder. It is formulated as a purely inference-time method for modern DiT backbones, specifically FLUX.1-dev, and is built around three mechanisms: pivotal attention sharing (PAS), adjustment of noise shifting (ANS), and semantic features compensation (SFC) via multimodal LLMs (MLLMs). The method is presented as activating latent zero-shot subject-transfer capacity already present in pretrained DiTs rather than introducing a new trained personalization module (Zhang et al., 21 Jul 2025).

1. Problem setting and conceptual position

FreeCus addresses subject-driven text-to-image generation, where the objective is to preserve the identity of a subject—such as a person, pet, object, or character—while changing scene, style, pose, outfit, accessories, or action according to text. The paper positions this problem as personalization without retraining, in contrast to two broad families of prior methods: per-subject optimization methods, such as Textual Inversion, DreamBooth, and LoRA variants, and approaches that avoid per-subject retraining but still rely on prior supervised training of specialized encoders or adapters, such as BLIP-Diffusion, IP-Adapter variants, MS-Diffusion, and RealCustom++ (Zhang et al., 21 Jul 2025).

The central claim is that modern DiTs, especially FLUX, already contain substantial zero-shot subject-transfer ability, but that standard prompting does not reliably expose it. Naive attention sharing can preserve layout or appearance, yet in subject-driven customization it tends to copy the reference image too literally, reducing editability and weakening alignment with the new prompt. FreeCus is therefore organized around a specific tension: it attempts to preserve subject identity without collapsing into near-reconstruction, and to do so entirely at inference time (Zhang et al., 21 Jul 2025).

This placement is important because the method is explicitly described as training-free and optimization-free, but not as a passive prompting trick. It still performs inference-time preprocessing, segmentation, caption generation, reference noising, and attention caching. A plausible implication is that FreeCus reframes personalization as a control problem over internal DiT representations rather than as a separate fitting problem.

2. End-to-end pipeline

The pipeline begins with a reference image containing the target subject and a target text prompt describing the desired output. The reference image is encoded into latent space with the pretrained VAE of the backbone. A segmentation model, BirefNet, extracts a subject mask so that background regions from the reference do not contaminate subject transfer. In parallel, a two-stage MLLM pipeline produces a concise subject caption: Qwen2-VL-7B-Instruct first describes the image in at most 20 tokens, emphasizing key subject attributes, and Qwen2.5-7B-Instruct then filters that description to remove irrelevant context such as background, scene actions, or accidental details. The filtered subject caption is concatenated with the user prompt (Zhang et al., 21 Jul 2025).

Reference attention features are not obtained through DDIM inversion or any other inversion procedure. Instead, FreeCus constructs a forward noising trajectory from the clean reference latent $z_0=z^{ref}$ using

$z_t = (1-\sigma_t)z_0 + \sigma_t \epsilon.$

As denoising proceeds and $\sigma_t \to 0$, the trajectory returns to the true reference image, so late, low-noise steps become reliable sources of subject information. During denoising of this reference trajectory, the method caches the relevant attention keys and values for later reuse in target generation (Zhang et al., 21 Jul 2025).

The target image is then generated from the augmented prompt while selected DiT layers are modified so that target tokens can attend to masked reference subject features. The generic shared-attention form presented in the paper is

$$A = \operatorname{softmax}\left( \frac{ [Q_p, Q_{tgt}] [K_r, K_p, K_{tgt}]^\top }{ \sqrt{d_k} } \right)\cdot [V_r, V_p, V_{tgt}],$$

but FreeCus does not apply this uniformly. Instead, it restricts such sharing to specific layers and modifies the reference branch so that the reused features remain identity-rich rather than excessively noisy (Zhang et al., 21 Jul 2025).

3. Core mechanisms

The three mechanisms are complementary: PAS supplies structurally important subject information, ANS improves the granularity of extracted subject details, and SFC restores semantic attributes that attention reuse may not transmit reliably.

Component	Mechanism	Reported role
PAS	Shared attention only in ten vital DiT layers	Preserves subject layout integrity while retaining editing flexibility
ANS	Reversed dynamic shift in the reference noising schedule	Improves fine-grained feature extraction
SFC	Filtered MLLM-generated subject caption appended to prompt	Improves cross-modal semantic representation

PAS is the principal attention-routing mechanism. Rather than sharing reference attention in all layers, FreeCus applies sharing only in ten “vital” layers identified as disproportionately important for image content. For these layers, the modified key and value tensors are

$$Q=[Q_p,Q_{tgt}],\quad K=[K_p,K_{tgt}],\quad V=[V_p,V_{tgt}],$$

$$K'=[\lambda_r \cdot K_r \odot m_r,\; \lambda_p \cdot K_p,\; K_{tgt}],$$

$V'=[V_r \odot m_r,\; V_p,\; V_{tgt}],$

where $m_r$ is the reference subject mask, $\lambda_r$ controls the influence of the reference subject, and $\lambda_p$ controls the prompt. This formulation makes the identity–text trade-off explicit and suppresses reference background leakage (Zhang et al., 21 Jul 2025).

ANS modifies FLUX’s dynamic shifting only for the reference branch. FLUX uses

$z_t = (1-\sigma_t)z_0 + \sigma_t \epsilon.$0

FreeCus argues that this shifted schedule is useful for high-resolution generation but too noisy for extracting subject features from the reference. It therefore reverses the shift direction for the reference branch: $z_t = (1-\sigma_t)z_0 + \sigma_t \epsilon.$1 and uses

$z_t = (1-\sigma_t)z_0 + \sigma_t \epsilon.$2

The intended effect is less noise at corresponding timesteps and therefore finer-grained reference attention maps, including facial textures, fur, and small object parts (Zhang et al., 21 Jul 2025).

SFC compensates for the fact that attention sharing mainly transfers structure and localized appearance, while higher-level semantic descriptors such as color may remain underspecified. The captioner is intentionally concise. The paper reports that overly detailed captions reduce flexibility, whereas filtered short captions provide the best overall balance between identity and prompt controllability (Zhang et al., 21 Jul 2025).

4. Empirical evaluation and ablation structure

FreeCus is evaluated on DreamBench++, described as five times larger than DreamBench. The benchmark includes animal, human, and object subjects. For each subject-prompt pair, the method generates four images. Evaluation uses CLIP-T for text controllability and CLIP-I plus DINO for subject similarity, with SAM used to segment generated and reference subjects so that subject regions rather than full images are compared (Zhang et al., 21 Jul 2025).

The reported aggregate and class-wise results are as follows.

Setting	CLIP-T / CLIP-I / DINO
FreeCus overall	0.308 / 0.853 / 0.696
Animal	0.328 / 0.902 / 0.738
Human	0.276 / 0.788 / 0.675
Object	0.321 / 0.869 / 0.677

The paper emphasizes that FreeCus does not dominate every competing method on every metric. IP-Adapter-Plus and MS-Diffusion obtain higher identity scores in some comparisons, while FreeCus is presented as occupying a stronger balance between identity retention, prompt alignment, and visual cleanliness. The qualitative comparisons described in the paper are consistent with that interpretation: OminiControl and DreamBooth-LoRA are said to follow prompts relatively well but often lose subject fidelity, IP-Adapter-Plus may over-copy the reference, Qwen2VL-Flux exhibits weak disentanglement when replacing text embeddings with multimodal features, and MS-Diffusion can create noticeable background artifacts (Zhang et al., 21 Jul 2025).

The ablation study assigns distinct functions to the three components. Removing PAS causes the largest degradation, with CLIP-I dropping to 0.810 and DINO to 0.590. Removing ANS yields $z_t = (1-\sigma_t)z_0 + \sigma_t \epsilon.$3, and removing SFC yields $z_t = (1-\sigma_t)z_0 + \sigma_t \epsilon.$4. The paper therefore attributes structure and identity preservation primarily to PAS, fine-detail retention to ANS, and semantic correctness to SFC (Zhang et al., 21 Jul 2025).

Hyperparameter studies make the fidelity–controllability trade-off explicit. When $z_t = (1-\sigma_t)z_0 + \sigma_t \epsilon.$5 is increased from $z_t = (1-\sigma_t)z_0 + \sigma_t \epsilon.$6 to $z_t = (1-\sigma_t)z_0 + \sigma_t \epsilon.$7, identity improves while text alignment declines slightly; the default is $z_t = (1-\sigma_t)z_0 + \sigma_t \epsilon.$8, corresponding to $z_t = (1-\sigma_t)z_0 + \sigma_t \epsilon.$9. Likewise, stronger negative shifting in ANS improves identity but lowers prompt adherence, and the paper selects $\sigma_t \to 0$0 as the default compromise. A supplementary ablation also reports that sharing attention in ten random non-vital layers or using dropout-based sparse sharing across all 57 layers is inferior, supporting the specific “vital layer” hypothesis rather than a generic sparsification effect (Zhang et al., 21 Jul 2025).

5. Compatibility, implementation, and operating regime

A notable aspect of FreeCus is that it is designed as an inference-time modification of existing DiT pipelines rather than as a self-contained retrained model. The paper demonstrates compatibility with FLUX.1-Fill-dev for subject-driven inpainting. In that setting, a fully black mask is first used to reconstruct the reference image and obtain high-quality shared attention, after which the same paradigm is applied during inpainting to place the personalized subject into masked regions while preserving surrounding context. The method is also shown to work with FLUX.1-Depth-dev for structure or layout guidance (Zhang et al., 21 Jul 2025).

The paper further reports that FreeCus can be combined with Qwen2VL-Flux, improving that system’s scores from $\sigma_t \to 0$1 to $\sigma_t \to 0$2. It is also adapted to style transfer by treating style as an abstract “subject” and changing the prompt supplied to the MLLMs. This suggests that the method is orthogonal to some existing DiT-based controls rather than exclusive of them (Zhang et al., 21 Jul 2025).

Implementation details are concrete. The backbone is FLUX.1-dev. Inference uses 30 denoising steps, guidance scale $\sigma_t \to 0$3, and $\sigma_t \to 0$4 resolution. The shared-attention weights are $\sigma_t \to 0$5. BirefNet performs segmentation; Qwen2-VL-7B-Instruct and Qwen2.5-7B-Instruct provide caption generation and filtering. The modified attention is applied in ten critical layers out of FLUX’s 57 layers. The main setup uses one reference image, and the code is released at https://github.com/Monalissaa/FreeCus (Zhang et al., 21 Jul 2025).

6. Limitations, trade-offs, and significance

The main failure mode reported for FreeCus is the appearance of artifacts resembling outline residues or ghost traces of the reference subject, attributed to attention sharing. The authors experimented with position-index shifting of shared attention, inspired by OminiControl, but report that it reduced subject similarity. They also explored masking and random attention dropout; dropout provided the best artifact–fidelity compromise, but the issue remains unresolved (Zhang et al., 21 Jul 2025).

A second limitation is dependence on MLLM caption quality. If the multimodal captioner misdescribes a rare subject or includes misleading attributes, semantic compensation can become imperfect. The method is primarily designed for single-subject customization. Multi-subject scenes are described as possible but not as the core setting. The paper also implies that severe prompt conflicts, strong occlusion, and challenging geometric transformations remain difficult, though these cases are not quantitatively dissected (Zhang et al., 21 Jul 2025).

In methodological terms, FreeCus is significant because it treats subject-driven customization in DiTs as an inference-time control problem rather than a retraining problem. The framework is training-free and optimization-free in the sense that it avoids gradient-based latent optimization, test-time fine-tuning, embedding fitting, inversion optimization, and specialized subject-encoder training. At the same time, it is not computation-free: it still requires VAE encoding, segmentation, MLLM captioning, construction of a reference noising trajectory, and caching of attention tensors. This distinction is central to understanding its place in the personalization literature. It neither replaces the generative backbone nor leaves it untouched; instead, it selectively routes information already encoded inside a pretrained DiT (Zhang et al., 21 Jul 2025).