---
title: 'FreeCus: Inference-Time Image Customization'
url: https://www.emergentmind.com/topics/freecus
type: topic
---

# FreeCus: Inference-Time Image Customization

FreeCus is a subject-driven image customization framework for diffusion transformers (DiTs) that generates new images of a user-specified subject in novel contexts from a single reference image and a text prompt, without per-subject training, embedding optimization, or a learned personalization encoder. It is formulated as a purely inference-time method for modern DiT backbones, specifically FLUX.1-dev, and is built around three mechanisms: pivotal attention sharing (PAS), adjustment of noise shifting (ANS), and semantic features compensation (SFC) via multimodal large language models (MLLMs). The method is presented as activating latent zero-shot subject-transfer capacity already present in pretrained DiTs rather than introducing a new trained personalization module [2507.15249].

## 1. Problem setting and conceptual position

FreeCus addresses subject-driven text-to-image generation, where the objective is to preserve the identity of a subject—such as a person, pet, object, or character—while changing scene, style, pose, outfit, accessories, or action according to text. The paper positions this problem as personalization without retraining, in contrast to two broad families of prior methods: per-subject optimization methods, such as Textual Inversion, DreamBooth, and LoRA variants, and approaches that avoid per-subject retraining but still rely on prior supervised training of specialized encoders or adapters, such as BLIP-Diffusion, IP-Adapter variants, MS-Diffusion, and RealCustom++ [2507.15249].

The central claim is that modern DiTs, especially FLUX, already contain substantial zero-shot subject-transfer ability, but that standard prompting does not reliably expose it. Naive attention sharing can preserve layout or appearance, yet in subject-driven customization it tends to copy the reference image too literally, reducing editability and weakening alignment with the new prompt. FreeCus is therefore organized around a specific tension: it attempts to preserve subject identity without collapsing into near-reconstruction, and to do so entirely at inference time [2507.15249].

This placement is important because the method is explicitly described as training-free and optimization-free, but not as a passive prompting trick. It still performs inference-time preprocessing, segmentation, caption generation, reference noising, and attention caching. A plausible implication is that FreeCus reframes personalization as a control problem over internal DiT representations rather than as a separate fitting problem.

## 2. End-to-end pipeline

The pipeline begins with a reference image containing the target subject and a target text prompt describing the desired output. The reference image is encoded into latent space with the pretrained VAE of the backbone. A segmentation model, BirefNet, extracts a subject mask so that background regions from the reference do not contaminate subject transfer. In parallel, a two-stage MLLM pipeline produces a concise subject caption: Qwen2-VL-7B-Instruct first describes the image in at most 20 tokens, emphasizing key subject attributes, and Qwen2.5-7B-Instruct then filters that description to remove irrelevant context such as background, scene actions, or accidental details. The filtered subject caption is concatenated with the user prompt [2507.15249].

Reference attention features are not obtained through DDIM inversion or any other inversion procedure. Instead, FreeCus constructs a forward noising trajectory from the clean reference latent \(z_0=z^{ref}\) using
\[
z_t = (1-\sigma_t)z_0 + \sigma_t \epsilon.
\]
As denoising proceeds and \(\sigma_t \to 0\), the trajectory returns to the true reference image, so late, low-noise steps become reliable sources of subject information. During denoising of this reference trajectory, the method caches the relevant attention keys and values for later reuse in target generation [2507.15249].

The target image is then generated from the augmented prompt while selected DiT layers are modified so that target tokens can attend to masked reference subject features. The generic shared-attention form presented in the paper is
\[
A = \operatorname{softmax}\left( \frac{ [Q_p, Q_{tgt}] [K_r, K_p, K_{tgt}]^\top }{ \sqrt{d_k} } \right)\cdot [V_r, V_p, V_{tgt}],
\]
but FreeCus does not apply this uniformly. Instead, it restricts such sharing to specific layers and modifies the reference branch so that the reused features remain identity-rich rather than excessively noisy [2507.15249].

## 3. Core mechanisms

The three mechanisms are complementary: PAS supplies structurally important subject information, ANS improves the granularity of extracted subject details, and SFC restores semantic attributes that attention reuse may not transmit reliably.

| Component | Mechanism | Reported role |
|---|---|---|
| PAS | Shared attention only in ten vital DiT layers | Preserves subject layout integrity while retaining editing flexibility |
| ANS | Reversed dynamic shift in the reference noising schedule | Improves fine-grained feature extraction |
| SFC | Filtered MLLM-generated subject caption appended to prompt | Improves cross-modal semantic representation |

PAS is the principal attention-routing mechanism. Rather than sharing reference attention in all layers, FreeCus applies sharing only in ten “vital” layers identified as disproportionately important for image content. For these layers, the modified key and value tensors are
\[
Q=[Q_p,Q_{tgt}],\quad K=[K_p,K_{tgt}],\quad V=[V_p,V_{tgt}],
\]
\[
K'=[\lambda_r \cdot K_r \odot m_r,\; \lambda_p \cdot K_p,\; K_{tgt}],
\]
\[
V'=[V_r \odot m_r,\; V_p,\; V_{tgt}],
\]
where \(m_r\) is the reference subject mask, \(\lambda_r\) controls the influence of the reference subject, and \(\lambda_p\) controls the prompt. This formulation makes the identity–text trade-off explicit and suppresses reference background leakage [2507.15249].

ANS modifies FLUX’s dynamic shifting only for the reference branch. FLUX uses
\[
\sigma_t= \frac{e^\mu}{e^\mu+\frac{1}{t}-1}, \qquad \mu = L_x \cdot m + b.
\]
FreeCus argues that this shifted schedule is useful for high-resolution generation but too noisy for extracting subject features from the reference. It therefore reverses the shift direction for the reference branch:
\[
\sigma_t' = \frac{e^{-\mu}}{e^{-\mu}+\frac{1}{t}-1},
\]
and uses
\[
z_t = (1-\sigma_t') z^{ref} + \sigma_t' \epsilon.
\]
The intended effect is less noise at corresponding timesteps and therefore finer-grained reference attention maps, including facial textures, fur, and small object parts [2507.15249].

SFC compensates for the fact that attention sharing mainly transfers structure and localized appearance, while higher-level semantic descriptors such as color may remain underspecified. The captioner is intentionally concise. The paper reports that overly detailed captions reduce flexibility, whereas filtered short captions provide the best overall balance between identity and prompt controllability [2507.15249].

## 4. Empirical evaluation and ablation structure

FreeCus is evaluated on DreamBench++, described as five times larger than DreamBench. The benchmark includes animal, human, and object subjects. For each subject-prompt pair, the method generates four images. Evaluation uses CLIP-T for text controllability and CLIP-I plus DINO for subject similarity, with SAM used to segment generated and reference subjects so that subject regions rather than full images are compared [2507.15249].

The reported aggregate and class-wise results are as follows.

| Setting | CLIP-T / CLIP-I / DINO |
|---|---|
| FreeCus overall | 0.308 / 0.853 / 0.696 |
| Animal | 0.328 / 0.902 / 0.738 |
| Human | 0.276 / 0.788 / 0.675 |
| Object | 0.321 / 0.869 / 0.677 |

The paper emphasizes that FreeCus does not dominate every competing method on every metric. IP-Adapter-Plus and MS-Diffusion obtain higher identity scores in some comparisons, while FreeCus is presented as occupying a stronger balance between identity retention, prompt alignment, and visual cleanliness. The qualitative comparisons described in the paper are consistent with that interpretation: OminiControl and DreamBooth-LoRA are said to follow prompts relatively well but often lose subject fidelity, IP-Adapter-Plus may over-copy the reference, Qwen2VL-Flux exhibits weak disentanglement when replacing text embeddings with multimodal features, and MS-Diffusion can create noticeable background artifacts [2507.15249].

The ablation study assigns distinct functions to the three components. Removing PAS causes the largest degradation, with CLIP-I dropping to 0.810 and DINO to 0.590. Removing ANS yields \(0.324 / 0.829 / 0.624\), and removing SFC yields \(0.322 / 0.822 / 0.633\). The paper therefore attributes structure and identity preservation primarily to PAS, fine-detail retention to ANS, and semantic correctness to SFC [2507.15249].

Hyperparameter studies make the fidelity–controllability trade-off explicit. When \(\lambda_r=\lambda_p\) is increased from \(1.00\) to \(1.15\), identity improves while text alignment declines slightly; the default is \(1.10\), corresponding to \(0.308 / 0.853 / 0.696\). Likewise, stronger negative shifting in ANS improves identity but lowers prompt adherence, and the paper selects \(-1.0\) as the default compromise. A supplementary ablation also reports that sharing attention in ten random non-vital layers or using dropout-based sparse sharing across all 57 layers is inferior, supporting the specific “vital layer” hypothesis rather than a generic sparsification effect [2507.15249].

## 5. Compatibility, implementation, and operating regime

A notable aspect of FreeCus is that it is designed as an inference-time modification of existing DiT pipelines rather than as a self-contained retrained model. The paper demonstrates compatibility with FLUX.1-Fill-dev for subject-driven inpainting. In that setting, a fully black mask is first used to reconstruct the reference image and obtain high-quality shared attention, after which the same paradigm is applied during inpainting to place the personalized subject into masked regions while preserving surrounding context. The method is also shown to work with FLUX.1-Depth-dev for structure or layout guidance [2507.15249].

The paper further reports that FreeCus can be combined with Qwen2VL-Flux, improving that system’s scores from \(0.267 / 0.841 / 0.664\) to \(0.274 / 0.853 / 0.658\). It is also adapted to style transfer by treating style as an abstract “subject” and changing the prompt supplied to the MLLMs. This suggests that the method is orthogonal to some existing DiT-based controls rather than exclusive of them [2507.15249].

Implementation details are concrete. The backbone is FLUX.1-dev. Inference uses 30 denoising steps, guidance scale \(3.5\), and \(512\times512\) resolution. The shared-attention weights are \(\lambda_r=\lambda_p=1.1\). BirefNet performs segmentation; Qwen2-VL-7B-Instruct and Qwen2.5-7B-Instruct provide caption generation and filtering. The modified attention is applied in ten critical layers out of FLUX’s 57 layers. The main setup uses one reference image, and the code is released at `https://github.com/Monalissaa/FreeCus` [2507.15249].

## 6. Limitations, trade-offs, and significance

The main failure mode reported for FreeCus is the appearance of artifacts resembling outline residues or ghost traces of the reference subject, attributed to attention sharing. The authors experimented with position-index shifting of shared attention, inspired by OminiControl, but report that it reduced subject similarity. They also explored masking and random attention dropout; dropout provided the best artifact–fidelity compromise, but the issue remains unresolved [2507.15249].

A second limitation is dependence on MLLM caption quality. If the multimodal captioner misdescribes a rare subject or includes misleading attributes, semantic compensation can become imperfect. The method is primarily designed for single-subject customization. Multi-subject scenes are described as possible but not as the core setting. The paper also implies that severe prompt conflicts, strong occlusion, and challenging geometric transformations remain difficult, though these cases are not quantitatively dissected [2507.15249].

In methodological terms, FreeCus is significant because it treats subject-driven customization in DiTs as an inference-time control problem rather than a retraining problem. The framework is training-free and optimization-free in the sense that it avoids gradient-based latent optimization, test-time fine-tuning, embedding fitting, inversion optimization, and specialized subject-encoder training. At the same time, it is not computation-free: it still requires VAE encoding, segmentation, MLLM captioning, construction of a reference noising trajectory, and caching of attention tensors. This distinction is central to understanding its place in the personalization literature. It neither replaces the generative backbone nor leaves it untouched; instead, it selectively routes information already encoded inside a pretrained DiT [2507.15249].

Source: https://www.emergentmind.com/topics/freecus