Text Template Tokens Are Implicit Semantic Registers in Diffusion Transformers

Abstract: Text-to-image diffusion transformers (DiTs) jointly process text and image tokens, yet their internal computation during denoising remains poorly understood. We introduce a causal interpretability framework for modern large-scale DiTs that combines attention decomposition with targeted interventions across token spans, heads, and layers. Using it to separate prompt-content tokens from structural template tokens, we find that the structural tokens carry little prompt-specific information at the encoder output. Yet surprisingly, they emerge as dominant image-to-text attention sinks and causally maintain object identity inside the DiT, acting as implicit semantic registers. We show that they acquire this identity indirectly, with prompt semantics first injected into the image latents and then read back into the template tokens rather than transferred directly from the prompt tokens. Inspired by the above findings, we design a training-free pruning rule for DiTs. Heads that attend most strongly to prompt tokens are dispensable, and pruning them removes $20\%$ of attention FLOPs with only a $1.4$-point drop on GenEval. We further reveal how generative computation in DiTs is organized across heads and depth, separating semantic routing from visual synthesis and progressing from identity formation to propagation and refinement. Our work not only reveals that the tokens encoding semantics at input need not be those that maintain it during generation, but also provides a causal view of internal mechanisms in DiTs.

• The paper demonstrates that structural chat-template tokens dominate image-to-text attention—receiving 76–78% of attention on GenEval—yet acquire and preserve object identity indirectly through image latents rather than prompt content.
• Causal interventions show that prompt semantics enter the image stream in early layers, while register heads maintain identity across denoising and rendering heads organize visual coherence through early commitment, middle propagation, and late refinement.
• A training-free pruning method that removes heads with high semantic-span attention cuts up to 20% of joint-attention FLOPs with only a 1.4-point GenEval decrease, although perceptual quality and generalization remain limitations.

Motivation and scope

Modern text-to-image systems have shifted from U-Net backbones with cross-attention conditioning toward multimodal diffusion transformers (MMDiTs), where text and image tokens interact through a single joint attention stream. In parallel, text conditioning has moved from CLIP or T5 encoders to hidden states of large vision–LLMs (VLMs), in which the user prompt is serialized into a chat template containing system, user, assistant, and delimiter tokens. These structural tokens have conventionally been treated as formatting residue: necessary for the encoder interface but not part of the semantic request. This paper asks which of the three plausible hypotheses holds—structural tokens as inert residue, as transparent conditioning tokens, or as deeper computational sites—and answers with the strongest of the three, using the Qwen-Image family (Qwen-Image, Qwen-Image-2512) as its primary subject, with extensions to FLUX.2, Krea-2-Turbo, few-step distilled variants, and an instruction-editing model.

A causal interpretability framework

The authors combine four intervention types: token-level decomposition of the joint attention matrix into T2T, T2I, I2T, and I2I blocks, with attention mass $m^{(l,h)}_{\mathcal{K}}$ measured from image queries to text key spans; span-level conditioning interventions (cross-prompt swaps and averaging of token spans); cross-trajectory head transplantation, in which per-head $(q,k,v)$ projections are progressively copied between two denoising trajectories with different prompts; and layer-wise causal masking of specific query-to-key attention before the softmax. The conditioning sequence is split into a semantic span $\mathcal{S}$ of prompt-content tokens and a trailing structural span $\mathcal{R}$ of chat-template tokens, and the I2T block—the cross-modal pathway—is the primary object of measurement.

Structural tokens are dominant attention sinks

On the minimal prompt "An apple" ($|\mathcal{S}|=2$, $|\mathcal{R}|=5$), image queries place an order of magnitude more attention on the content-free template span than on the prompt content: the mean structural mass $\bar m_{\mathcal{R}}$ is 0.23 (Qwen-Image) versus $\bar m_{\mathcal{S}} = 0.020$, and 0.21 versus 0.016 on Qwen-Image-2512—an 11–13× span-level gap and 4.6–5.3× per token. The top sink head reaches $m_{\mathcal{R}} = 0.998$, and the single delimiter <|im_end|> absorbs 12–13% of all image-query attention at the highest noise level, more than 6× the content noun.

This is not an artifact of short prompts. Across GenEval (553 prompts, mean 7.9 content tokens), DPG-Bench (1,065 prompts, mean 82.1 content tokens), and Qwen-Image-Bench (1,000 Chinese prompts, mean 60.5 content tokens), the per-token structural-to-semantic ratio remains stable at roughly 3–7.5×. Span-level mass inverts on long prompts simply because semantic mass is diluted over many tokens, but the per-token sink strengthens with prompt length (7.2× on Qwen-Image at DPG-Bench versus 6.4× on GenEval), and the pattern holds across languages. On GenEval, 76–78% of all I2T attention lands on the template span, and a structural token is the most-attended text token at ~90% of (timestep, layer, head) sites. The appendix extends the result to FLUX.2 (92% of I2T attention on 33 template tokens, structural dominance at 99.8% of sites), Krea-2-Turbo (67%, per-token ratio 3.1×), the 2-step distilled checkpoint (77%, within a point of the undistilled model), and the editing model, where each structural token draws 7.7× the attention of a vision token and 194× that of a reference-image latent. The sink is thus robust across models, encoders, architectures, sampling schedules, prompt complexity, and language.

Depth and temporal structure suggest a learned rather than input-driven mechanism: per-token structural mass is near zero in the first block, crosses half its plateau only around block 5–11, peaks near block 20, and grows relative to semantic mass over the denoising trajectory (from ~5–6× at high noise to ~8× near $t=0$). The authors argue this onset pattern is inconsistent with a mere large-norm embedding artifact, though they do not fully establish the alternative.

Template tokens carry little semantics

Despite absorbing most I2T attention, the structural span carries almost no prompt-specific information at the encoder output. A cross-prompt swap $(q,k,v)$0—substituting prompt $(q,k,v)$1's template tokens into prompt $(q,k,v)$2's sequence—leaves the generated image faithful to $(q,k,v)$3. More strikingly, replacing $(q,k,v)$4 with a single prompt-agnostic average computed from 100 GenEval prompts changes the structural representation by a relative $(q,k,v)$5 of 0.36 yet preserves the object, with median own-vs-average DINOv3 similarity of 0.99 (Qwen-Image) and 0.96 (Qwen-Image-2512). The rare failures collapse to a generic human portrait—the model's unconditional default, reproduced by generating from an empty prompt—and are largely suppressed by classifier-free guidance. The authors interpret the residual effect as a thresholding phenomenon rather than evidence of meaningful template semantics, and note the caveat that the degree of perturbation varies across prompts and seeds.

Registers acquire identity implicitly

The central causal result contradicts the conventional "semantic-head" picture. In the progressive head-swap experiment between "An apple" and "A banana" trajectories, transplanting heads ranked by descending image-to-semantic attention $(q,k,v)$6 fails to transfer object identity and instead collapses generation toward the unconditional portrait. The reverse order—starting with heads that barely read $(q,k,v)$7—flips apple to banana with only ~18% of the 1,440 heads. Reading the prompt and carrying the object are therefore causally decoupled.

Layer-wise masking explains how the registers acquire their content. Blocking $(q,k,v)$8 attention leaves the object essentially intact; blocking $(q,k,v)$9 (register attention to image latents) collapses the object within the first two blocks. Masking only the first block has no effect, implying that the prompt-to-image semantic injection ($\mathcal{S}$0) completes within the first block, after which the template tokens read the identity back from the image stream rather than from the prompt tokens. The registers are thus implicit: the tokens that encode semantics at input are not the ones that maintain it during generation. A per-span progressive swap in the appendix qualifies this: a text-side-only swap of $\mathcal{S}$1 disrupts the image early but does not reproduce the target object, indicating that part of the semantics is carried by $\mathcal{S}$2 and the image tokens as well.

Training-free head pruning

Because high-$\mathcal{S}$3 heads are causally inert, ranking all 1,440 heads by descending $\mathcal{S}$4 and pruning the top $\mathcal{S}$5 over the last 80% of denoising steps yields a prompt-independent acceleration. Pruning $\mathcal{S}$6 heads (a quarter of all heads) removes 20% of joint-attention FLOPs at a cost of only 1.4 GenEval points (76.1 → 74.7); $\mathcal{S}$7 gives the best trade-off (75.8 GenEval, 12% FLOPs, LPIPS 0.21). Head selection matters decisively: at $\mathcal{S}$8, GenEval falls from 75.5 under the $\mathcal{S}$9 ranking to 69.6 under the structural-sink ranking and 51.3 under random selection, with HPSv3 dropping from 9.29 to 8.03 and 4.86 respectively. Perceptual quality degrades faster than object correctness, which the authors acknowledge as a limitation of the rule.

Division of labor across heads and depth

Heads separate into two functional groups. Register heads encode and hold object identity across the trajectory, while rendering heads concentrate attention in the I2I block and knit latents into spatial coherence; ablating even a small fraction of the latter collapses output into incoherent patches, though a few heads blend both roles. Across depth, the ~270 heads that most drive identity transfer are bimodally distributed (89 in blocks 1–10, 85 in blocks 51–60, 96 thinly spread across blocks 11–50). Early heads alone flip the object cleanly in both directions (identity commitment), middle heads leave it nearly unchanged (propagation), and late heads flip it only with residual distortion (refinement). This early-commit/middle-carry/late-refine organization mirrors stage structure reported in LLMs, and the paper presents it as descriptive evidence rather than a fully mechanistic account.

Limitations and open questions

The paper is candid that the pruning rule is a first-cut application; sink-aware sparse attention that preserves register positions as keys while pruning the remainder is proposed as a likely stronger design but is not evaluated. The causal mechanism experiments are conducted on a single denoising step of a 2-step distilled LoRA checkpoint to fix the timestep, and the generalization of the implicit-acquisition result to full multi-step schedules is asserted rather than exhaustively demonstrated. Most fundamentally, the paper leaves open why content-free structural tokens come to serve as semantic registers at all, and its depth-localization argument against an input-artifact explanation, while suggestive, is not conclusive.

Conclusion

This paper establishes that chat-template structural tokens in LLM-conditioned diffusion transformers are not formatting residue: they are dominant image-to-text attention sinks that causally function as implicit semantic registers, acquiring object identity indirectly from the image latents after prompt semantics are injected in the first block. The decoupling of semantic reading from semantic carrying yields a practical training-free pruning rule (20% attention FLOPs removed at 1.4 GenEval points) and a coherent account of generative computation organized as register-mediated identity routing distinct from visual synthesis, deployed sequentially as early commitment, middle propagation, and late refinement. The work provides a causal template for mechanistic analysis of diffusion transformers and identifies the origin of register behavior as its principal open question.