Elastic Latent Interface Transformer (ELIT)

• The paper shows that ELIT decouples image resolution from compute by introducing a learnable latent interface, allowing a flexible trade-off between latency and quality.
• It employs lightweight Read and Write cross-attention layers to transfer information between spatial tokens and ordered latent tokens, focusing computation on critical image regions.
• Empirical results across various architectures demonstrate significant gains in FID, FDD, and IS metrics, confirming ELIT’s robust, architecture-agnostic improvements.

Searching arXiv for the specified ELIT paper and closely related diffusion transformer work to ground the encyclopedia entry. Elastic Latent Interface Transformer (ELIT) is a drop-in, DiT-compatible mechanism for diffusion transformers that inserts a learnable, variable-length latent interface between an initial spatial processing stage and a final spatial decoding stage, thereby decoupling input image size from compute while preserving the standard rectified flow objective and leaving the DiT stack unchanged. In the formulation introduced in "One Model, Many Budgets: Elastic Latent Interfaces for Diffusion Transformers" (Haji-Ali et al., 12 Mar 2026), ELIT addresses two linked limitations of conventional DiTs: FLOPs are rigidly tied to image resolution, and computation is allocated uniformly across spatial tokens even when some regions are less important. The method uses lightweight Read and Write cross-attention layers to move information between spatial tokens and latent tokens, trains the model with random dropping of tail latents to induce an importance ordering, and allows the number of active latents at inference to be adjusted to match compute constraints. Across datasets and architectures, including DiT, U-ViT, HDiT, and MM-DiT, the paper reports consistent gains; on ImageNet-1K 512px, it reports an average gain of $35.3\%$ and $39.6\%$ in FID and FDD scores (Haji-Ali et al., 12 Mar 2026).

1. Motivation and problem setting

The point of departure for ELIT is the observation that diffusion transformers achieve high generative quality but lock FLOPs to image resolution, which limits principled latency-quality trade-offs, and allocate computation uniformly across input spatial tokens, which wastes resource allocation to unimportant regions (Haji-Ali et al., 12 Mar 2026). In a vanilla DiT, the spatial token sequence directly determines the cost of the main transformer stack, so raising resolution raises compute in a rigid way. ELIT is designed to break that coupling.

The central claim is that compute can be mediated through a latent sequence whose length is not tied to the number of spatial tokens. This permits a single trained model to operate across multiple budgets by varying the latent count at test time. The paper presents this not as an auxiliary pruning or masking heuristic, but as an architectural mechanism: the model reads from spatial tokens into a latent interface, performs most transformer computation on the latent sequence, and then writes back to spatial tokens before producing the velocity prediction (Haji-Ali et al., 12 Mar 2026).

A recurrent misconception would be to treat ELIT as merely a token-dropping procedure inside a standard DiT. The paper instead defines a structured separation between spatial tokens and latent tokens, with explicit cross-attention operators and a training protocol that makes the latent sequence importance-ordered. This distinction matters because the reported latency-quality trade-off depends on learned latent organization rather than on a training-free reduction of the original spatial token set.

2. Architectural organization

ELIT augments a standard Diffusion Transformer by inserting a learnable, variable-length latent interface between a short spatial head and a short spatial tail. The architecture is divided into three segments (Haji-Ali et al., 12 Mar 2026):

• Short spatial head $(B_1$ blocks): converts the noised input $X_t \in \mathbb{R}^{N \times d}$, with $N$ spatial tokens, into a refined spatial embedding $s \in \mathbb{R}^{N \times d}$.
• Latent core $(B_2$ blocks): introduces $K$ latent tokens $l \in \mathbb{R}^{K \times d}$, referred to as the latent interface, on which most transformer blocks operate.
• Short spatial tail $(B_3$ blocks): takes the written-back spatial tokens and projects them to the velocity prediction space.

Because $39.6\%$0 is independent of $39.6\%$1, inference FLOPs become adjustable via $39.6\%$2, breaking the rigid image-resolution-to-compute link of vanilla DiT. This is the defining elastic property of the method. The latent interface is not a fixed bottleneck in the usual sense; rather, it is explicitly variable-length, so the same model can be evaluated under different budgets by choosing different numbers of latents (Haji-Ali et al., 12 Mar 2026).

The paper emphasizes that ELIT is deliberately minimal. It adds two cross-attention layers while leaving the rectified flow objective and the DiT stack unchanged. This minimality is significant because it situates ELIT as an interface mechanism rather than as a wholesale redefinition of the diffusion transformer backbone. A plausible implication is that the design is meant to preserve compatibility with existing DiT-family model families and training pipelines.

3. Read/Write cross-attention and grouped computation

The communication pathway between spatial tokens and latent tokens is implemented through two cross-attention layers, Read and Write. Let $39.6\%$3 denote the spatial tokens after the head, and $39.6\%$4 denote the latent interface. In pre-norm style with adaLN-Zero time conditioning and QK-normalization, the Read layer is defined as (Haji-Ali et al., 12 Mar 2026)

$39.6\%$5

$39.6\%$6

where $39.6\%$7 denotes multi-head cross-attention with separate learnable projections for queries, keys, and values. The Write layer is symmetric:

$39.6\%$8

$39.6\%$9

These operators implement the movement of information into and out of the latent core. Read pulls information from the spatial representation into the latent interface; Write returns the processed latent information to the spatial stream before prediction.

To control cross-attention cost, spatial tokens are grouped into $(B_1$0 non-overlapping windows. Each group shares $(B_1$1 latent tokens. Under this grouping, each cross-attention is computed in parallel across $(B_1$2 groups of size $(B_1$3 spatial tokens attending to $(B_1$4 latents, yielding a cost of $(B_1$5 rather than $(B_1$6 per layer (Haji-Ali et al., 12 Mar 2026). This grouped formulation is central to ELIT’s practicality: without it, the cross-interface communication could offset the computational savings gained by moving most blocks onto the latent sequence.

The paper also states that the Read attention learns to focus on the spatial tokens where flow-matching loss is highest, thereby automatically allocating more compute to “hard” regions. In that sense, the latent interface is not only a budget-control mechanism but also a learned mechanism for nonuniform resource allocation across the input (Haji-Ali et al., 12 Mar 2026).

4. Training objective and importance-ordered latents

ELIT uses the standard Rectified Flow (RF) objective unchanged. With $(B_1$7, $(B_1$8, and $(B_1$9, the velocity field is $X_t \in \mathbb{R}^{N \times d}$0, and the loss is (Haji-Ali et al., 12 Mar 2026)

$X_t \in \mathbb{R}^{N \times d}$1

The architectural intervention is therefore separated from the training objective: ELIT does not introduce an auxiliary optimization target to obtain elasticity. Instead, the key training mechanism is random dropping of tail latents. At each training iteration, the method uniformly samples a budget $X_t \in \mathbb{R}^{N \times d}$2 and keeps only the first $X_t \in \mathbb{R}^{N \times d}$3 latents, dropping the remaining tail in every Read layer, transformer block, and Write layer (Haji-Ali et al., 12 Mar 2026).

The purpose of this procedure is to enforce an importance ordering among the $X_t \in \mathbb{R}^{N \times d}$4 latents in each group. Since earlier latents are kept more often, they learn to store the most critical information, characterized in the paper as global structure, while later latents contain information that refines details. No auxiliary loss is added. The resulting latent sequence is therefore ordered by utility under budget variation rather than only by representational capacity (Haji-Ali et al., 12 Mar 2026).

This training rule also clarifies why ELIT is not equivalent to arbitrary latent truncation after training. The truncation behavior is explicitly anticipated during optimization, and the paper’s interpretation is that the model learns a coarse-to-fine organization of information across the latent index. At inference, degrading quality by dropping later latents is therefore intended to be smooth rather than catastrophic.

5. Inference-time elasticity and cheap guidance

At test time, ELIT exposes a direct budget parameter: the number of latents $X_t \in \mathbb{R}^{N \times d}$5 per group. The user chooses $X_t \in \mathbb{R}^{N \times d}$6 such that $X_t \in \mathbb{R}^{N \times d}$7, and the total latent count becomes $X_t \in \mathbb{R}^{N \times d}$8. The paper gives the per-step FLOPs as roughly proportional to (Haji-Ali et al., 12 Mar 2026)

$X_t \in \mathbb{R}^{N \times d}$9

In practice, $N$0 is tuned upward or downward to meet a desired FLOP or latency target. The paper describes the result as a continuous quality-compute curve that outperforms simply reducing sampling steps. This is an important point of comparison: the elastic budget is applied to representational capacity inside each denoising step, not only to the number of denoising steps themselves (Haji-Ali et al., 12 Mar 2026).

The paper further states that, at inference, dropping latents smoothly degrades quality: early latents capture global layout, and later latents fill in texture. This behavior is consistent with the training-induced importance ordering. Compared to mask-or-merge methods, ELIT continues improving beyond DiT’s quality even at $N$1 tokens, whereas training-free drop/merge stalls at the baseline (Haji-Ali et al., 12 Mar 2026). The comparison is framed as a distinction between learned latent budgeting and training-free token reduction.

ELIT also supports a form of Cheap Classifier-Free Guidance (CCFG), defined as

$N$2

where the unconditional path uses a smaller $N$3 guidance budget. The paper reports that CCFG matches or surpasses standard CFG at approximately $N$4 fewer FLOPs, without auxiliary models (Haji-Ali et al., 12 Mar 2026). This extends the elastic-budget principle beyond unconditional generation into the guidance mechanism itself.

6. Empirical results across image and video generation

The most detailed quantitative results reported in the summary concern ImageNet-1K at 512px with 40-step Euler sampling, evaluated both without guidance and with $N$5 CFG. All models use roughly the same train-step FLOPs, approximately $N$6 TFLOPs per iteration, and the standard RF loss (Haji-Ali et al., 12 Mar 2026).

For DiT-XL/2, the baseline configuration has $N$7M parameters and $N$8 TFLOPs at 512px, with FID $N$9 and $s \in \mathbb{R}^{N \times d}$0, FDD $s \in \mathbb{R}^{N \times d}$1 and $s \in \mathbb{R}^{N \times d}$2, and IS $s \in \mathbb{R}^{N \times d}$3 and $s \in \mathbb{R}^{N \times d}$4, for no-guidance and guidance settings respectively. ELIT-DiT has $s \in \mathbb{R}^{N \times d}$5M parameters and $s \in \mathbb{R}^{N \times d}$6 TFLOPs at 512px, with FID $s \in \mathbb{R}^{N \times d}$7 and $s \in \mathbb{R}^{N \times d}$8, FDD $s \in \mathbb{R}^{N \times d}$9 and $(B_2$0, and IS $(B_2$1 and $(B_2$2. ELIT-MB has $(B_2$3M parameters and $(B_2$4 TFLOPs at 512px, with FID $(B_2$5 and $(B_2$6, FDD $(B_2$7 and $(B_2$8, and IS $(B_2$9 and $K$0. The relative improvements listed in the summary for ELIT-MB are $K$1 and $K$2 in FID, $K$3 and $K$4 in FDD, and $K$5 and $K$6 in IS (Haji-Ali et al., 12 Mar 2026).

The summary also states that, on U-ViT-XL and HDiT-XL at approximately $K$7B parameters, ELIT-MB reduces FID by $K$8 to $K$9 and FDD by $l \in \mathbb{R}^{K \times d}$0 to $l \in \mathbb{R}^{K \times d}$1 while boosting IS by $l \in \mathbb{R}^{K \times d}$2 to $l \in \mathbb{R}^{K \times d}$3. On video generation for Kinetics-700 at 256px and 29 frames, ELIT-DiT improves FID from $l \in \mathbb{R}^{K \times d}$4 to $l \in \mathbb{R}^{K \times d}$5 $l \in \mathbb{R}^{K \times d}$6 and FVD from $l \in \mathbb{R}^{K \times d}$7 to $l \in \mathbb{R}^{K \times d}$8 $l \in \mathbb{R}^{K \times d}$9 (Haji-Ali et al., 12 Mar 2026).

These results are used in the paper to support two broader claims. First, ELIT is presented as architecture-agnostic within the diffusion-transformer family, since gains are reported across DiT, U-ViT, HDiT, and MM-DiT. Second, the gains are not restricted to a single metric: improvements are reported in FID, FDD, IS, and FVD. This suggests that the latent interface is functioning both as a compute-control mechanism and as a more selective representational pathway.

7. Interpretation within diffusion-transformer research

Within the terms of the paper, ELIT can be understood as a method for replacing uniform spatial-token processing with a learned, elastic latent bottleneck whose width is selectable at inference time. The mechanism is “drop-in” because it retains the standard RF loss and keeps the DiT stack unchanged apart from the insertion of the latent interface and two cross-attention layers (Haji-Ali et al., 12 Mar 2026). The method’s contribution is therefore architectural and procedural rather than objective-level.

A central conceptual feature is the importance-ordered latent set. Earlier latents are trained to capture global structure, later latents to refine details, and random tail dropping makes that ordering operational under budget changes. This yields a continuous quality-compute trade-off that the paper describes as far superior to either step-count scheduling or naïve token masking (Haji-Ali et al., 12 Mar 2026). The phrase “one model, many budgets” is therefore descriptive: a single set of parameters is intended to support multiple deployment regimes without retraining separate models for each latency target.

Another point of clarification concerns region prioritization. The paper does not describe ELIT as explicitly supervised to detect salient regions or as using an auxiliary saliency loss. Instead, it reports that Read attention learns to focus on the spatial tokens where flow-matching loss is highest, thereby allocating more compute to difficult regions (Haji-Ali et al., 12 Mar 2026). A plausible implication is that the model’s budget allocation emerges from the generative training signal itself rather than from an externally imposed notion of importance.

In summary, ELIT denotes a latent-interface formulation for diffusion transformers in which compute is decoupled from resolution, latent tokens are trained to be importance-ordered through random tail dropping, and inference exposes a direct latent-budget control that supports dynamic latency-quality trade-offs, region-sensitive computation, and cheaper guidance within a single model family (Haji-Ali et al., 12 Mar 2026).