ELT: Elastic Looped Transformers for Visual Generation

Abstract: We introduce Elastic Looped Transformers (ELT), a highly parameter-efficient class of visual generative models based on a recurrent transformer architecture. While conventional generative models rely on deep stacks of unique transformer layers, our approach employs iterative, weight-shared transformer blocks to drastically reduce parameter counts while maintaining high synthesis quality. To effectively train these models for image and video generation, we propose the idea of Intra-Loop Self Distillation (ILSD), where student configurations (intermediate loops) are distilled from the teacher configuration (maximum training loops) to ensure consistency across the model's depth in a single training step. Our framework yields a family of elastic models from a single training run, enabling Any-Time inference capability with dynamic trade-offs between computational cost and generation quality, with the same parameter count. ELT significantly shifts the efficiency frontier for visual synthesis. With $4\times$ reduction in parameter count under iso-inference-compute settings, ELT achieves a competitive FID of $2.0$ on class-conditional ImageNet $256 \times 256$ and FVD of $72.8$ on class-conditional UCF-101.

• The paper introduces Elastic Looped Transformers (ELT) as a recurrent model that reuses transformer blocks to drastically reduce parameters while maintaining high synthesis quality.
• It proposes Intra-Loop Self Distillation (ILSD) to regularize intermediate loop outputs, ensuring robust predictions and enabling dynamic Any-Time inference.
• Experimental results demonstrate state-of-the-art efficiency on tasks like ImageNet image generation and UCF-101 video generation with significant throughput gains.

Elastic Looped Transformers for Visual Generation

Introduction

The paper "ELT: Elastic Looped Transformers for Visual Generation" (2604.09168) introduces Elastic Looped Transformers (ELTs), a highly parameter-efficient recurrent transformer model for high-fidelity image and video generation. This work reframes standard deep transformer architectures by reusing a compact set of transformer blocks iteratively within each generative step, drastically reducing parameter count without sacrificing synthesis quality. To enable robust generation quality at every recursion depth, ELT models are trained with a novel Intra-Loop Self Distillation (ILSD) procedure, ensuring that intermediate loop outputs remain informative and can be used for Any-Time (elastic) inference—allowing the model to dynamically trade off between computational cost and generation quality at test time.

Architectural Innovations

Recurrent and Parameter-Efficient Design

Standard generative transformers use deep stacks of unique layers, resulting in a significant memory and parameter burden. ELT models define a composite block of $N$ unique transformer layers, which is recursively looped $L$ times per generation step, such that the effective depth is $N \times L$ but the unique parameter set is that of only $N$ layers. This disassociates the model’s parameter count from its computational depth, providing two key benefits: extreme parameter efficiency and high compute throughput by reducing memory bandwidth bottlenecks associated with sharded deep models.

Intra-Loop Self Distillation (ILSD)

Training recurrent architectures is non-trivial, as intermediate states may be degenerate or uninterpretable if solely supervised at the final iteration. ELT employs ILSD, where the output at the maximal loop count (teacher) supervises intermediate loop outputs (students) via a blended objective comprising both ground-truth and distillation losses. This forces the model to compress relevant computations into earlier loops and produces meaningful predictions at every loop depth, essential for robust Any-Time inference.

Figure 1: The ELT framework: Training involves recurrent looping with ILSD, aligning intermediate representations for Any-Time inference via early exit at any loop.

Any-Time Inference

A core property of the ELT architecture is the ability for Any-Time inference: at test-time, the loop count $L$ per generative step can be dynamically selected to match available computation or latency constraints, yielding a single trained model that implicitly realizes a continuous spectrum of accuracy-efficiency tradeoffs. This is achieved without architectural modifications or retraining, but instead emerges due to the regularization imposed by ILSD.

Figure 2: Latent trajectories during sampling: With ILSD (right), intermediate states are progressively closer to the target, allowing flexible early stopping.

Experimental Results

Image Generation: ImageNet 256x256

ELT models demonstrate state-of-the-art parameter efficiency for class-conditional generation tasks. For instance, an ELT-XL configuration with only 111M parameters achieves FID = 2.0 on ImageNet 256×256, matching much larger baselines (e.g., MaskGIT-XL with 446M parameters) under iso-inference-compute settings. This represents a 4× reduction in parameter count for equivalent generative quality.

Figure 3: ELT enables robust, high-quality image generation across a spectrum of compute budgets via dynamic loop count selection.

The flexibility of compute allocation is captured by the pareto frontier between FID and inference GFLOPs. ELT models scale efficiently with loop count, and the optimal FID at a given compute budget is always near the black pareto front—demonstrating that increases in loop count translate to higher quality, but with diminishing returns.

Figure 4: Pareto front for FID vs GFLOPs on ImageNet; ELT configurations traverse the curve through loop scaling.

Model Scaling

Investigations into the relationship between model width, unique layer count $(N)$, and depth $(L)$ reveal power-law scaling trends, with most performance gains attributable to increased width. Nonetheless, within a bounded parameter budget, recursive looping enables smaller models to approximate the FID of much larger baselines, especially at deployment-relevant scales.

Figure 5: Best-achievable FID vs. parameter count: recursive looping (ELT) enables small models to compete with large networks.

Training Efficiency and Throughput

Due to parameter sharing, ELT models exhibit up to 3.5× throughput gains compared to baselines on TPU hardware for the same compute budget, as weights can be kept on-chip and memory transfers are minimized. Further, ELTs converge significantly faster—requiring fewer optimization steps to reach low FID—than large, fully unrolled diffusion transformers.

Figure 6: ELT trains faster and more efficiently than comparable-size baselines in diffusion frameworks.

Video Generation: UCF-101

For class-conditional video generation, compact ELT models (76M parameters) outperform more complex architectures. An ELT configuration achieves FVD = 72.8 on UCF-101, compared to FVD = 76 from MAGVIT-L (306M parameters) at similar inference compute. This demonstrates that looped transformers can provide robustness and regularization in data-constrained regimes, resisting overfitting while maintaining high expressiveness.

Figure 7: ILSD yields substantial improvements across all tested loop counts in video generation.

Effect of ILSD

Ablations show that vanilla looped transformers (without ILSD) rapidly lose generation quality for any $L \neq L_{max}$, whereas ELT maintains stable, high-fidelity outputs for all loop counts, and can even generalize to $L > L_{max}$. This confirms the necessity of intra-loop supervision for robust elastic inference.

(Figure 3) again, showing visual results.

Implications and Future Directions

On the practical side, ELT enables deployment of high-quality generative models in resource-constrained environments (mobile, edge) due to extreme parameter efficiency and dynamic compute allocation. Theoretically, the demonstrated scaling properties indicate that looped transformers can reliably match standard scaling laws, provided a sufficiently large block size is maintained. Furthermore, ILSD regularization appears to provide modest extrapolation capabilities even beyond training depths.

Future research directions include:

• Combining ELT with fast-sampling and few-step approaches (e.g., Consistency Models, progressive distillation) for jointly reducing the number of denoising/sampling steps and the compute per step.
• Integrating ELT with adaptively scheduled sampling in diffusion models, enabling allocation of compute to the most challenging denoising intervals.
• Extending ILSD to transformer architectures outside the visual domain, such as language modeling or multimodal tasks.
• Systematic exploration of the limits of loop extrapolation as a regularization and robustness mechanism.

Conclusion

Elastic Looped Transformers provide a compelling architecture for efficient visual generation, surpassing existing baselines in parameter efficiency while enabling Any-Time inference through a single, regularized training run. The proposed ILSD regime ensures robust generation at all loop depths, and empirical results confirm that recursive looping, when combined with proper regularization, yields models that are both performant and deployment-ready for variable compute budgets. ELT represents a substantial advancement toward scalable, flexible, and hardware-efficient generative models.