Interaction between tensorization and other efficiency methods

Determine whether tensorization and neighboring language-model efficiency techniques yield additive gains in practice and characterize how their approximation or optimization errors compose.

Background

Tensorization can be combined with quantization, pruning, knowledge distillation, and other efficiency techniques because these methods modify different aspects of a model, such as structure, numerical precision, sparsity, or training signals.

However, the combined effect cannot be inferred by multiplying independently reported improvements. The order of applying the methods may alter factor distributions, sensitivities, approximation quality, and hardware bottlenecks; errors may amplify, cancel, or interact as an implicit regularizer.

References

Most of them are orthogonal to tensorization in principle, but their practical interaction, including whether the gains add up and how the errors compose, remains an open question.

— Tensor Methods for Language Models: From Token Representation to Training, Adaptation, Inference, Compression, and Interpretability  (2608.30505 - Tarasov et al., 31 Aug 2026) in Section 7.2, paragraph “Compatibility with neighboring efficiency methods”