Necessity of Tokenization for High-Energy-Physics Data

Determine whether tokenization is necessary for high-energy-physics data, particularly in multimodal settings, where discrete representations may offer scaling and uniform-representation advantages over continuous inputs.

Background

The paper studies discrete tokenization of several high-energy-physics detector modalities and quantifies the resulting rate–distortion trade-offs and downstream performance costs. However, the broader question of whether tokenization is necessary remains unresolved, especially for multimodal foundation models and at data scales beyond those addressed by existing comparisons.

The question is motivated by prior evidence that continuous generative decoders can outperform tokenized objectives at current scale, while a direct comparison in the large-data and multimodal regimes has not yet been established. Such a comparison would clarify whether the compression and architectural-unification benefits of discrete sequences justify their information loss and implementation costs.

References

The community has begun exploring tokenization and scaling, but whether tokenization is necessary, especially in multi-modal settings, remains an open question.

— Precision Tokenization for Multi-Modal Data in High Energy Physics  (2610.13876 - Krupa et al., 9 Oct 2026) in Section 1, Introduction

Continuous generative decoders have been shown to outperform tokenized objectives at current scale, although the comparison has not yet been made in the large-data regime and in multi-modal settings where the scaling and uniform representation advantages of discrete sequences are expected to emerge.

— Precision Tokenization for Multi-Modal Data in High Energy Physics  (2610.13876 - Krupa et al., 9 Oct 2026) in Section 1, Introduction