Explore selective cross-script byte sharing

Develop and evaluate a selective byte-sharing mechanism for Universal Byte-Level Encoding that mitigates regressions in high-coverage languages while preserving marker-free decoding through disjoint alphabet identification.

Background

Universal Byte-Level Encoding uses separate UTF-8 and UTF-16 byte alphabets to preserve marker-free decoding. This design sacrifices some coincidental byte sharing that standard BBPE can exploit across scripts and around universal characters such as whitespace and punctuation.

The paper reports that this trade-off occasionally harms high-coverage languages, including Georgian and Amharic. The authors explicitly identify selective sharing of a small set of universal characters as a future direction, subject to retaining the decoding guarantee.

References

future work could explore selective sharing for a small set of universal characters to mitigate these regressions while preserving the disjoint-alphabet decoding guarantee.

— Universal Byte-Level Encoding: UTF-8/UTF-16 Routing to Reduce Cross-Script Token-Budget Disparities  (2610.01984 - Kim et al., 1 Oct 2026) in Appendix H.4, “Cross-script byte sharing”