Determine whether stochastic-tokenization benefits are specific to unigram modeling

Determine whether the policy-learning benefit of stochastic segmentation arises specifically from the unigram language model used by TOAST or instead from stochastic segmentation more generally, including alternative schemes such as BPE-dropout.

Background

The experiments instantiate stochastic action tokenization with a unigram model and subword regularization. Although TOAST improves over deterministic tokenization, the study does not compare alternative stochastic tokenization mechanisms such as BPE-dropout. Consequently, it remains unresolved whether the observed improvement is attributable to the unigram model itself or to the broader principle of sampling among equivalent segmentations.