Bidirectional Tokenization Strategy

• The paper demonstrates that integrating bidirectional context significantly improves tokenization accuracy in noisy text, numeric reasoning, and speech processing.
• It employs dual-stream models, such as BiLSTM for text and state-space models for speech, to merge past and future cues for more robust token boundary detection.
• Empirical evaluations reveal notable gains in F-score, arithmetic accuracy, and spoken term detection metrics compared to traditional unidirectional approaches.

A bidirectional tokenization strategy is a modeling paradigm in which token boundaries or discrete representations are determined by integrating both past (left) and future (right) context within a data stream. Unlike unidirectional approaches, bidirectional methods access information from the full sequence when making boundary or token predictions. This paradigm has been successfully applied in diverse domains, including tokenization repair in noisy text (Bast et al., 2020), numeric chunking for improved arithmetic reasoning in LLMs (Singh et al., 2024), and robust speaker-invariant speech tokenization for spoken term detection (Singh et al., 2024). The bidirectional strategy improves consistency, context sensitivity, and empirical accuracy compared to unidirectional baselines.

1. Bidirectional Tokenization in Noisy Text Segmentation

The work "Tokenization Repair in the Presence of Spelling Errors" introduces a bidirectional strategy for repairing tokenization in sequences suffering from arbitrary space insertions/deletions—typically arising in OCR, PDF extraction, or human error scenarios (Bast et al., 2020). The core architecture consists of two modules:

• A unidirectional character-level LSTM LLM ("UNI") computes left-context language probabilities for character sequences.
• A bidirectional boundary-labeler ("BID")—implemented as a BiLSTM boundary predictor—assigns, at each character position, the probability $\hat i_i$ that a space should be inserted before that character. Unlike unidirectional models, the BiLSTM leverages both past and future character sequences to make its prediction.

During inference, a beam search process generates candidate sequences by integrating LLM scores, bidirectional boundary probabilities, and penalties based on agreement with observed (possibly corrupted) space positions. The resulting decoding is linear in sequence length and beam size.

Empirically, the bidirectional approach (BID+) outperforms both the unidirectional LLM (UNI+) and all classical baselines across six benchmarks, improving F-score and sequence accuracy by several percentage points (e.g., F = 96.1% for arXiv OCR with UNI+ vs. 97.5% with BID+) (Bast et al., 2020).

2. Bidirectional Numeric Tokenization in LLMs

"Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs" examines numeric tokenization’s directionality and its impact on arithmetic computations in LLMs (Singh et al., 2024). Although the method is not symmetrical in the processing direction, the analysis explicitly compares left-to-right (L2R) and right-to-left (R2L) chunking of numbers into k-digit tokens (where k=3 in GPT-3.5/4 tokenizers):

• L2R: partitions numbers into most-significant-to-least-significant chunks.
• R2L: partitions from least-significant side, outputting tokens in canonical order but reflecting R2L segmentation.

Crucially, enforcing R2L tokenization at inference (e.g., by comma separation) yields large gains in addition accuracy (for GPT-3.5 turbo, 75.6% → 97.8%) and eliminates characteristic error syndromes at L2R token boundaries. Chain-of-thought-style conversion from L2R input to R2L representation before arithmetic further achieves nearly maximal performance.

This suggests that the model’s token-level computation is inherently sensitive to chunking direction, with R2L chunking (alignment with human carry propagation in addition) providing an inductive bias for more reliable arithmetic (Singh et al., 2024).

3. Bidirectional Semantic Tokenization in Speech

"BID-STD: Bidirectional Mamba-Enhanced Speech Tokenization for Spoken Term Detection" demonstrates a bidirectional state-space modeling (SSM) approach for encoding audio into discrete semantic tokens, facilitating efficient and speaker-invariant spoken term retrieval (Singh et al., 2024):

• The method employs parallel forward and backward Mamba SSM encoders, yielding context-aware frame embeddings aggregated from both past and future acoustic observations.
• Embeddings are quantized against a learned codebook (VQ-EMA), generating discrete token sequences.
• Self-supervised losses (contrastive and commitment) train the encoder to produce tokenizations that are consistent for the same term across utterances and speakers.

Empirical evaluation shows that bigram Jaccard similarity of tokens for the same word rises significantly under bidirectional Mamba (0.77) compared to unidirectional Transformer models (0.64). Retrieval metrics (MAP, MRR, MTWV) on LibriSpeech and TIMIT again favor the bidirectional strategy, with MAP for in-vocabulary queries up to 0.86.

The increased consistency and speaker invariance are attributed to the model’s capacity to propagate contextual cues bidirectionally, smoothing out local variations, and aligning representations for the same underlying phonetic content across speakers (Singh et al., 2024).

4. Mathematical Formalism and Decoding Algorithms

Bidirectional tokenization strategies are typically formalized by hybrid or dual-stream scoring functions that combine left-to-right generative probabilities with bidirectional boundary posteriors. In text repair (Bast et al., 2020), the decoding objective becomes: $$R^* = \arg\min_{R}\Bigl[-\log p(R) + n_{\rm ins}P_{\rm ins} + n_{\rm del}P_{\rm del}\Bigr]$$ where $p(R)=\prod_{i} p(R_i\mid R_{1:i-1})$ (uniLM), and insert/delete penalties are conditioned on space agreement with the input.

In bidirectional decoding, the boundary probabilities $\hat i_i$ modulate extension scores: $$S_i = S_{i-1} - \log \bigl[p(T_i \mid R_{1:i-1})(1-\hat i_i)\bigr] + P_{\rm del}$$ for no-space extensions, and

$$S_i' = S_{i-1} - \log \bigl[p(T_i\mid R_{1:i-1})\hat i_i\bigr] + P_{\rm ins}$$

for space insertions.

In speech tokenization (Singh et al., 2024), forward and backward SSMs produce hidden states $h_t^{\to}$ and $h_t^{\gets}$, which are concatenated and projected to obtain $z_t$ for quantization. The codebook assignment is $z_q = \arg\min_{c_i \in C} \|z - c_i\|_2^2$.

5. Empirical Results and Comparative Evaluations

The following table summarizes representative results for bidirectional tokenization strategies across domains:

Domain	Metric	Unidirectional	Bidirectional	Reference
Text Repair	F-score (OCR)	96.1%	97.5%	(Bast et al., 2020)
Text Repair	Seq. Accuracy	92.8%	94.1%	(Bast et al., 2020)
LLM Arithmetic	Accuracy (8-shot)	75.6% (L2R)	97.8% (R2L)	(Singh et al., 2024)
Speech STD	Bigram Jaccard	0.64	0.77	(Singh et al., 2024)
Speech STD	MAP (LibriSpeech)	—	0.86	(Singh et al., 2024)

These results consistently indicate that incorporating bidirectional context yields substantial improvements over corresponding unidirectional strategies, both in boundary precision and in representation consistency.

6. Implications, Limitations, and Modal Extensions

Bidirectional tokenization improves context sensitivity, stability, and invariance in token representations and boundary detection. Context aggregation enables more accurate handling of ambiguities, long-range dependencies, and local noise.

Nevertheless, bidirectional methods are not universally applicable. They require access to the full input context—precluding online, streaming, or real-time requirements where future data may not be available. There is an inference-time cost due to dual passes in encoder architectures (e.g., BiLSTM, bidirectional SSM), though implementations remain efficient in non-streaming settings (Bast et al., 2020, Singh et al., 2024).

Practical guidance includes explicit evaluation of tokenization direction or boundary modeling in new domains, enforcing R2L chunking for numeric reasoning, leveraging inverted indices for token sequence retrieval, and using ablations to understand inductive biases induced by tokenization schemes (Singh et al., 2024, Singh et al., 2024).

Furthermore, strategies presented for text and speech can plausibly generalize to other sequential or multimodal settings—such as video or time-series sensor data—where bidirectional aggregation may similarly enhance the stability and discriminative power of discrete tokenizations (Singh et al., 2024).

7. Conclusion

Bidirectional tokenization strategies integrate information from both directions to make boundary or code assignments, improving upon unidirectional models in accuracy and robustness. In text, numeric, and speech domains, these strategies consistently outperform alternatives, indicating the importance of full-context modeling in modern tokenization pipelines. Their effectiveness is evidenced by significant accuracy gains, speaker invariance, robust error profiles, and computational efficiency when coupled with efficient decoding or retrieval schemes (Bast et al., 2020, Singh et al., 2024, Singh et al., 2024). Future methodological development should incorporate bidirectionality as a standard component in any domain where full context can be accessed at inference.