Codomain Tokenization in Neural Modeling

• Codomain tokenization is the process of mapping raw, continuous or structured inputs (e.g., text, images, audio) into discrete tokens from a fixed codebook.
• It drives neural model efficiency by defining token granularity and representational capacity, thereby optimizing downstream generation and comprehension tasks.
• Applications span NLP, multimodal transformers, and scientific modeling, with adaptive strategies enhancing domain-specific performance while reducing computational costs.

Codomain tokenization refers to the process of mapping raw, continuous, or structured input data (such as text, images, audio, or functions) into a discrete set of tokens drawn from a fixed vocabulary or codebook—the codomain of the tokenizer. This operation is foundational in neural modeling because it defines the atomic units upon which subsequent computational modules operate, determines information granularity, and constrains the representational capacity for both generation and comprehension tasks. Codomain tokenization is employed across a spectrum of modalities and architectures, including autoregressive LLMs, multimodal transformers, visual tokenizers, and neural operators for PDEs.

1. Formalization and Mechanisms of Codomain Tokenization

The codomain, $C$, of a tokenizer is its set of possible token outputs, which may correspond to subwords, bytes, codebook indices, or function channels depending on context (Mielke et al., 2021, Jia et al., 18 Feb 2025, Rahman et al., 19 Mar 2024).

Pipeline Overview

Stage	Operation	Typical Model Component
Encoding	$z = \mathrm{Enc}(x)$	CNN, Transformer, MLP
Quantization	$j = \arg\min_{k} D(z, c_k)$	Codebook $C \subseteq \mathbb{R}^{m \times d}$
Decoding (optional)	$\hat{x} = \mathrm{Dec}(c)$	Transposed encoder or autoregressive model

• Encoding: Raw input $x$ is mapped into latent space $z$.
• Quantization: Latent vectors are discretized via nearest-neighbor or attention-based assignment to codebook elements $\{c_k\}$.
• Decoding: Reconstruction or downstream modeling, often supervised by signal losses.

Distinct quantization strategies modify the mapping to $C$:

• Vanilla: Direct lookup.
• Residual: Errors are sequenced and quantized, $r_{s+1} = r_s - C_s(r_s)$.
• Product: Split sub-vectors quantized independently.
• Lookup-free: Scalar rounding (FSQ, LFQ).

For function spaces, as in CoDA-NO (Rahman et al., 19 Mar 2024), the codomain is indexed by physical variables and functions:

$$w(x) = [w_1(x), w_2(x), \ldots, w_L(x)],\qquad w_i: D \to \mathbb{R}^{d'}$$

Each $w_i$ is a token in function space, allowing flexible modeling of variable channels.

2. Tokenization Algorithms and Theoretical Properties

Subword tokenizers (BPE, WordPiece, UnigramLM) define canonical codomains for NLP (Mielke et al., 2021, Cognetta et al., 21 Oct 2024):

• Byte-Pair Encoding (BPE):

  Iteratively merges most frequent pairs:

  $V_{t+1} = V_t \cup \{\text{merged symbol}\}$

  Codomain grows to encompass frequent subwords/morphemes.

• Finite-State Transduction Representation: (Cognetta et al., 21 Oct 2024) Tokenization as FST $$\mathcal{T} = (\Sigma, \Gamma, Q, q_{\mathrm{start}}, F, \delta)$$, composing character-level patterns and subword lexicons:

  $$\min\left(\mathrm{Proj}\left(\mathcal{A} \circ \mathcal{T}\right)\right)$$

  Allows guided generation and pattern enforcement at both subword and character levels.

These schemes regulate the tradeoff between vocabulary size (compact codomain) and token granularity (semantic fidelity). Static, frequency-based codomain construction yields robust open-vocabulary coverage but is often resistant to adaptation (Mielke et al., 2021, Owodunni et al., 17 Jul 2025).

3. Adaptation, Learnability, and Domain-Specific Codomain Tokenization

Codomain tokenization is central to domain adaptation. Adaptive tokenization augments pretrained models' codomains to introduce domain-specific subword sequences (Sachidananda et al., 2021):

• Statistic-Based Selection: Compute conditional probabilities:

  $P_C(s) = \frac{C_s}{C_t}$

  Divergence scoring via pointwise KL:

  $R(s) = P_D(s) \log \frac{P_D(s)}{P_S(s)}$

  Top sequences are added to the codomain, with token embeddings initialized from constituent or projected vectors.

• Efficiency: Quantitative results show 97% of domain-adaptive pretraining performance at a fraction of compute cost; AT adaptations with 10k tokens add ~6% parameters and are 72× faster than corpus pretraining (Sachidananda et al., 2021).

Learnable tokenization modules (FLEXITOKENS (Owodunni et al., 17 Jul 2025)) allow codomain boundaries to be dynamically learned and updated:

• Boundary Predictor: Transformer + MLP predicts $b_t\in[0,1]$; hard Gumbel sigmoid reparameterizes this into binary boundary decision.
• Margin-Aware Loss: $L_{\mathrm{boundary}} = \max(k - B, 0)$, where $B$ is a lower-bound related to anchor rate and sample variance.
• Benefits: Dramatic reduction in over-fragmentation, improved downstream performance (up to 10%), effective balance of tokenization rates across morphologically diverse languages and domains.

4. Codomain Tokenization for Multimodal and Non-Textual Data

Discrete tokenization generalizes across modalities (Jia et al., 18 Feb 2025, Liu et al., 22 Mar 2025):

• Visual Tokenization (CODA (Liu et al., 22 Mar 2025)): Decouples compression and discretization. A pretrained continuous VAE encodes images; discretization is learned via attention-based and residual quantization:
  • Attention: $Q = \mathrm{rms\_norm}(F W_q)$, $K = \mathrm{rms\_norm}(C W_k)$, $A = \mathrm{softmax}((QK^\top)/\sqrt{d})$
  • Full codebook utilization (100%) and low reconstruction FID (0.43 at $8\times$ compression, 1.34 at $16\times$) with $6\times$ training budget reduction.
• Multimodal Tokenization: Encoders (CNNs, Transformers) map images, audio, etc. to latent vectors; quantization into multimodal codomains bridges representation for downstream multimodal LLMs.

Challenges:

• Codebook collapse, semantic misalignment, fidelity/compression trade-off, and differentiability approximation errors (STE).

5. Codomain Tokenization in Scientific and Mathematical Modeling

In operator learning for scientific computing, codomain tokenization governs the handling of multivariate output channels (Rahman et al., 19 Mar 2024):

• Function-Space Tokenization: Each channel or physical variable is a token, enabling representation of arbitrary variable sets ($w: D \to \mathbb{R}^d$).
• Transformer Extensions: Variable-specific positional encoders $e_j: D \to \mathbb{R}^{d_{en}}$, attention operators over function spaces, normalization over domain:

  $$o_i(x) = \mathrm{Softmax}\left(\frac{(q_i, k_j)}{T}\right)v_j(x),\quad (q_i, k_j) = \int_D q_i(x) k_j(x)\,dx$$

• Sample Efficiency: CoDA-NO achieves >36% L2 error reduction over baselines in few-shot transfer for complex multiphysics PDEs.

6. Impact on Model Expressivity, Reasoning, and Downstream Performance

Codomain tokenization dictates not only representational granularity but also computational and reasoning capability. The granularity and boundaries of tokens directly affect the ability of models to perform fine-grained tasks such as counting and stepwise reasoning (Zhang et al., 25 Oct 2024):

• Granularity Effect: For tasks like counting, character-level tokenization (fine codomain) allows accurate induction,

  $$a = \sum_{i=1}^n \mathbb{1}\{l_i = l_{\text{target}}\}$$

whereas BPE can obscure the necessary atomic units, leading to substantial performance drops (up to 40% improvement when switching to fine-grained tokenization).

• Reasoning Limits: Transformers without recurrent connections (TC$^0$ class) cannot realize the linear-depth update required for counting unless codomain tokenization preserves inductive granularity. Methods such as CoT do not circumvent this if tokens aggregate target items.
• Guided Generation: Finite-state transduction techniques (Cognetta et al., 21 Oct 2024) allow both character-level constraints and canonical tokenization preservation, important for maintaining inductive bias in controlled generation.

7. Open Challenges and Future Directions

• Adaptive/Dynamic Codomains: Design tokenizers that adjust granularity and codebook size per content and downstream task, hybrid tokenization for arithmetic/logical reasoning (Zhang et al., 25 Oct 2024, Jia et al., 18 Feb 2025).
• Learnable Tokenization: Gradient-based boundary predictors (FLEXITOKENS) enable semantic coherence and more equitable token allocation, reducing computational load and improving task performance (Owodunni et al., 17 Jul 2025).
• Cross-Modal Codebook Learning: Ensuring semantic alignment and codebook coverage in multimodal settings remains an active area (Jia et al., 18 Feb 2025, Liu et al., 22 Mar 2025).
• Efficient Architectures: Targeting lightweight, end-to-end training frameworks that jointly optimize tokenization and downstream modeling.
• Theoretical Insights and Guarantees: Leveraging finite-state frameworks (Cognetta et al., 21 Oct 2024) to ensure polynomial-time tokenization and compatibility with logic-level constraints.

---

Codomain tokenization is a foundational mechanism by which models translate raw data into tractable, semantically meaningful representations. Its design choices—spanning algorithmic, statistical, and learnable frameworks—have profound implications for model efficiency, expressivity, reasoning capacity, domain adaptation, and multimodal integration. Current research demonstrates diverse strategies adapted to task requirements (textual, visual, scientific) and highlights both the impact of granularity control and the necessity for dynamic, adaptive tokenization modules capable of bridging computational efficiency, semantic fidelity, and theoretical soundness.