TOFU: Token-Based Foundation Models

• TOFU is a token-centric framework that treats tokens as the core unit for input processing, computation, and resource billing in foundation models.
• It underpins decentralized multimodal systems by converting text, audio, and visual data into unified tokens, optimizing performance under bandwidth and latency constraints.
• TOFU also serves as a benchmark in machine unlearning and knowledge graph reasoning, providing measurable metrics for token-level forgetting and efficiency.

Token-based Foundation Model (TOFU) denotes a token-centric conception of foundation models in which tokens serve as the primary interface between raw inputs, intermediate representations, downstream reasoning, and, in some formulations, system-level accounting of computation, latency, energy, and price (Zhu, 10 Jun 2026). In current literature, however, the term is context-dependent rather than uniform. It can refer to a token-interface architecture for decentralized multimodal systems that is described as “TOFU-like” rather than explicitly branded as TOFU (Zhang et al., 6 May 2025), an explicit token-based foundation model for multi-modal knowledge graph reasoning (Zhang et al., 11 Feb 2026), or, in machine unlearning, the TOFU benchmark—“Task of Fictitious Unlearning”—introduced by Maini et al. and used to evaluate token-level forgetting methods (Wu et al., 1 May 2026). The common thread across these usages is the treatment of tokens as the operative unit through which models perceive, communicate, optimize, or forget.

1. Token-centric definition

In the broadest formulation, a TOFU perspective begins with a tokenizer

$\tau : \mathcal X \rightarrow V^{*},$

where raw input $x \in \mathcal X$ is mapped to a token sequence $\tau(x) = (t_1,\ldots,t_n)$ over a vocabulary $V$ (Zhu, 10 Jun 2026). Tokens are then not merely discrete symbols for transformer inference; they are the common unit linking information processing, computation, memory, latency, energy, and monetary cost.

This perspective distinguishes several token categories. Input tokens $T_I$ represent prompts and task inputs; context tokens $T_C$ represent conversation history or persistent memory; retrieval tokens $T_R$ represent externally retrieved documents; output tokens $T_O$ represent visible responses; and hidden reasoning tokens $T_H$ represent internal “thinking” or chain-of-thought activity. The total token footprint is

$T_{\mathrm{tot}} = T_I + T_C + T_R + T_O + T_H.$

A central claim of this framework is that providers must provision resources for $x \in \mathcal X$0, even though end users usually observe only $x \in \mathcal X$1 (Zhu, 10 Jun 2026).

The same paper formalizes the engineering and economic consequences of token counts. For a transformer with $x \in \mathcal X$2 layers and width $x \in \mathcal X$3, FLOPs are approximated by

$x \in \mathcal X$4

while KV-cache memory scales as $x \in \mathcal X$5, latency as $x \in \mathcal X$6, and energy as $x \in \mathcal X$7 (Zhu, 10 Jun 2026). Billing is likewise tokenized:

$x \in \mathcal X$8

or, in an extended form,

$x \in \mathcal X$9

This makes tokens the practical accounting unit of model services.

The same framework also separates token expenditure from token value. Workflow quality is written as

$\tau(x) = (t_1,\ldots,t_n)$0

and enterprise utility as $\tau(x) = (t_1,\ldots,t_n)$1. The marginal value of tokens at workflow $\tau(x) = (t_1,\ldots,t_n)$2 is

$\tau(x) = (t_1,\ldots,t_n)$3

so token value depends on local token efficiency and downstream propagation through the workflow graph, not on token volume alone (Zhu, 10 Jun 2026). This is the most general and explicit statement of TOFU as a systems-and-economics abstraction.

2. TOFU-like token interfaces in distributed multimodal systems

A concrete architectural instantiation appears in decentralized deployment of multimodal large models over resource-constrained wireless networks (Zhang et al., 6 May 2025). The setting is a multiuser wireless network in which edge devices produce text, audio, and vision data, while a base station with an edge server hosts the heavy multimodal large model backbone. Devices cannot host the full model; raw multimodal inputs and high-dimensional intermediate features are too expensive to transmit; and links operate under limited bandwidth, low or medium SNR, and strict latency.

The proposed solution is a token communication paradigm. Edge devices run modality encoders and token transmitters, while the base station runs token receivers and the foundation model. For a non-text modality $\tau(x) = (t_1,\ldots,t_n)$4, a device encoder produces a long sequence $\tau(x) = (t_1,\ldots,t_n)$5; the paper gives $\tau(x) = (t_1,\ldots,t_n)$6 for video with ViViT and $\tau(x) = (t_1,\ldots,t_n)$7 for audio with AST. Because transmitting all tokens is too costly, a sliding-window subsampling reduces the sequence to $\tau(x) = (t_1,\ldots,t_n)$8 with $\tau(x) = (t_1,\ldots,t_n)$9, preserving temporal order. A projector then maps the result into the text embedding dimension $V$0:

$V$1

These tokens are reshaped into complex-valued channel symbols, transmitted, and reconstructed at the base station through

$V$2

followed by a 2-layer MLP receiver head $V$3 that produces channel-robust aligned tokens $V$4 (Zhang et al., 6 May 2025).

The foundation model is Qwen2.5-1.5B, fully deployed at the edge server. Devices host only pretrained modality-specific encoders—Qwen’s text encoder, AST for audio, and ViViT for vision—plus linear projectors into the shared token space. The model therefore sees a single unified sequence of text, audio, and visual tokens, all expressed in dimension $V$5. The paper explicitly characterizes this setup as very close to a TOFU-style token interface: the model is treated as a token-based foundation model; all upstream sensing, encoding, and wireless modules are optimized to produce good tokens rather than reconstruct raw data; and tokens form the only interface between physical/link layers, multimodal encoders, and the foundation model (Zhang et al., 6 May 2025).

Training proceeds in two stages. First, contrastive split fine-tuning aligns modalities in a shared space while preserving modality-specific semantics. Text is the anchor modality, and non-text pooled embeddings are aligned to paired text embeddings with an InfoNCE-style loss at temperature $V$6. A text reconstruction loss

$V$7

stabilizes the text receiver. Second, task-oriented fine-tuning concatenates all received tokens and optimizes both the transceivers and a LoRA adapter on the frozen backbone for audio-visual question answering. Under this design, token generation, transmission, reception, and backbone adaptation are jointly tuned under channel noise and token budget constraints.

The reported results are explicitly token-budget dependent. Moderate compression with $V$8 and $V$9 can outperform uncompressed $T_I$0, whereas extreme compression with $T_I$1 causes information loss (Zhang et al., 6 May 2025). Relative to an unaligned model with long tokens, the cross-modal aligned model yields $T_I$2 test accuracy at SNR $T_I$3 dB and up to $T_I$4 at SNR $T_I$5 dB. It also reaches $T_I$6 accuracy on MUSIC-AVQA, surpassing OneLLM-7B at $T_I$7 and ChatBridge-13B at $T_I$8. In this usage, TOFU is not a benchmark but a token-only architectural boundary for multimodal deployment.

3. Explicit TOFU in multi-modal knowledge graph reasoning

A more literal use of the term appears in a knowledge graph foundation model for multi-modal knowledge graph reasoning, where TOFU is the model’s name and stands for a token-based foundation model for MMKGR (Zhang et al., 11 Feb 2026). The underlying object is a multi-modal knowledge graph

$T_I$9

with entities, relations, structural triples, and multi-modal contents comprising visual and textual information.

The model is designed for transductive, inductive, and fully-inductive settings. In transductive MMKGR, train and test share the same entities and relations. In inductive MMKGR, new entities appear only at inference time. In fully-inductive MMKGR, both entities and relations can be unseen at test time (Zhang et al., 11 Feb 2026). The motivation is that standard MMKGR systems usually learn dataset-specific entity and relation embeddings and therefore do not transfer well across KGs, while prior KG foundation models emphasize structure and underuse text and image signals.

TOFU addresses this with modality-specific discrete tokens. Textual content is tokenized with a BERT tokenizer and uses frozen token embeddings of dimension $T_C$0, projected into a common model dimension. Visual content is tokenized with a pre-trained vector-quantized image tokenizer such as BEiT with VQ-VAE, which discretizes images into codebook vectors; the paper notes that BEiT produces a $T_C$1 grid of visual tokens, though TOFU typically retains only a subset such as 8 tokens for efficiency. Structural information is converted into structural tokens through relative position tuples $T_C$2 computed from sampled subgraphs around a query triple (Zhang et al., 11 Feb 2026).

Architecture is hierarchical. A Structural Encoder, implemented as a GNN over sampled subgraphs, processes structural tokens. A Multi-modal Encoder, implemented as a Transformer over concatenated text and image tokens with a learnable $T_C$3 readout token, produces consolidated multi-modal features. A Gated Fusion module combines structural and multi-modal outputs into a unified entity representation. Finally, a second GNN performs Global Propagation with a Mixture-of-Messages mechanism that combines TransE-, DistMult-, and RotatE-style message functions with relation-guided gating and temperature (Zhang et al., 11 Feb 2026). This removes dependence on entity- and relation-specific embedding tables and makes all learnable components shared across KGs.

Empirical results are reported on 17 transductive, inductive, and fully-inductive MMKGs. Averaged across these datasets, zero-shot MRR is $T_C$4 for TOFU, compared with $T_C$5 for KG-ICL, $T_C$6 for MOTIF, and $T_C$7 for ULTRA; zero-shot Hit@10 is $T_C$8 for TOFU (Zhang et al., 11 Feb 2026). Fine-tuned MRR is $T_C$9, again exceeding the cited KG foundation model baselines. On classic transductive MMKGR benchmarks DB15K, MKG-W, and MKG-Y, TOFU reaches average MRR $T_R$0, compared with $T_R$1 for the best listed baseline, LMBKGC. In this usage, TOFU is an explicit multimodal foundation model whose tokenization scheme is intended to be fine-grained and transferable.

4. TOFU as the “Task of Fictitious Unlearning” benchmark

In machine unlearning, TOFU usually does not denote a foundation model architecture at all. It denotes the Tofu benchmark, or “Task of Fictitious Unlearning,” introduced by Maini et al. and used as a controlled testbed for factual forgetting in LLMs (Wu et al., 1 May 2026). The benchmark consists of 200 fictitious author profiles, each associated with 20 question-answer pairs, for a total of 4,000 QA pairs. Models are fine-tuned on this synthetic corpus to create a target model that memorizes the fictitious facts.

The dominant task variant in one token-level unlearning study is forget10, which requires unlearning 10% of the dataset: 400 QA pairs from 20 authors form the forget set, and the remaining 3,600 QA pairs form the retain set (Wu et al., 1 May 2026). The same benchmark is also used in forget01 and forget05 variants in later work (Yüce et al., 4 Jun 2026). Because the data are fictitious, the benchmark isolates unlearning behavior from real-person privacy or copyright concerns while keeping the forget/retain split explicit and controlled.

Evaluation protocols differ across papers. One protocol reports Forget Quality (FQ), Model Utility (MU), and exact-match retain and unlearn scores under the OpenUnlearning setting (Wu et al., 1 May 2026). Another uses judge-based metrics robust to paraphrase and in-context relearning, defining Forget Quality $T_R$2, Retain Degradation $T_R$3, and Unlearning Quality $T_R$4, together with extraction strength and general capability probes such as MMLU, repetitiveness, and win rate (Yüce et al., 4 Jun 2026). A third protocol, used for UniErase, reports Forget Efficacy (FE), Retain Efficacy (RE), Retain Average (RA), and Balance, alongside MMLU, TriviaQA, GSM8k, and HumanEval, as well as “Idk” ratio and output length (Yu et al., 21 May 2025). This suggests that TOFU has become a benchmark family with multiple evaluation conventions rather than a single fixed scorecard.

The benchmark’s structure is particularly important for token-level analysis. The unlearning papers repeatedly emphasize that only a minority of tokens in a forget sequence encode the knowledge to be removed, while many other tokens are scaffolding or template language (Wu et al., 1 May 2026). In that sense, TOFU is not merely a dataset; it is a controlled environment for asking whether forgetting should be localized at the token level rather than imposed uniformly on entire sequences.

5. Token-level unlearning methods evaluated on TOFU

Three lines of work illustrate how the TOFU benchmark has driven token-level unlearning research. The first is TokenUnlearn, which assigns each token an importance score by combining a masking-based knowledge signal and an entropy-based signal. Given normalized log-probability shifts and entropy terms, the score is

$T_R$5

with default $T_R$6. The method then either hard-selects the top $T_R$7 fraction of tokens or soft-weights all tokens with a softmax, using default $T_R$8 and temperature $T_R$9 (Wu et al., 1 May 2026). On TOFU, hard token selection yields consistent gains over sequence-level baselines. For Llama-2-7B, T-WGA improves retain from $T_O$0 to $T_O$1 and reduces unlearn from $T_O$2 to $T_O$3 relative to sequence-level WGA; for Qwen-3-8B, T-WGA improves retain from $T_O$4 to $T_O$5 and reduces unlearn from $T_O$6 to $T_O$7 (Wu et al., 1 May 2026).

A second line is Alternating Token-Weighted Unlearning (ATWU), which replaces heuristic token importance with a learned forget-specificity score derived from retain conflict. The method jointly optimizes model parameters and token weights, then relaxes the discrete selector with entropy and budget penalties and parameterizes token scores as a linear scorer over hidden states:

$T_O$8

The practical objective combines retain loss, a saturated token-weighted forget loss, and regularizers for entropy and score budget (Yüce et al., 4 Jun 2026). On TOFU with Llama-3.1-8B-Instruct, forget10, ATWU reports $T_O$9, $T_H$0, and $T_H$1, while preserving MMLU at $T_H$2 versus $T_H$3 for the original model. Because TOFU also provides ground-truth forget-specific spans from Zhou et al. (2026), the same work measures token-detection AUROC and reports $T_H$4 for ATWU, higher than heuristic or auxiliary-model baselines (Yüce et al., 4 Jun 2026).

A third line is UniErase, which introduces a learnable parametric suffix token $T_H$5 as a universal erasure primitive. In Phase I, only the new token embedding and unembedding are optimized so that a forget query followed by $T_H$6 yields an “I don’t know” style response. In Phase II, a lightweight model edit modifies selected MLP down-projection matrices so that the model automatically emits $T_H$7 as the first answer token for forget queries (Yu et al., 21 May 2025). The paper states that UniErase modifies only around $T_H$8 of model parameters. On TOFU-injected Llama-3.1-8B-Instruct, Forget10 batch unlearning, it reports FE $T_H$9, RE $T_{\mathrm{tot}} = T_I + T_C + T_R + T_O + T_H.$0, and Balance $T_{\mathrm{tot}} = T_I + T_C + T_R + T_O + T_H.$1, and it states that UniErase outperforms the previous forgetting SOTA baseline by around $T_{\mathrm{tot}} = T_I + T_C + T_R + T_O + T_H.$2 times for model ability while also surpassing the previous retaining SOTA by $T_{\mathrm{tot}} = T_I + T_C + T_R + T_O + T_H.$3 for unlearning efficacy (Yu et al., 21 May 2025).

Taken together, these studies make TOFU a benchmark for token-localized forgetting rather than for architectural token interfaces. They also show that the word “token-based” in current TOFU literature may refer either to token-based model design or to token-based intervention in unlearning objectives. A plausible implication is that the benchmark’s synthetic, templated QA format makes it especially sensitive to methods that can isolate answer-bearing spans from generic sequence structure.

6. Terminological ambiguity and related usages

The literature uses closely related labels for distinct objects, and this is a persistent source of confusion.

Usage	Domain	Meaning
TOFU-like token interface	Wireless multimodal systems	Tokens are the only communicated object between devices and the foundation model
TOFU	MMKGR	Token-based foundation model with structural, textual, and visual tokens
TOFU benchmark	Machine unlearning	“Task of Fictitious Unlearning” for forgetting fictitious author facts
ToFu	Vision transformers	“Token Fusion,” a token-reduction method combining pruning and merging

The vision usage is particularly distinct. “Token Fusion” (ToFu) is a method for Vision Transformers that combines token pruning and token merging within a depth-aware schedule and introduces MLERP, a norm-preserving multi-token variant of SLERP (Kim et al., 2023). It is motivated by the observation that early ViT layers are sensitive to interpolations while later layers are more approximately linear, so ToFu prunes in early layers and merges in later ones. On ImageNet, ToFu MLERP improves over ToMe at the same FLOPs; for ViT-B with reduction ratio $T_{\mathrm{tot}} = T_I + T_C + T_R + T_O + T_H.$4, the paper reports $T_{\mathrm{tot}} = T_I + T_C + T_R + T_O + T_H.$5 Top-1 for ToFu MLERP versus $T_{\mathrm{tot}} = T_I + T_C + T_R + T_O + T_H.$6 for ToMe at $T_{\mathrm{tot}} = T_I + T_C + T_R + T_O + T_H.$7 GFLOPs (Kim et al., 2023). Despite the shared acronym, this is an efficiency method for token reduction in vision backbones, not a token-based foundation model in the architectural or benchmark sense.

The resulting nomenclature is therefore plural rather than singular. In systems papers, TOFU names a token interface or token middleware; in knowledge graph research, it names a specific transferable multimodal foundation model; in unlearning, it almost always names a benchmark; and in vision, ToFu names Token Fusion. Any technical use of the term requires immediate contextual disambiguation.