---
title: 'TOFU: Token-Based Foundation Models'
url: https://www.emergentmind.com/topics/token-based-foundation-model-tofu
type: topic
---

# TOFU: Token-Based Foundation Models

Token-based Foundation Model (TOFU) denotes a token-centric conception of foundation models in which tokens serve as the primary interface between raw inputs, intermediate representations, downstream reasoning, and, in some formulations, system-level accounting of computation, latency, energy, and price [2606.24616]. In current literature, however, the term is context-dependent rather than uniform. It can refer to a token-interface architecture for decentralized multimodal systems that is described as “TOFU-like” rather than explicitly branded as TOFU [2505.07841], an explicit token-based foundation model for multi-modal knowledge graph reasoning [2602.15896], or, in machine unlearning, the TOFU benchmark—“Task of Fictitious Unlearning”—introduced by Maini et al. and used to evaluate token-level forgetting methods [2605.00364]. The common thread across these usages is the treatment of tokens as the operative unit through which models perceive, communicate, optimize, or forget.

## 1. Token-centric definition

In the broadest formulation, a TOFU perspective begins with a tokenizer
$$
\tau : \mathcal X \rightarrow V^{*},
$$
where raw input $x \in \mathcal X$ is mapped to a token sequence $\tau(x) = (t_1,\ldots,t_n)$ over a vocabulary $V$ [2606.24616]. Tokens are then not merely discrete symbols for transformer inference; they are the common unit linking information processing, computation, memory, latency, energy, and monetary cost.

This perspective distinguishes several token categories. Input tokens $T_I$ represent prompts and task inputs; context tokens $T_C$ represent conversation history or persistent memory; retrieval tokens $T_R$ represent externally retrieved documents; output tokens $T_O$ represent visible responses; and hidden reasoning tokens $T_H$ represent internal “thinking” or chain-of-thought activity. The total token footprint is
$$
T_{\mathrm{tot}} = T_I + T_C + T_R + T_O + T_H.
$$
A central claim of this framework is that providers must provision resources for $T_{\mathrm{tot}}$, even though end users usually observe only $T_I + T_O$ [2606.24616].

The same paper formalizes the engineering and economic consequences of token counts. For a transformer with $L$ layers and width $m$, FLOPs are approximated by
$$
F(T) \approx \kappa L m^{2} T,
$$
while KV-cache memory scales as $M(T) \propto LmT$, latency as $\ell_{\mathrm{tot}} = \bar\ell T$, and energy as $\mathcal E_{\mathrm{tok}}(T) = \bar e T$ [2606.24616]. Billing is likewise tokenized:
$$
C = p_I (T_I + T_C + T_R) + p_O T_O,
$$
or, in an extended form,
$$
C = p_I (T^I + T^C + T^R) + p_O T^O + p_H T^H.
$$
This makes tokens the practical accounting unit of model services.

The same framework also separates token expenditure from token value. Workflow quality is written as
$$
q_w = \phi_w\!\left(T_w,\{q_j\}_{j\in\mathrm{Pa}(w)}\right),
$$
and enterprise utility as $\mathcal V(T)=U(q(T))$. The marginal value of tokens at workflow $w$ is
$$
\frac{\partial \mathcal V}{\partial T_w} = \mu_w \frac{\partial \phi_w}{\partial T_w},
$$
so token value depends on local token efficiency and downstream propagation through the workflow graph, not on token volume alone [2606.24616]. This is the most general and explicit statement of TOFU as a systems-and-economics abstraction.

## 2. TOFU-like token interfaces in distributed multimodal systems

A concrete architectural instantiation appears in decentralized deployment of multimodal large models over resource-constrained wireless networks [2505.07841]. The setting is a multiuser wireless network in which edge devices produce text, audio, and vision data, while a base station with an edge server hosts the heavy multimodal large model backbone. Devices cannot host the full model; raw multimodal inputs and high-dimensional intermediate features are too expensive to transmit; and links operate under limited bandwidth, low or medium SNR, and strict latency.

The proposed solution is a token communication paradigm. Edge devices run modality encoders and token transmitters, while the base station runs token receivers and the foundation model. For a non-text modality $m$, a device encoder produces a long sequence $\mathbf E_m \in \mathbb R^{b\times \hat s_m \times d_m}$; the paper gives $\hat s_m = 3137$ for video with ViViT and $\hat s_m = 1214$ for audio with AST. Because transmitting all tokens is too costly, a sliding-window subsampling reduces the sequence to $\bar{\mathbf E}_m \in \mathbb R^{b\times s_m \times d_m}$ with $s_m \ll \hat s_m$, preserving temporal order. A projector then maps the result into the text embedding dimension $d_T$:
$$
\mathbf Y_m = \mathcal F_m(\bar{\mathbf E}_m) \in \mathbb R^{b\times s_m \times d_T}.
$$
These tokens are reshaped into complex-valued channel symbols, transmitted, and reconstructed at the base station through
$$
\tilde{\mathbf y}_m = h_m \mathbf y_m + \mathbf n_m,
$$
followed by a 2-layer MLP receiver head $\mathcal G_m$ that produces channel-robust aligned tokens $\mathbf Y'_m$ [2505.07841].

The foundation model is Qwen2.5-1.5B, fully deployed at the edge server. Devices host only pretrained modality-specific encoders—Qwen’s text encoder, AST for audio, and ViViT for vision—plus linear projectors into the shared token space. The model therefore sees a single unified sequence of text, audio, and visual tokens, all expressed in dimension $d_T$. The paper explicitly characterizes this setup as very close to a TOFU-style token interface: the model is treated as a token-based foundation model; all upstream sensing, encoding, and wireless modules are optimized to produce good tokens rather than reconstruct raw data; and tokens form the only interface between physical/link layers, multimodal encoders, and the foundation model [2505.07841].

Training proceeds in two stages. First, contrastive split fine-tuning aligns modalities in a shared space while preserving modality-specific semantics. Text is the anchor modality, and non-text pooled embeddings are aligned to paired text embeddings with an InfoNCE-style loss at temperature $\tau = 0.07$. A text reconstruction loss
$$
\mathcal L^{\text{Rec}}_T = \|\mathbf Y'_T - \mathbf Y_T\|^2
$$
stabilizes the text receiver. Second, task-oriented fine-tuning concatenates all received tokens and optimizes both the transceivers and a LoRA adapter on the frozen backbone for audio-visual question answering. Under this design, token generation, transmission, reception, and backbone adaptation are jointly tuned under channel noise and token budget constraints.

The reported results are explicitly token-budget dependent. Moderate compression with $s_A=64$ and $s_V=32$ can outperform uncompressed $s_A=s_V=128$, whereas extreme compression with $s_A=s_V=16$ causes information loss [2505.07841]. Relative to an unaligned model with long tokens, the cross-modal aligned model yields $+7.3\%$ test accuracy at SNR $=0$ dB and up to $+13.7\%$ at SNR $=6$ dB. It also reaches $47.9\%$ accuracy on MUSIC-AVQA, surpassing OneLLM-7B at $47.6\%$ and ChatBridge-13B at $43.0\%$. In this usage, TOFU is not a benchmark but a token-only architectural boundary for multimodal deployment.

## 3. Explicit TOFU in multi-modal knowledge graph reasoning

A more literal use of the term appears in a knowledge graph foundation model for multi-modal knowledge graph reasoning, where TOFU is the model’s name and stands for a token-based foundation model for MMKGR [2602.15896]. The underlying object is a multi-modal knowledge graph
$$
\mathcal{KG} = (\mathcal E,\mathcal R,\mathcal T,\mathcal M),
$$
with entities, relations, structural triples, and multi-modal contents comprising visual and textual information.

The model is designed for transductive, inductive, and fully-inductive settings. In transductive MMKGR, train and test share the same entities and relations. In inductive MMKGR, new entities appear only at inference time. In fully-inductive MMKGR, both entities and relations can be unseen at test time [2602.15896]. The motivation is that standard MMKGR systems usually learn dataset-specific entity and relation embeddings and therefore do not transfer well across KGs, while prior KG foundation models emphasize structure and underuse text and image signals.

TOFU addresses this with modality-specific discrete tokens. Textual content is tokenized with a BERT tokenizer and uses frozen token embeddings of dimension $768$, projected into a common model dimension. Visual content is tokenized with a pre-trained vector-quantized image tokenizer such as BEiT with VQ-VAE, which discretizes images into codebook vectors; the paper notes that BEiT produces a $14\times14$ grid of visual tokens, though TOFU typically retains only a subset such as 8 tokens for efficiency. Structural information is converted into structural tokens through relative position tuples $[d(e,h), d(e,t)]$ computed from sampled subgraphs around a query triple [2602.15896].

Architecture is hierarchical. A Structural Encoder, implemented as a GNN over sampled subgraphs, processes structural tokens. A Multi-modal Encoder, implemented as a Transformer over concatenated text and image tokens with a learnable $[\mathrm{ENT}]$ readout token, produces consolidated multi-modal features. A Gated Fusion module combines structural and multi-modal outputs into a unified entity representation. Finally, a second GNN performs Global Propagation with a Mixture-of-Messages mechanism that combines TransE-, DistMult-, and RotatE-style message functions with relation-guided gating and temperature [2602.15896]. This removes dependence on entity- and relation-specific embedding tables and makes all learnable components shared across KGs.

Empirical results are reported on 17 transductive, inductive, and fully-inductive MMKGs. Averaged across these datasets, zero-shot MRR is $45.93$ for TOFU, compared with $44.28$ for KG-ICL, $38.71$ for MOTIF, and $38.24$ for ULTRA; zero-shot Hit@10 is $61.67$ for TOFU [2602.15896]. Fine-tuned MRR is $47.41$, again exceeding the cited KG foundation model baselines. On classic transductive MMKGR benchmarks DB15K, MKG-W, and MKG-Y, TOFU reaches average MRR $41.04$, compared with $38.57$ for the best listed baseline, LMBKGC. In this usage, TOFU is an explicit multimodal foundation model whose tokenization scheme is intended to be fine-grained and transferable.

## 4. TOFU as the “Task of Fictitious Unlearning” benchmark

In machine unlearning, TOFU usually does not denote a foundation model architecture at all. It denotes the Tofu benchmark, or “Task of Fictitious Unlearning,” introduced by Maini et al. and used as a controlled testbed for factual forgetting in language models [2605.00364]. The benchmark consists of 200 fictitious author profiles, each associated with 20 question-answer pairs, for a total of 4,000 QA pairs. Models are fine-tuned on this synthetic corpus to create a target model that memorizes the fictitious facts.

The dominant task variant in one token-level unlearning study is forget10, which requires unlearning 10% of the dataset: 400 QA pairs from 20 authors form the forget set, and the remaining 3,600 QA pairs form the retain set [2605.00364]. The same benchmark is also used in forget01 and forget05 variants in later work [2606.06320]. Because the data are fictitious, the benchmark isolates unlearning behavior from real-person privacy or copyright concerns while keeping the forget/retain split explicit and controlled.

Evaluation protocols differ across papers. One protocol reports Forget Quality (FQ), Model Utility (MU), and exact-match retain and unlearn scores under the OpenUnlearning setting [2605.00364]. Another uses judge-based metrics robust to paraphrase and in-context relearning, defining Forget Quality $\FQ$, Retain Degradation $\RD$, and Unlearning Quality $\UQ$, together with extraction strength and general capability probes such as MMLU, repetitiveness, and win rate [2606.06320]. A third protocol, used for UniErase, reports Forget Efficacy (FE), Retain Efficacy (RE), Retain Average (RA), and Balance, alongside MMLU, TriviaQA, GSM8k, and HumanEval, as well as “Idk” ratio and output length [2505.15674]. This suggests that TOFU has become a benchmark family with multiple evaluation conventions rather than a single fixed scorecard.

The benchmark’s structure is particularly important for token-level analysis. The unlearning papers repeatedly emphasize that only a minority of tokens in a forget sequence encode the knowledge to be removed, while many other tokens are scaffolding or template language [2605.00364]. In that sense, TOFU is not merely a dataset; it is a controlled environment for asking whether forgetting should be localized at the token level rather than imposed uniformly on entire sequences.

## 5. Token-level unlearning methods evaluated on TOFU

Three lines of work illustrate how the TOFU benchmark has driven token-level unlearning research. The first is TokenUnlearn, which assigns each token an importance score by combining a masking-based knowledge signal and an entropy-based signal. Given normalized log-probability shifts and entropy terms, the score is
$$
\phi_i = \alpha \cdot \bar{\Delta}^{\text{unlearn}}_i + (1-\alpha)\cdot \bar H_i,
$$
with default $\alpha=0.7$. The method then either hard-selects the top $r$ fraction of tokens or soft-weights all tokens with a softmax, using default $r=0.2$ and temperature $\tau=0.5$ [2605.00364]. On TOFU, hard token selection yields consistent gains over sequence-level baselines. For Llama-2-7B, T-WGA improves retain from $0.6418$ to $0.6791$ and reduces unlearn from $0.1255$ to $0.0949$ relative to sequence-level WGA; for Qwen-3-8B, T-WGA improves retain from $0.5761$ to $0.6857$ and reduces unlearn from $0.1072$ to $0.0723$ [2605.00364].

A second line is Alternating Token-Weighted Unlearning (ATWU), which replaces heuristic token importance with a learned forget-specificity score derived from retain conflict. The method jointly optimizes model parameters and token weights, then relaxes the discrete selector with entropy and budget penalties and parameterizes token scores as a linear scorer over hidden states:
$$
g_{\mathbf w}(\tok{x}{t}) = \sigma\big(\langle \mathbf w,\mathbf h_{\theta}(\tok{x}{t})\rangle\big).
$$
The practical objective combines retain loss, a saturated token-weighted forget loss, and regularizers for entropy and score budget [2606.06320]. On TOFU with Llama-3.1-8B-Instruct, forget10, ATWU reports $\FQ=95.2$, $\RD=3.5$, and $\UQ=91.7$, while preserving MMLU at $66.5$ versus $66.6$ for the original model. Because TOFU also provides ground-truth forget-specific spans from Zhou et al. (2026), the same work measures token-detection AUROC and reports $75\pm9$ for ATWU, higher than heuristic or auxiliary-model baselines [2606.06320].

A third line is UniErase, which introduces a learnable parametric suffix token $[\mathrm{UNL}]$ as a universal erasure primitive. In Phase I, only the new token embedding and unembedding are optimized so that a forget query followed by $[\mathrm{UNL}]$ yields an “I don’t know” style response. In Phase II, a lightweight model edit modifies selected MLP down-projection matrices so that the model automatically emits $[\mathrm{UNL}]$ as the first answer token for forget queries [2505.15674]. The paper states that UniErase modifies only around $3.66\%$ of model parameters. On TOFU-injected Llama-3.1-8B-Instruct, Forget10 batch unlearning, it reports FE $79.43$, RE $82.32$, and Balance $75.38$, and it states that UniErase outperforms the previous forgetting SOTA baseline by around $4.01$ times for model ability while also surpassing the previous retaining SOTA by $35.96\%$ for unlearning efficacy [2505.15674].

Taken together, these studies make TOFU a benchmark for token-localized forgetting rather than for architectural token interfaces. They also show that the word “token-based” in current TOFU literature may refer either to token-based model design or to token-based intervention in unlearning objectives. A plausible implication is that the benchmark’s synthetic, templated QA format makes it especially sensitive to methods that can isolate answer-bearing spans from generic sequence structure.

## 6. Terminological ambiguity and related usages

The literature uses closely related labels for distinct objects, and this is a persistent source of confusion.

| Usage | Domain | Meaning |
|---|---|---|
| TOFU-like token interface | Wireless multimodal systems | Tokens are the only communicated object between devices and the foundation model |
| TOFU | MMKGR | Token-based foundation model with structural, textual, and visual tokens |
| TOFU benchmark | Machine unlearning | “Task of Fictitious Unlearning” for forgetting fictitious author facts |
| ToFu | Vision transformers | “Token Fusion,” a token-reduction method combining pruning and merging |

The vision usage is particularly distinct. “Token Fusion” (ToFu) is a method for Vision Transformers that combines token pruning and token merging within a depth-aware schedule and introduces MLERP, a norm-preserving multi-token variant of SLERP [2312.01026]. It is motivated by the observation that early ViT layers are sensitive to interpolations while later layers are more approximately linear, so ToFu prunes in early layers and merges in later ones. On ImageNet, ToFu MLERP improves over ToMe at the same FLOPs; for ViT-B with reduction ratio $r=16$, the paper reports $80.70\%$ Top-1 for ToFu MLERP versus $78.88\%$ for ToMe at $8.78$ GFLOPs [2312.01026]. Despite the shared acronym, this is an efficiency method for token reduction in vision backbones, not a token-based foundation model in the architectural or benchmark sense.

The resulting nomenclature is therefore plural rather than singular. In systems papers, TOFU names a token interface or token middleware; in knowledge graph research, it names a specific transferable multimodal foundation model; in unlearning, it almost always names a benchmark; and in vision, ToFu names Token Fusion. Any technical use of the term requires immediate contextual disambiguation.

Source: https://www.emergentmind.com/topics/token-based-foundation-model-tofu