MedTPE: Medical Token-Pair Encoding

• Medical Token-Pair Encoding (MedTPE) is a lossless compression technique that merges frequently co-occurring medical token pairs to streamline processing of long EHR sequences.
• It employs dependency-aware replacement and self-supervised fine-tuning to preserve vocabulary size and model parameters while enhancing prediction performance.
• MedTPE demonstrably reduces inference time and mitigates over-segmentation, achieving latency reductions of up to 62.5% in clinical tasks.

Medical Token-Pair Encoding (MedTPE) is a tokenisation-driven, lossless prompt compression method for using LLMs on electronic health records (EHRs). It is formulated as a layered method that extends standard tokenisation for EHR sequences by merging frequently co-occurring medical token pairs into composite tokens, while preserving vocabulary size, tokenisation complexity, and symbolic reversibility. In the reported formulation, MedTPE operates after a base tokenizer, replaces rare low-utility original tokens with composite tokens under a dependency-aware replacement strategy, and fine-tunes only the embeddings of the newly introduced tokens through self-supervised learning. This design targets the characteristic length and density of longitudinal clinical records, where standard subword tokenisers over-segment medical terms, templated measurements, and units, producing excessively long sequences that increase latency and can impair downstream prediction (Zhu et al., 12 May 2026).

1. Definition and problem setting

MedTPE is defined in the context of EHR-to-text pipelines in which diagnoses, medications, laboratory values, vital signs, procedures, and related events are converted into natural-language descriptions and then concatenated with a task prompt for inference by a general LLM. The central technical observation is that real EHRs are extremely long under standard tokenisation: in MIMIC-IV and EHRSHOT, average EHR token counts with standard tokenisers are reported to be on the order of 10–200k tokens per patient, with examples ranging from 13k to 190k tokens depending on model and dataset (Zhu et al., 12 May 2026). Standard subword tokenisers such as BPE, WordPiece, and SentencePiece are trained on general text and tend to over-segment clinical terms, repeated measurement templates, units, and boilerplate phrases. The cited example “Spirometry” $\rightarrow$ Spi, rom, etry is presented as representative of this failure mode (Zhu et al., 12 May 2026).

The immediate consequences are longer input sequences, quadratic or near-quadratic compute in sequence length for transformer inference, higher latency and cost, and, in some settings, degraded performance because models are “lost in the middle” and cannot effectively use long context (Zhu et al., 12 May 2026). MedTPE is positioned against three broad alternatives. New medical vocabularies that retrain the tokenizer and LLM can reduce over-segmentation but require retraining or extensive continual pretraining and may degrade general capabilities. Removal-based compression methods such as LLMLingua and CPC are lossy because they score and drop tokens or sentences. Merge-based compression approaches such as AdaptiVocab, LTSC meta-tokens, and ZeTT may require larger vocabularies, additional modules, supervised alignment, or extra inference-time passes (Zhu et al., 12 May 2026).

Within this framing, MedTPE’s stated goal is lossless compression of EHR text at the tokenizer level with no increase in vocabulary size or model parameter count, no change in tokenisation complexity, and only a tiny fraction of parameters fine-tuned in a label-free manner. A plausible implication is that MedTPE treats prompt compression as a vocabulary reallocation problem rather than as an architectural modification or a selection problem over already-tokenised text.

2. Formalism and token-pair encoding procedure

For each patient $i$, the EHR is represented as an event stream

$$\mathbf{E}^{(i)} = \{e^{(i)}_j\}_{j=1}^{T^{(i)}}, \qquad e^{(i)}_j = (c^{(i)}_j, o^{(i)}_j, t^{(i)}_j),$$

where $c^{(i)}_j$ is a clinical concept, $o^{(i)}_j$ is an observation value, and $t^{(i)}_j$ is a timestamp. Each event is mapped to natural text,

$s^{(i)}_j = \phi(e^{(i)}_j),$

and the full patient history is formed by concatenation,

$$S^{(i)} = s^{(i)}_1 \oplus s^{(i)}_2 \oplus \dots \oplus s^{(i)}_{T^{(i)}}.$$

A base tokenizer $\tau$ then yields a sequence

$$\mathcal{X}^{(i)} = \{x^{(i)}_n\}_{n=1}^{L^{(i)}}, \qquad x^{(i)}_n \in \mathcal{V},$$

to which a prompt $i$0 may be appended, producing $i$1 (Zhu et al., 12 May 2026).

MedTPE is a layer on top of this base tokenizer. It first mines frequent contiguous token $i$2-grams from a large medical corpus tokenised with $i$3:

$i$4

For each candidate composite token

$i$5

the procedure computes its raw frequency $i$6 and its original token length $i$7 (Zhu et al., 12 May 2026). In practice, the reported method primarily emphasizes bigrams.

Each candidate is associated with a left-to-right merge path,

$i$8

and all merge paths are concatenated, sorted by empirical frequency, and deduplicated into a global TPE merge table $i$9 (Zhu et al., 12 May 2026). Encoding is then two-stage: first apply the base tokenizer $$\mathbf{E}^{(i)} = \{e^{(i)}_j\}_{j=1}^{T^{(i)}}, \qquad e^{(i)}_j = (c^{(i)}_j, o^{(i)}_j, t^{(i)}_j),$$0, then apply TPE merges from $$\mathbf{E}^{(i)} = \{e^{(i)}_j\}_{j=1}^{T^{(i)}}, \qquad e^{(i)}_j = (c^{(i)}_j, o^{(i)}_j, t^{(i)}_j),$$1 over the token sequence in a single left-to-right pass. At each position, the encoder greedily takes the longest span whose merge path is present in the merge table and replaces it by the composite token. Because merges are implemented as table lookups in a single pass, the stated tokenisation complexity remains $$\mathbf{E}^{(i)} = \{e^{(i)}_j\}_{j=1}^{T^{(i)}}, \qquad e^{(i)}_j = (c^{(i)}_j, o^{(i)}_j, t^{(i)}_j),$$2 (Zhu et al., 12 May 2026).

The method is described as lossless in the sense of symbolic reversibility:

$$\mathbf{E}^{(i)} = \{e^{(i)}_j\}_{j=1}^{T^{(i)}}, \qquad e^{(i)}_j = (c^{(i)}_j, o^{(i)}_j, t^{(i)}_j),$$3

Because TPE tokens are defined as concatenations of original token strings, no information is discarded at the text level; the sequence is represented with fewer discrete symbols rather than by deleting content (Zhu et al., 12 May 2026). This point is central to the method’s positioning. A common misconception is to equate all prompt compression with removal or summarisation. MedTPE explicitly rejects that paradigm: it compresses by re-encoding frequent medical fragments, not by omitting events.

3. Dependency-aware replacement and self-supervised fine-tuning

A defining feature of MedTPE is that it does not simply enlarge the vocabulary. Instead, it preserves vocabulary size by replacing rare, low-utility original tokens with composite tokens while ensuring that all dependencies required to reproduce the original tokenisation remain available. Candidate insertions are scored by a length-aware frequency criterion,

$$\mathbf{E}^{(i)} = \{e^{(i)}_j\}_{j=1}^{T^{(i)}}, \qquad e^{(i)}_j = (c^{(i)}_j, o^{(i)}_j, t^{(i)}_j),$$4

which favors both frequent and longer merges. The top-$$\mathbf{E}^{(i)} = \{e^{(i)}_j\}_{j=1}^{T^{(i)}}, \qquad e^{(i)}_j = (c^{(i)}_j, o^{(i)}_j, t^{(i)}_j),$$5 candidates are selected as the insertion set $$\mathbf{E}^{(i)} = \{e^{(i)}_j\}_{j=1}^{T^{(i)}}, \qquad e^{(i)}_j = (c^{(i)}_j, o^{(i)}_j, t^{(i)}_j),$$6 (Zhu et al., 12 May 2026).

To make replacement safe, MedTPE defines a protected dependency set $$\mathbf{E}^{(i)} = \{e^{(i)}_j\}_{j=1}^{T^{(i)}}, \qquad e^{(i)}_j = (c^{(i)}_j, o^{(i)}_j, t^{(i)}_j),$$7 consisting of all tokens required by the merge paths of the selected composite tokens, including recursive sub-tokens. Unprotected tokens form $$\mathbf{E}^{(i)} = \{e^{(i)}_j\}_{j=1}^{T^{(i)}}, \qquad e^{(i)}_j = (c^{(i)}_j, o^{(i)}_j, t^{(i)}_j),$$8, from which the $$\mathbf{E}^{(i)} = \{e^{(i)}_j\}_{j=1}^{T^{(i)}}, \qquad e^{(i)}_j = (c^{(i)}_j, o^{(i)}_j, t^{(i)}_j),$$9 least frequent tokens are evicted to produce an eviction set $c^{(i)}_j$0. The resulting vocabulary is

$c^{(i)}_j$1

When an evicted token is encountered at inference time, its byte-level or lower-level sub-token representation is used, ensuring that any input string remains tokenisable (Zhu et al., 12 May 2026). This dependency-aware replacement strategy is the mechanism by which MedTPE simultaneously claims fixed vocabulary size, fixed parameter count, and computational completeness.

Model integration is correspondingly minimal. Given a pretrained LLM with tokenizer $c^{(i)}_j$2, vocabulary $c^{(i)}_j$3, and embedding matrix $c^{(i)}_j$4, MedTPE constructs $c^{(i)}_j$5, $c^{(i)}_j$6, and a split embedding matrix in which surviving original token embeddings are frozen and the rows corresponding to new TPE tokens are trainable. The embedding matrix size does not change; some rows are simply reinterpreted as composite tokens (Zhu et al., 12 May 2026).

New composite token embeddings are initialized by averaging constituent embeddings:

$c^{(i)}_j$7

followed by norm-based rescaling with $c^{(i)}_j$8 to match the magnitude range of existing embeddings (Zhu et al., 12 May 2026). This initialization is not sufficient by itself. MedTPE therefore uses self-supervised fine-tuning (SSFT): all LLM parameters except the new token embeddings are frozen, the original model provides pseudo-labels on the medical corpus, and the MedTPE-augmented model is trained to predict those pseudo-labels using standard cross-entropy. Reported hyperparameters include AdamW, learning rate $c^{(i)}_j$9, effective batch size 4, maximum input/output length 4,096 tokens, linear warmup over 10% of steps, cosine decay, and early stopping after three non-improving validations (Zhu et al., 12 May 2026).

The ablation results are unusually direct. Without SSFT, the paper reports a catastrophic performance drop, near-zero F1, and degraded latency because average-initialised composite tokens behave as noisy out-of-distribution embeddings. Without dependency-aware replacement, the paper reports a slight accuracy drop and increased inference time because the enlarged vocabulary forces the model to resolve competition between original multi-token and new single-token representations. This establishes that MedTPE is not merely a bigram-merging heuristic; its reported effectiveness depends on both replacement and SSFT (Zhu et al., 12 May 2026).

4. Empirical behavior on clinical prediction and other domains

The principal experiments cover two structured EHR datasets. MIMIC-IV is described as an ICU dataset with 42,117 patients, 57,523 ICU stays, and approximately 42k distinct events; average tokens per patient are reported as approximately 13,456 for Qwen2, 12,145 for Llama3, and 13,486 for Gemma2. EHRSHOT is described as a longitudinal Stanford Hospital dataset with 6,739 patients, 921,499 visits, approximately 31k distinct events, and average token counts of approximately 190,660 for Qwen2 and 158,004 for Llama3 (Zhu et al., 12 May 2026). The four main tasks are ICU mortality, phenotyping, 30-day readmission, and 1-year pancreatic cancer prediction.

Across tasks and tokenisers, MedTPE achieves a compression rate

$o^{(i)}_j$0

between 22.8% and 32.4%, and reduces end-to-end inference time by 33.9% to 62.5% (Zhu et al., 12 May 2026). Representative examples include Qwen2.5-1.5B on MIMIC-IV phenotyping, where baseline time is 29.7 minutes and MedTPE time is 13.0 minutes, a 56.2% reduction, and Llama3-1B on EHRSHOT pancreatic cancer, where baseline time is 28.5 minutes and MedTPE time is 15.9 minutes, a 44.3% reduction (Zhu et al., 12 May 2026). The paper further states that with a budget of 5,000 bigrams, MedTPE approaches the empirical compression bound on MIMIC-IV and EHRSHOT, and that bigrams give the best compression rate because longer $o^{(i)}_j$1-grams are rarer (Zhu et al., 12 May 2026).

On predictive performance and output format compliance, the reported pattern is task- and model-dependent but generally favorable. For Llama3-1B, ICU mortality F1 rises from 0.033 to 0.109 and format compliance rate (FCR) from 0.330 to 0.891; phenotyping F1 rises from 0.050 to 0.114 and FCR from 0.492 to 0.870; 30-day readmission F1 rises from 0.098 to 0.191 and FCR from 0.425 to 0.877; 1-year pancreatic cancer F1 rises from 0.039 to 0.047 and FCR from 0.748 to 0.900. For Qwen2.5-1.5B, ICU mortality F1 remains 0.122 while latency falls by 38.6% and FCR rises from 0.998 to 1.0; phenotyping F1 rises from 0.201 to 0.218 and FCR from 0.969 to 0.999; readmission F1 rises from 0.205 to 0.209 and FCR from 0.989 to 0.990; pancreatic cancer F1 rises from 0.047 to 0.066 and FCR from 0.877 to 0.947 (Zhu et al., 12 May 2026).

The same study reports cross-domain generalisability. On MIMIC-IV-Note, MedTPE improves Llama3-1B MDC coding F1 from 0.019 to 0.048, FCR from 0.763 to 0.994, and reduces time from 11.7 to 2.7 minutes; for 30-day readmission from discharge summaries, F1 rises from 0.201 to 0.302, FCR from 0.702 to 0.916, and time drops from 13.2 to 9.0 minutes (Zhu et al., 12 May 2026). On ARC-Challenge, accuracy and F1 decline modestly while latency falls sharply; on ECTSum, ROUGE and BERTScore remain similar or improve slightly while time decreases; on the Chinese medical QA dataset CMedQA2, ROUGE metrics remain almost identical and latency decreases (Zhu et al., 12 May 2026). This suggests that the gains are not confined to one EHR formatting regime, although the paper also reports domain sensitivity when an EHR-specific MedTPE vocabulary mined on MIMIC-IV is transferred directly to EHRSHOT without re-mining or SSFT.

5. Relation to adjacent medical tokenization research

MedTPE sits within a broader line of work arguing that tokenization is not a neutral preprocessing step in medical NLP. The MedEIR study, which focuses on embedding models for retrieval rather than prompt compression, reports that general-domain tokenizers often misinterpret medical vocabulary and that a custom WordPiece tokenizer trained on curated medical data reduced the number of sub-tokens by 30% on 5,000 MedRAG abstracts compared to Jina, translating to up to 20% lower memory usage (Selvadurai et al., 12 May 2025). The examples given there—such as ib, ##up, ##ro, ##fen becoming ibuprofen, or gas, ##tro, ##es, ##op, ##ha, ##ge, ##al, ref, ##lux becoming gastroesophageal, reflux—illustrate the same underlying pathology that MedTPE targets in EHR text: general-domain segmentation fails to preserve medically meaningful units (Selvadurai et al., 12 May 2025).

The conceptual link is precise. MedEIR extends a general-domain tokenizer with medical terminology, deduplicates overlaps, and retrains the encoder end to end. MedTPE, by contrast, does not primarily seek a richer single-token medical lexicon; it mines frequent contiguous token sequences already produced by the base tokenizer and promotes selected pairs to composite tokens under a fixed-size vocabulary constraint. A plausible interpretation is that MedEIR addresses medical mis-segmentation at the level of token identity, whereas MedTPE addresses it at the level of recurrent token adjacency.

A second adjacent line is represented by MedTok, a multimodal medical code tokenizer for structured EHR foundation models. MedTok does not implement token-pair encoding in the textual bigram sense. Instead, it tokenizes medical codes using textual descriptions and relational graph context, quantizing both modalities into a unified token space. Swapping standard EHR tokenizers with MedTok improved AUPRC across all tested EHR models, by 4.10% on MIMIC-III, 4.78% on MIMIC-IV, and 11.32% on EHRShot, with the largest gains in drug recommendation (Su et al., 6 Feb 2025). This result is relevant because it shows, from a different direction, that clinically grounded tokenization can improve downstream inference without altering the downstream task definition.

Together, these studies delineate three distinct but related design philosophies. MedEIR uses a custom WordPiece medical vocabulary for better segmentation in long-form retrieval (Selvadurai et al., 12 May 2025). MedTok uses multimodal code representations and vector quantization to build a compact semantic tokenizer for structured medical codes (Su et al., 6 Feb 2025). MedTPE uses token-pair mining, dependency-aware replacement, and self-supervised embedding adaptation to compress long EHR prompts without adding modules or increasing model size (Zhu et al., 12 May 2026). This suggests that “medical tokenization” is not a single problem: vocabulary extension, semantic code abstraction, and pair-based compression target different failure modes of general-purpose tokenizers.

6. Limitations, failure modes, and future directions

The most immediate limitation is deployment access. MedTPE requires direct access to the tokenizer and embedding matrix and therefore cannot be used with closed-weight APIs where the vocabulary is fixed (Zhu et al., 12 May 2026). It also has clear domain dependence: TPE vocabularies mined from MIMIC-IV or EHRSHOT reflect local documentation styles, and transferring a vocabulary without re-mining or SSFT degrades both performance and compression rate (Zhu et al., 12 May 2026). This is not merely an engineering inconvenience; it implies that the token-pair inventory is tied to corpus-specific distributions of units, templates, abbreviations, and recurrent phrase fragments.

The method also favors frequent patterns. The paper explicitly notes that rare and evolving concepts may not be merged because MedTPE optimizes for frequent $o^{(i)}_j$2-grams (Zhu et al., 12 May 2026). A plausible implication is that token-pair compression improves the dense middle of the medical distribution while leaving the long tail to back-off tokenisation. This is consistent with the reported entropy-based analysis, according to which compression saturates at approximately 30% as the budget $o^{(i)}_j$3 increases because the remaining patterns are too rare to justify dedicated tokens (Zhu et al., 12 May 2026).

Another boundary condition concerns prompting and instruction alignment. Under chain-of-thought prompting, MedTPE still reduces inference time and often improves F1 for several general models, but on Meditron3-8B for phenotyping it causes F1 to decline from 0.176 to 0.137, FCR to drop from 0.987 to 0.852, and time to increase from 94 to 135.8 minutes. The reported qualitative failure modes are reasoning drift with preserved JSON structure and output format breakdown in which the model emits free-form explanations instead of the required "answer": [...] object (Zhu et al., 12 May 2026). The paper interprets this as evidence that clinically continual-pretrained models may have fragile instruction alignment under changed input tokenisation distributions.

The future directions identified in the source material extend naturally beyond bigrams. MedTPE explicitly mentions exploring more structured $o^{(i)}_j$4-gram or concept-level merging, instruction-aware token-pair selection, integration with retrieval-augmented generation, multimodal EHRs, and deeper work on full-text clinical notes (Zhu et al., 12 May 2026). MedEIR’s emphasis on jointly optimized tokenization and retrieval-oriented embedding training suggests one path in which token-pair boundaries are tuned for downstream retrieval and RAG rather than only for compression (Selvadurai et al., 12 May 2025). MedTok suggests another path in which pair tokens could incorporate code descriptions, ontology structure, and code-code relations rather than only contiguous token frequency (Su et al., 6 Feb 2025). In that sense, MedTPE can be understood not as an isolated compression trick but as one instance of a broader research program: redesigning the tokenizer itself as a domain-adapted representational layer for medical foundation models.