Prompt–Token Disaggregation

• Prompt–token disaggregation is a framework that explicitly separates and analyzes language model inputs into individual tokens to optimize computation and model alignment.
• It enables dynamic token-level routing, parallel prompt pruning, and fine-grained supervision to improve efficiency in multi-turn serving and privacy routing.
• Empirical results demonstrate substantial latency reductions and performance gains, making it valuable for compression, security, and robustness in modern LLM deployments.

Prompt–token disaggregation refers to the explicit separation, analysis, and management of LLM input at the level of individual tokens, rather than treating prompts as monolithic, undifferentiated strings. This paradigm enables systems to exploit per-token computational, optimization, and routing strategies, dramatically improving efficiency, robustness, and interpretability in multi-turn serving, prompt engineering, privacy routing, and model alignment. Techniques cover architectural disaggregation in LLM serving, per-token supervision or attribution, fine-grained sequence labeling, and token-aware prompt compression.

1. Architectural Motivation and Formal Definitions

Prompt–token disaggregation has multiple formalizations depending on context, but shares the core principle of distinguishing between prompt input structure (prefill, context, new tokens) and per-token computational or semantic value. In LLM inference, particularly in multi-turn chat and agentic systems, the distinction between the prefill and decode stages is foundational (Li et al., 9 Mar 2026, Liu et al., 1 Dec 2025). Let $N$ denote the total token length of a prompt, decomposed over conversation history plus new user input.

• Prefill Stage: Given an input prompt of length $N$, prefill runs full self-attention across all $N$ tokens, materializing the entire key/value (KV) cache. This is a compute-bound operation with $O(N^2)$ complexity (or $O(N)$ with FlashAttention).
• Decode Stage: Given the populated KV cache, the autoregressive decode step generates output tokens, with cost linear in KV length per output (Li et al., 9 Mar 2026).
• Append-Prefill: For multi-turn conversations, where $n$ is the cached KV and $m \ll n$ are new tokens, only the new inputs need full attention, reducing complexity to $O(m(n+m))$ (Li et al., 9 Mar 2026).

This stage separation underlies standard PD (Prefill–Decode) disaggregation, mapping stages onto separate GPU pools and enabling parallelization. Prompt–token disaggregation generalizes this insight, proposing further specialization at the token-level by routing only necessary computations for increments of conversation state, not for the entire prompt history.

2. Algorithmic and Optimization Frameworks

Prompt–token disaggregation is operationalized by designing architectures, algorithms, and optimization objectives that keep track of, or route, token-level computation:

• Dynamic Routing (PPD): Prefill Prefill-capable Decode (PPD) introduces a per-request routing variable $x \in \{0, 1\}$, signifying whether a Turn 2+ request should be handled via append-prefill on the decode node (with cached KV) or follow the original PD logic (Li et al., 9 Mar 2026). The optimization target is

$$\min_{x\in\{0,1\}} w_{\mathrm{ttft}}\;\mathrm{TTFT}(x) + w_{\mathrm{tpot}}\;\mathrm{TPOT}(x)$$

with operator-selected service level objective (SLO) weights. The policy table is constructed offline via benchmarking, then decisions are performed online at runtime on a per-session-turn basis.

• Continuous Batching and Stage-aware Scheduling: In vector-search augmented retrieval-augmented generation (RAG) serving, Trinity (Liu et al., 1 Dec 2025) merges PD disaggregation with a dedicated vector-search pool, and fine-tuned scheduling across prefill (prompt), decode (token), and retrieval components further exploits the separation of prompt–token dependencies to achieve higher tail-latency compliance.
• Token-wise Prompt Supervision: In sequence labeling for cross-lingual tasks, ToPro (Ma et al., 2024) decomposes input sentences $N$0 into $N$1 per-token prompts, each with a distinct [MASK] and verbalizer, and aggregates predictions:

$N$2

with associated cross-entropy loss over all tokens (Ma et al., 2024).

• Parallel Prompt Pruning via Diffusion: DiffuMask (Zheng et al., 8 Apr 2026) formulates token-level retention as a binary mask $N$3, with diffusion over per-token coordinates, learning to denoise all retention decisions in parallel. The model is trained to minimize

$N$4

enabling massive speedup over sequential or greedy token-level pruning (Zheng et al., 8 Apr 2026).

3. Empirical Evidence and Quantitative Impact

Empirical evaluation across LLM serving, compression, and supervision tasks demonstrates the practical impact of prompt–token disaggregation:

Context	Technique	Key Metric	Improvement	Source
Multi-turn LLM serving	PPD dynamic	Turn 2+ TTFT	–68% average; up to –73% at scale	(Li et al., 9 Mar 2026)
	routing	Turn 2+ TPOT slowdown	Only 2–21% for append-prefill	(Li et al., 9 Mar 2026)
Prompt pruning	DiffuMask	Prompt length reduction	~80% tokens cut, no accuracy drop	(Zheng et al., 8 Apr 2026)
		Pruning time	0.75 min vs. >1000 min (greedy)	(Zheng et al., 8 Apr 2026)
Salience attribution	FrugalPrompt	Performance at 20% pruned tokens	–1 to –3% (QA, sentiment, summarization)	(Raiyan et al., 18 Oct 2025)
Per-token labeling	ToPro	Zero-shot NER (mT5, PAN-X, F1)	92.8 vs. 64.2 (vanilla)	(Ma et al., 2024)

In large-scale real-world workloads, PPD eliminates network/queue-induced Turn 2+ service degradation, achieving ≥95% success rate and removing high-QPS failure points seen in standard PD serving (Li et al., 9 Mar 2026).

4. Tokenization, Robustness, and Model Behavior

Prompt–token disaggregation exposes structural vulnerabilities related to tokenization boundaries and model alignment. Xu et al. (Xu et al., 30 Jan 2026) precisely formalize the partial-token problem (PTP): for a prompt $N$5 and continuation $N$6,

$N$7

results in catastrophic distortion of $N$8, with log-probability drops of $N$9 to $N$0 and 60–95% absolute accuracy loss, particularly in languages or inputs with high word-token misalignment (e.g., Chinese, compounding languages, and code) (Xu et al., 30 Jan 2026). Even “natural” prompts respecting word boundaries can fail, as up to 25% of word ends in Chinese fall within token boundaries.

Recommended mitigation involves either strictly aligning user input to token boundaries (by prompt truncation) or marginalizing over all possible tokenizations with exact inference-time samplers such as ByteSampler, restoring 100% accuracy at the cost of at most 1.2 extra forward passes (Xu et al., 30 Jan 2026).

Furthermore, token-level differences can drive significant behavioral drift in LLMs even when semantic intent is preserved (prompt variance). The Prompt-Based Semantic Shift (PBSS) diagnostic (Li et al., 11 Jun 2025) shows that model response shift is quantitatively tied to token-level realization; drift is measurable in cosine space and correlates with tokenizer granularity ($N$1 across 9 models). Instruction-tuned models still exhibit about 20% of prompt pairs with $N$2 semantic drift, and post-tokenization normalization steps are recommended for stability (Li et al., 11 Jun 2025).

5. Applications in Compression, Privacy, and Fuzzing

Prompt–token disaggregation underpins both efficient and privacy-preserving LLM operations, as well as security analysis:

• Prompt Compression: FrugalPrompt (Raiyan et al., 18 Oct 2025) and DiffuMask (Zheng et al., 8 Apr 2026) both implement per-token retention estimation, using attribution signals (GlobEnc, DecompX) or learned diffusion over binary masks to remove low-utility tokens. In discriminative, generative, and reasoning tasks, 20–40% context reduction can be achieved with negligible accuracy loss for most tasks, but breakages emerge for mathematical or chain-of-thought reasoning, where token continuity is essential.
• Privacy Guard: The contextual compression operator $N$3 in Privacy Guard (Langiu, 30 Mar 2026) links token reduction directly to privacy risk via projections $N$4 and $N$5. Automatic Prompt Optimization (APO) disaggregates the prompt into minimal, task-specific subprompts, enforcing both OpEx (operational cost) reduction and zero information leakage, proven with 100% redaction of personal secrets and 45% blended OpEx savings (Langiu, 30 Mar 2026).
• Token-Aware Fuzzing: For safety and robustness analysis, prompt–token disaggregation enables query-efficient jailbreak fuzzing via per-token refusal attribution, guiding focused mutations to only the most impactful tokens (TriageFuzz (Chen et al., 24 Mar 2026)). The approach yields 90% attack success with 70% fewer queries, versus uniform fuzzing (Chen et al., 24 Mar 2026).

6. Extension to Vision, Multi-modal, and Federated Settings

The paradigm generalizes beyond text-only LLMs:

• Visual Prompting: Works such as TCPA (Liu et al., 5 May 2025) and APLe (Cao et al., 2024) posit that visual prompt matrices in ViTs and CLIP should be disaggregated so that each token (e.g., image patch or CLS embedding) interacts with its own learned or assigned prompts. TCPA matches tokens to prompt pools via affinity measures, enhancing feature diversity and clustering, and overcoming the low-rank barrier of standard shared prompt tuning, yielding higher accuracy and improved representation quality (Liu et al., 5 May 2025).
• Federated Prompt Learning: TRIP (Gong et al., 29 Apr 2025) routes individual vision encoder tokens to prompt expert pools by capacity-aware clustering and cost-minimizing optimal transport—parameter free routing—assigning per-token prompt mixtures and achieving substantial improvements in domain generalization under strict communication constraints.

7. Practical Guidance and Limitations

Prompt–token disaggregation delivers on multiple practical objectives:

1. Serving Efficiency: Route append-prefill jobs to decode nodes in multi-turn serving where possible; treat static PD as a baseline but recognize its inability to jointly optimize TTFT, TPOT, and throughput (Li et al., 9 Mar 2026).
2. Prompt Engineering: Inspect tokenizer boundaries, purposely align prompt edges or use exact sampler marginalization when serving in languages, coding scenarios, or heterogeneous deployment environments (Xu et al., 30 Jan 2026, Li et al., 11 Jun 2025).
3. Compression Policy: Token-level salience analysis or parallel pruning (DiffuMask) for cost, privacy, and efficiency; reasoned retention of full token chains for mathematical or logical reasoning.
4. Privacy/Risk Management: Disaggregate conversations into minimal, intent-preserving, low-entropy token sets, using local LMs for first-pass optimization and risk triage before cloud inference (Langiu, 30 Mar 2026).
5. Security Fuzzing: Assess per-token influence on refusal behavior in adversarial testing; assign mutation quotas based on measured impact rather than uniform allocation (Chen et al., 24 Mar 2026).

Constraints include attribution misalignment where analysis models do not match inference models (Raiyan et al., 18 Oct 2025), generalization gaps across modalities, and the potential for catastrophic drift in non-token-aligned prompt editing. Adaptive or hybrid prompt strategies are recommended for future work.

---

References:

• (Li et al., 9 Mar 2026) Not All Prefills Are Equal: PPD Disaggregation for Multi-turn LLM Serving
• (Xu et al., 30 Jan 2026) Are you going to finish that? A Practical Study of the Tokenization Boundary Problem
• (Ma et al., 2024) ToPro: Token-Level Prompt Decomposition for Cross-Lingual Sequence Labeling Tasks
• (Zheng et al., 8 Apr 2026) DiffuMask: Diffusion LLM for Token-level Prompt Pruning
• (Raiyan et al., 18 Oct 2025) FrugalPrompt: Reducing Contextual Overhead in LLMs via Token Attribution
• (Langiu, 30 Mar 2026) Privacy Guard & Token Parsimony by Prompt and Context Handling and LLM Routing
• (Liu et al., 1 Dec 2025) Trinity: Disaggregating Vector Search from Prefill-Decode Disaggregation in LLM Serving
• (Li et al., 11 Jun 2025) When Meaning Stays the Same, but Models Drift: Evaluating Quality of Service under Token-Level Behavioral Instability in LLMs
• (Liu et al., 5 May 2025) Token Coordinated Prompt Attention is Needed for Visual Prompting
• (Cao et al., 2024) APLe: Token-Wise Adaptive for Multi-Modal Prompt Learning
• (Chen et al., 24 Mar 2026) Not All Tokens Are Created Equal: Query-Efficient Jailbreak Fuzzing for LLMs
• (Gong et al., 29 Apr 2025) Token-Level Prompt Mixture with Parameter-Free Routing for Federated Domain Generalization
• (Wang et al., 2023) TokenCompose: Text-to-Image Diffusion with Token-level Supervision
• (Hedderich et al., 22 Apr 2025) What's the Difference? Supporting Users in Identifying the Effects of Prompt and Model Changes Through Token Patterns