FreeToken: Elastic Edge Serving for MoE Models

• FreeToken is an edge-native MoE serving system that unifies GPU and CPU memory, PCIe bandwidth, storage, computation, and runtime state to adapt inference to changing hardware and agent workloads.
• Its elastic expert cache, bandwidth-aware CPU–GPU execution, double-buffered prefill, and semantic state reuse support models from 35B parameters on an 8 GB laptop GPU to 753B on a 96 GB workstation GPU.
• Evaluations report 1.3–2.1× higher decode throughput than leading baselines across consumer systems, 1.5–2.3× gains on key MoE models, and worst-turn TTFT below 44 seconds in tested agent workloads.

FreeToken is an edge-native serving system for mixture-of-experts (MoE) LLMs that orchestrates GPU memory, CPU memory, host computation, PCIe bandwidth, storage, and runtime state as a unified elastic inference platform. Rather than assuming datacenter-scale GPU capacity or a fixed CPU–GPU offloading policy, it adapts model placement and execution to currently available hardware resources and changing agent workloads. The system supports more than 20 MoE models and reports deployments ranging from a 35B-parameter model on an 8 GB laptop GPU to a 753B-parameter model on a single 96 GB workstation GPU (Yang et al., 17 Aug 2026).

1. Problem scope and design rationale

An MoE model contains $E$ experts, but each token is routed to only $k$ experts, where $k \ll E$. This sparsity reduces the arithmetic required per token, but it does not eliminate the storage requirement for the complete expert pool. When the pool exceeds GPU memory, experts must remain in host memory or storage and be transferred or executed on demand.

FreeToken addresses this distinction between computational sparsity and storage capacity. Its target environment is a personal computer or workstation with limited VRAM, heterogeneous PCIe and DRAM bandwidth, shared system resources, and dynamically changing availability of GPU memory. Agentic workloads further complicate serving because they repeatedly modify context, invoke tools, spawn subagents, and re-enter the model with altered histories.

The system is designed around several characteristics of local AI serving:

• Expert routing changes token by token and workload by workload.
• Prefill can activate a broad union of experts even when decoding is sparse.
• Full-attention KV state and recurrent state have different reuse properties.
• CPU and GPU memory bandwidth differ substantially across machines.
• GPU memory must be divided dynamically between expert residency, KV cache, and other runtime state.
• Personal machines may experience interference from browsers, games, desktop applications, and other processes.

The paper uses DeepSeek-V4-Flash to illustrate the storage problem. The model has 284B total parameters, 13B active parameters per token, 256 routed experts, 6 routed experts activated per layer, and 43 MoE layers. Although the active computation may fit within the effective working memory of a consumer GPU, the complete expert pool cannot. The resulting bottleneck is primarily data movement, placement, and scheduling rather than arithmetic capacity (Yang et al., 17 Aug 2026).

Prefill is especially demanding. A long prompt may route across most experts in each layer, making demand-fetch strategies expose nearly the entire expert pool’s transfer time as GPU idle time. For the FP4 deployment of DeepSeek-V4-Flash, approximately 140 GB of expert weights are transferred during prefill, requiring roughly 2 seconds on an RTX 5090 with PCIe 5.0 $\times16$, roughly 5 seconds on RTX 4090/3090-class systems with PCIe 4.0 $\times16$, and 10 seconds or more on laptop links commonly using $\times8$.

2. System architecture and model residency

FreeToken uses a two-level expert-memory hierarchy:

1. A CPU-resident expert pool containing the complete routed-expert weights and serving as the correctness source of truth.
2. A GPU expert cache containing a subset of complete experts and accelerating the current working set.

Non-expert weights remain resident on the GPU. The remaining GPU memory is dynamically divided between the KV cache and other runtime state, and a shared expert cache used across MoE layers.

Each cache entry represents a complete logical $(\text{layer},\text{expert})$ pair. Residency, lookup, eviction, transfer, and execution operate on logical experts rather than fragmented tensor shards. This abstraction allows CPU and GPU kernels, storage layout, and scheduling logic to share a common identity for each expert.

FreeToken introduces the FreeToken Weight (FTW) format. Model checkpoints are normalized into expert banks using a flattened identifier:

$\text{expert\_id}=lE+e,$

where $l$ is the layer index, $E$ is the number of experts per layer, and $k$0 is the expert index within the layer. Rows with the same identifier across all banks collectively form one complete expert.

FTW stores expert weights already merged into the runtime bank layout. Startup can consequently use aligned chunks, parallel direct I/O, direct reads into exact-size host banks, and delayed memory pinning. This avoids tensor discovery and runtime repacking. A 140 GB expert pool read from a 7 GB/s NVMe device would take roughly 20 seconds before warmup; FreeToken reduces startup overhead by reading directly into the final host-bank layout, using aligned parallel direct I/O, and pinning populated memory only after loading.

The CPU pool remains available even when GPU cache capacity changes. Evicting an expert affects performance but not correctness, because the expert can always be recovered from host memory. This separation enables cache resizing without reloading the complete host pool or restarting the serving engine.

During prefill, FreeToken uses full-layer double buffering. Two full-layer buffers are obtained from a global slot pool. While the GPU computes layer $k$1, a transfer stream loads all experts for layer $k$2, after which the buffers exchange roles. Loading an entire next layer before its routing is known is deliberate: long prompts commonly activate most experts, so demand fetching provides little benefit during prefill.

The prefill and decode cache share the same slot pool. Experts loaded during prefill can therefore seed the decode cache rather than being discarded at the phase boundary. If two full-layer buffers cannot fit, FreeToken falls back to on-demand prefill loading.

During decode, routed experts are divided into cache hits and misses. Hits refresh recency in a shared global LRU cache. Newly fetched experts are admitted, and the least recently demanded experts are evicted. On identical routing traces and equal cache capacity, the paper reports the following decode-time miss rates on an RTX 5090:

Model	FreeToken global LRU	KTransformers placement	llama.cpp static split
Qwen3.6-35B-A3B	16%	41%	62%
DeepSeek-V4-Flash	39%	59%	89%

At that machine’s capacity, the cache holds approximately 37% of Qwen3.6’s expert pool and 11% of DeepSeek-V4-Flash’s expert pool (Yang et al., 17 Aug 2026).

3. Bandwidth-adaptive CPU–GPU execution

A cache miss does not necessarily trigger a PCIe transfer. FreeToken chooses between two execution paths:

• Transfer the expert over PCIe, execute it on the GPU, and retain it in the GPU cache.
• Execute the expert directly on the CPU from the host-resident pool.

Suppose a decode layer has $k$3 unique missing experts. FreeToken partitions them as

$k$4

where $k$5 is the set transferred into GPU cache slots and executed on the GPU, and $k$6 is the set executed directly on the CPU. If $k$7 denotes cache hits, the complete GPU execution set is

$k$8

The CPU branch returns gate-weighted partial outputs. CPU and GPU partial results are merged exactly; the system does not approximate, skip, substitute, or alter routed computation.

FreeToken measures bandwidth on the actual deployed tensor shapes rather than relying solely on hardware specifications. The relevant quantities are the pinned host-to-device transfer bandwidth $k$9 and the effective CPU-side expert-processing bandwidth $k \ll E$0. Because PCIe transfers and CPU execution share the host-memory subsystem, PCIe activity leaves residual bandwidth for CPU execution:

$k \ll E$1

For an expert size of $k \ll E$2 bytes, the approximate concurrent branch times are

$k \ll E$3

and

$k \ll E$4

The layer latency is approximately

$k \ll E$5

Balancing the two branches yields the closed-form policy

$k \ll E$6

The runtime rounds $k \ll E$7 to an integer, uses cache replacement to select which misses enter $k \ll E$8, assigns the remaining misses to $k \ll E$9, and retains at least one fill so the cache continues warming.

The execution sequence is:

1. Launch the CPU branch.
2. Update the GPU cache.
3. Copy the selected $\times16$0 experts.
4. Evaluate $\times16$1 on the GPU.
5. Execute $\times16$2 concurrently on persistent CPU workers.
6. Merge CPU and GPU partial outputs.

The policy varies with the hardware balance. Strong PCIe bandwidth causes more misses to be fetched and cached. Relatively competitive CPU bandwidth causes more misses to execute in place. When PCIe nearly consumes available host bandwidth, CPU execution receives little residual bandwidth; when CPU bandwidth substantially exceeds PCIe bandwidth, the CPU can absorb more misses concurrently.

The evaluated hardware demonstrates this heterogeneity. Measured transfer and CPU bandwidths include 52.7 GB/s and 77.3 GB/s on an RTX 5090 server, 25.1 GB/s and 63.2 GB/s on an RTX 4090 server, 25.3 GB/s and 56.7 GB/s on an RTX 3090 server, 49.0 GB/s and 53.8 GB/s on an RTX 5090 desktop, 11.8 GB/s and 47.5 GB/s on an RTX 4060 laptop, and 51.5 GB/s and 178 GB/s on an RTX PRO 6000 system.

The closed-form model is intentionally bandwidth-oriented. Actual performance may also depend on CPU compute overhead, NUMA placement, PCIe contention, cache effects, synchronization, variable expert tensor sizes, and concurrent applications. The paper reports empirical validation of the policy but does not provide a detailed prediction-error analysis.

4. Agentic state reuse and runtime memory management

FreeToken treats agent execution as a state-reuse problem rather than merely a sequence of independent prompts. Agent workloads may append tool calls and results, remove or replace reasoning, compact context, spawn subagents, and re-enter the model with modified histories.

For full-attention layers, FreeToken uses a radix prefix tree for KV-cache reuse. Shared prefixes can be reused across requests and agent turns.

Recurrent layers require a different mechanism. A recurrent state summarizes the prefix into an evolving state and cannot be reused through ordinary token-level KV entries. FreeToken therefore maintains semantic-aware recurrent-state checkpoints attached to prefix-tree nodes. Checkpoints are preferentially placed at semantic boundaries such as thinking segments, tool calls, tool outputs, assistant outputs, and conversation turns.

If a context edit preserves a prefix up to an anchor, FreeToken can restore full-attention KV up to the anchor and recurrent states at the anchor, then recompute only the new suffix. This is intended for agent frameworks that edit complete semantic blocks, such as replacing an old tool output or removing an earlier reasoning segment.

The recurrent checkpoint pool uses LRU eviction independently of the KV pool. Each checkpoint contains complete recurrent state for all relevant layers, so only a limited number can be retained. Semantic placement is intended to increase the probability that useful checkpoints survive later agent edits. Arbitrary token-level edits, unusual context transformations, or workloads without stable semantic boundaries may reduce the benefit.

The decode expert cache also uses LRU eviction, but its locality signal is routing demand rather than context structure. Hits refresh recency, fills insert experts, and victims are the least recently demanded slots. This separates two forms of locality:

• Semantic locality for recurrent states and reusable prefixes.
• Short-range routing locality for experts during decoding.

GPU memory management is elastic. At scheduler safe points, FreeToken can rebuild the expert cache under a revised VRAM budget as KV-cache demand grows or other applications consume GPU memory. It does not reload the complete host-resident expert pool or restart the engine. Runtime reconfiguration may nevertheless add management work and temporarily reduce useful cache capacity; the supplied evaluation does not quantify resizing overhead.

If pinned or registered host memory is unavailable because of operating-system or driver restrictions, FreeToken falls back to a pure-CPU MoE backend. Experts remain in pageable host memory, routed experts execute on the CPU, non-expert layers remain on the GPU, and only activation-sized inputs, routing metadata, and aggregated outputs cross the boundary. This sacrifices peak bandwidth but improves deployability.

FreeToken also avoids a separate GPU warmup phase. The first request uses the ordinary cold-cache path, after which normal serving warms the cache.

5. CUDA-Graph-compatible execution

Dynamic MoE routing ordinarily creates host-side synchronization and variable work. FreeToken keeps routing-dependent control on the GPU to preserve a CUDA-Graph-compatible execution structure.

A GPU kernel deduplicates routed experts, classifies hits and misses, computes the fetch count $\times16$3, selects eviction victims, and rewrites logical expert identifiers into physical cache-slot identifiers or a CPU-assignment flag. The cache uses fixed-shape work buffers and device-resident valid counts, allowing unused entries to be masked within a captured CUDA Graph.

Victim selection identifies the least-recently-used candidate slots in one pass rather than scanning the complete cache separately for each eviction. A single fused transfer consumes the resulting copy list.

The CPU branch is also integrated into the graph-resident execution path through stable pinned I/O buffers, persistent task descriptors, device-to-host copies, a host-function submission node, concurrent GPU execution, synchronization, and host-to-device result copying. Persistent C++ workers are pinned to physical cores and use architecture-specific SIMD and in-kernel dequantization.

This design allows FreeToken to combine dynamic routing with bounded-shape execution. The scheduler changes expert assignments and cache state while avoiding a host-side control path that would otherwise repeatedly interrupt GPU execution.

6. Models, workloads, implementation, and evaluation

FreeToken builds on GPU-centric serving mechanisms associated with SGLang and vLLM, including paged KV-cache management, radix-based prefix reuse, FlashInfer, Flash Linear Attention where applicable, CUDA Graph execution, and custom GPU and CPU expert kernels.

The primary evaluated models are:

• Qwen3.6-35B-A3B: 35B total parameters and 3B active parameters, served mostly in BF16, with an official NVFP4 release used on the RTX 4060 laptop.
• DeepSeek-V4-Flash: 284B total parameters and 13B active parameters, with native MXFP4 routed experts and approximately 140 GB of FP4 expert weights in the discussed deployment.
• GLM-5.2: 753B total parameters and 40B active parameters, with NVFP4 routed experts and an approximately 433 GB checkpoint.

The implementation supports more than 20 MoE models, although the supplied quantitative evaluation focuses principally on these three.

Four agentic workloads are evaluated:

1. W1: AIME math reasoning, a long chain-of-thought, decode-dominated, single-turn workload without tool use.
2. W2: OpenCode coding agent, using a SWE-bench repository issue, real tool execution, three scripted user turns, and a requirement to produce the reference patch.
3. W3: Claude Code native protocol, using the same SWE-bench issue, an Anthropic-compatible endpoint, concurrent subagent requests, and sessions growing to approximately 56–65k tokens.
4. W4: OpenClaw email/calendar agent, using 13 fixed user turns and an approximately 24.5k-token system-context floor, with all turns required to complete.

The principal metrics are per-request mean decode throughput in tokens per second, per-request mean time to first token, worst-turn TTFT in some analyses, prefill throughput, expert miss rate, and cross-hardware scaling. Total wall-clock trajectory time is not compared across engines because agent trajectories diverge.

The baselines are llama.cpp, Ollama, KTransformers, and MoE-Infinity where supported. Formats are aligned: Qwen3.6 is served in BF16 across engines, while DeepSeek-V4-Flash uses its native MXFP4 expert blocks bit-exactly.

On an RTX 5090, FreeToken reports 77–83 tokens/s for Qwen3.6-35B-A3B and 22–25 tokens/s for DeepSeek-V4-Flash. Reported improvements over the strongest baseline in each workload are 1.8–2.3$\times16$4 for Qwen3.6 and 1.5–1.9$\times16$5 for DeepSeek-V4-Flash.

FreeToken’s decode rate remains within 12% of its single-turn W1 rate across the three agent workloads. KTransformers on DeepSeek-V4-Flash has lost 31% of its W1 rate by W2. MoE-Infinity serves only W1 at 8.8 tokens/s in the supplied comparison.

FreeToken reports the lowest mean TTFT in five of six multi-turn model/workload cells. The exception is Qwen3.6 on W3, where KTransformers’ GPU-prefill path performs better. Worst-turn TTFT remains below 44 seconds in every cell for FreeToken, while every baseline exceeds 150 seconds in at least one setting. Reported baseline maxima include 232 seconds for llama.cpp, 179 seconds for Ollama, and 946 seconds for KTransformers.

On Qwen3.6 using the RTX 5090, an 8,192-token prefill chunk completes in 1.19–1.22 seconds, approximately the time needed to stream a 64.4 GB expert pool at 52.7 GB/s. Prefill throughput reaches 6.7k tokens/s at 16k tokens. Disabling the second buffer reduces throughput by 19% at 4k tokens, 25% at 8k tokens, and 26% at 16k tokens.

Across five consumer systems, FreeToken improves decode throughput over the strongest baseline by 1.3$\times16$6 on an RTX 3090, 1.3$\times16$7 on an RTX 4090, 1.9$\times16$8 on an RTX 5090 server, 2.1$\times16$9 on an RTX 5090 desktop, and 1.8$\times16$0 on an RTX 4060 laptop. On the 8 GB RTX 4060 laptop, the NVFP4 Qwen3.6 build reaches 39.3 tokens/s.

The largest demonstrations are a 35B model on an 8 GB laptop GPU, 284B DeepSeek-V4-Flash on a 32 GB RTX 5090 gaming desktop, and 753B GLM-5.2 on a single 96 GB RTX PRO 6000. For GLM-5.2 on the RTX PRO 6000, FreeToken reaches 14.9 tokens/s compared with 7.3 tokens/s for llama.cpp, a 2.0$\times16$1 speedup. Mean TTFT is 7.5 seconds for FreeToken and 7.8 seconds for llama.cpp.

The paper identifies availability implications in addition to latency implications. OpenClaw has a 120-second idle watchdog, while Claude Code’s default request timeout is roughly ten minutes. Long TTFT tails can therefore cause an agent session to fail rather than merely respond slowly.

7. Limitations, comparisons, and broader significance

FreeToken does not eliminate bandwidth constraints. Slow PCIe, weak DRAM, insufficient CPU bandwidth, or simultaneous pressure on both resources can still limit performance. Full-layer double buffering hides transfer only when computation can overlap movement and enough GPU memory exists for two layer buffers. Otherwise the system falls back to on-demand prefill loading.

The LRU cache cannot guarantee low miss rates under cold starts, abrupt routing shifts, or very small GPU budgets. Correctness remains intact because the CPU pool is the source of truth, but latency may degrade. Semantic recurrent-state checkpoints consume memory and may be evicted before reuse. Runtime cache resizing can introduce management overhead. CPU execution depends on compatible kernels, physical-core placement, NUMA behavior, and the ability to register host memory for DMA.

The comparison is also bounded by baseline support. Ollama and MoE-Infinity lack DeepSeek-V4-Flash support in the supplied evaluation. MoE-Infinity has no usable multi-turn agent server in the evaluated setup. KTransformers lacks a servable GLM-5.2 path on the RTX PRO 6000 because its GLM methods require approximately 753 GB–1.5 TB of host-resident experts against 512 GiB available, and its CPU kernels do not support GLM-5.2’s NVFP4 layout. Agent trajectories diverge across systems, preventing universal comparison by total wall-clock time.

The supplied evaluation emphasizes throughput and TTFT. It does not report output-quality changes, energy consumption, heavy-concurrency performance, arbitrary desktop memory-pressure behavior, model-accuracy parity for every supported model, or detailed measurements for all more than 20 supported MoEs. It also does not provide a complete numerical ablation isolating semantic checkpoints, runtime resizing, or the $\times16$2 policy independently.

FreeToken’s principal conceptual contribution is the transition from static offloading to elastic orchestration. Static serving asks which experts should permanently reside on the GPU or CPU. FreeToken instead adapts to the current phase, routing pattern, memory budget, and measured hardware balance:

$\times16$3

This places FreeToken in a broader systems context involving model quantization, MoE routing, heterogeneous memory, CPU–GPU co-execution, KV-cache management, recurrent-state reuse, CUDA Graphs, and agent-runtime design. Related token-oriented systems address different layers of the token abstraction. FedToken uses contribution-based blockchain rewards for federated-learning clients (Pandey et al., 2022); PTTS studies privacy-preserving token transfer, although the supplied technical material is insufficient to reconstruct its protocol (Ismayilov et al., 2023); FORT uses zero-knowledge proofs to demonstrate possession of service rights without revealing sensitive attributes (Salleras et al., 2022); and Fermilab’s production migration demonstrates end-to-end lifecycle management for scoped, short-lived authentication tokens (Dykstra et al., 31 Mar 2025). These systems do not define FreeToken’s MoE serving architecture, but they illustrate that “token” may refer to computational units, economic rights, authentication credentials, or privacy-preserving authorization objects.

For FreeToken specifically, the evidence supports an edge-serving system that expands the practical deployment range of open-weight MoE models under constrained hardware. It does not imply that a personal computer becomes equivalent to a datacenter cluster. Host bandwidth, PCIe transfer time, startup cost, cache capacity, recurrent-state memory, concurrent workload pressure, and agent-specific context behavior remain fundamental constraints. The demonstrated result is narrower and more technical: coordinated use of heterogeneous local resources can make frontier-scale sparse models deployable in configurations where static GPU-centric or CPU-centric serving systems cannot provide comparable functionality or latency.