FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution

Abstract: Frontier open-weight models are increasingly available, but serving them still largely assumes datacenter infrastructure. We present FreeToken, an edge-native MoE serving system that treats a personal machine not as a small GPU, but as a unified, elastic inference platform. FreeToken co-designs the full serving stack, including model layout and loading, expert residency, CPU--GPU execution, agentic state reuse, and runtime memory management, around two realities of local AI: agent workloads continuously change their execution pattern, and edge hardware exposes heterogeneous resources whose balance differs from machine to machine. Rather than committing to a fixed offloading strategy, FreeToken continuously maps computation and model state onto the resources actually available. FreeToken supports more than 20 MoE models and real coding and tool-using agents across hardware ranging from an 8GB laptop GPU to a single workstation GPU. More importantly, it changes what these machines can practically serve, from a 35B model on a laptop to a 284B model on a gaming desktop and the 753B GLM-5.2 on a single workstation GPU. FreeToken turns open weights into deployable local software, making the machines users already own a practical platform for frontier-scale intelligence. We release the system at flashml.ai.

• The paper introduces an edge-native MoE runtime that combines pipelined full-layer prefill, semantic recurrent-state reuse, global LRU expert caching, and bandwidth-adaptive CPU–GPU execution without changing model outputs.
• FreeToken delivers 1.5–2.3× higher decode throughput than leading baselines, keeps worst-case TTFT below 44 seconds in tested agent workloads, and reaches 39.3 tokens/s for a 35B model on an 8 GB laptop GPU.
• The system expands local deployment options by serving models such as 753B-parameter GLM-5.2 at 14.9 tokens/s on one RTX PRO 6000, while remaining adaptable to PCIe bandwidth, host-memory limits, and heterogeneous hardware.

Problem formulation and contribution

FreeToken addresses a specific systems problem: serving frontier-scale sparse Mixture-of-Experts (MoE) models on heterogeneous personal hardware when the complete expert pool exceeds GPU memory. The paper’s central claim is that edge inference should not be designed as reduced-capacity GPU serving. Instead, the GPU, CPU, host memory, PCIe interconnect, storage, and dynamically available VRAM should be treated as a unified inference substrate whose execution policy changes with workload phase and measured hardware characteristics.

The motivation follows directly from the structure of contemporary MoE models. A model such as DeepSeek-V4-Flash contains 284B parameters but activates only 13B parameters per token: six of 256 routed experts across each of 43 layers. This sparsity makes per-token computation compatible with a 32 GB RTX 5090, but it does not make the complete expert pool fit in VRAM. Inactive experts must remain in host memory and be transferred or executed on the CPU when routed. Thus, the relevant bottleneck is not simply arithmetic capacity, but the coordination of expert residency, PCIe traffic, CPU memory bandwidth, GPU execution, and state reuse.

The paper identifies three interacting failure modes in existing edge serving systems. During prefill, long prompts activate the union of many experts and consequently behave nearly densely with respect to expert movement. During decode, individual tokens are sparse, but cache misses must be handled by a policy that chooses between PCIe transfer followed by GPU execution and direct CPU execution. Finally, edge machines differ substantially in PCIe bandwidth, DRAM bandwidth, CPU throughput, VRAM capacity, and concurrent memory pressure. A static placement policy cannot therefore be optimal across machines or even across phases of one agent session.

FreeToken’s contribution is an integrated runtime organized around three mechanisms:

1. Pipelined full-layer prefill overlaps expert movement for layer $l+1$ with computation for layer $l$ using double buffering.
2. Semantic-aware state and expert caching reuses recurrent states at tool-call and conversation boundaries while tracking decode-time expert locality through a shared LRU cache.
3. Bandwidth-adaptive CPU--GPU execution partitions residual cache misses between PCIe cache fills and direct CPU execution according to measured bandwidths on the deployed machine.

The resulting system supports more than 20 MoE models and is evaluated on six systems, four agentic workloads, and models ranging from Qwen3.6-35B-A3B to the 753B GLM-5.2. The paper’s principal empirical claim is that this design changes the practical model-capability boundary of local inference rather than merely accelerating a fixed set of small models.

Edge MoE serving as a bandwidth-management problem

FreeToken’s analysis distinguishes prefill and decode because sparsity has different systems consequences in each phase. During prefill, thousands of prompt tokens collectively route to most experts in most layers. For the FP4 deployment of DeepSeek-V4-Flash, approximately 140 GB of expert weights may need to traverse the CPU--GPU path for a prefill. The paper estimates that this adds approximately 2 seconds on PCIe 5.0 x16, approximately 5 seconds on PCIe 4.0 x16, and at least 10 seconds on common laptop x8 links. On edge hardware, exposing this transfer serially makes TTFT dominated by weight movement.

Agentic sessions aggravate this problem through repeated context editing. Tool calls, truncation, removal of reasoning segments, and replacement of old observations invalidate recurrent-state checkpoints and force recomputation. Hybrid-attention and recurrent layers make this particularly costly because their compressed state cannot be reconstructed from an ordinary KV prefix unless an appropriate recurrent checkpoint is available. The paper therefore treats agentic context management as a first-class serving problem rather than as an application-level concern.

Decode exhibits the opposite pattern. Each token routes to few experts, but static placement misses a large fraction of routed accesses because the active working set changes with the sequence and workload. CPU-only execution is also insufficient: expert computation is generally memory-bound, while host DRAM bandwidth is on the order of tens of GB/s, compared with approximately 1--1.8 TB/s of GPU memory bandwidth on the RTX 4090 and RTX 5090. PCIe transfer alone is likewise suboptimal because it leaves CPU-side memory bandwidth unused and does not exploit the possibility of computing an unavoidable miss where it resides.

FreeToken formalizes this choice using two measured quantities: host-to-device expert-transfer bandwidth and effective CPU expert-processing bandwidth. For $m$ missing experts, a subset is filled into the GPU cache and the remainder is executed directly on the CPU. The split is selected by balancing the concurrent PCIe and CPU branch times. Because the parameters are profiled on the target machine rather than inferred from nominal specifications, the policy adapts to differences such as the contrast between an RTX 4060 laptop connected through PCIe 4.0 x8 and an RTX 5090 desktop connected through PCIe 5.0 x16.

This formulation preserves exact model execution. CPU and GPU partial outputs are merged using the original routing and gating computation; FreeToken does not approximate the model, substitute experts, alter the router, or reduce precision beyond the supplied model format. That distinction separates it from systems that reduce bandwidth through expert substitution, reduced-precision replicas, or expert skipping, such as HOBBIT, SiDA, and SMoE (Tang et al., 2024, Du et al., 2023, Zhu et al., 26 Aug 2025).

Prefill codesign and semantic state reuse

FreeToken allocates a two-level expert hierarchy. The complete expert pool resides in pinned host memory and remains the correctness source of truth. GPU memory contains non-expert weights and an elastic cache of complete layer--expert entries. During prefill, two full-layer buffers are drawn from the same global slot pool. While the GPU evaluates layer $l$, a transfer stream loads all experts for layer $l+1$. Loading a full layer before its routing decisions are known is deliberate: because prefill’s aggregate routing is close to dense, on-demand expert fetching would not provide sufficient working-set reduction to justify its synchronization cost.

Figure 1: FreeToken combines double-buffered full-layer prefill, semantic recurrent-state checkpoints, shared LRU expert caching, and bandwidth-adaptive CPU--GPU miss execution.

The evaluation shows that this overlap approaches the PCIe ceiling. On the RTX 5090, an 8,192-token prefill chunk completes in 1.19--1.22 seconds, corresponding to streaming a 64.4 GB expert pool at approximately 52.7 GB/s. At 16,000 tokens, prefill throughput reaches 6.7k tokens/s. Removing the second buffer reduces throughput by 19% at 4k tokens, 25% at 8k tokens, and 26% at 16k tokens. The increasing penalty with prompt length supports the paper’s contention that serialized expert movement exposes a growing fraction of otherwise-overlappable work.

Figure 2: Full-layer pipelining reduces prefill cost, while a global LRU policy lowers decode-time expert misses relative to static placement.

The second prefill mechanism is a semantic-aware recurrent-state cache. FreeToken anchors checkpoints at special-token boundaries associated with reasoning segments, tool calls, tool outputs, and conversation turns. These boundaries are selected because agent frameworks tend to edit complete semantic blocks rather than arbitrary token spans. When a context edit occurs, the runtime restores the deepest surviving recurrent checkpoint and reuses the corresponding attention prefix, recomputing only the new suffix.

This design depends on an explicit workload assumption: agentic context modifications are aligned with semantic boundaries. It is well matched to the evaluated OpenCode, Claude Code, OpenClaw, and SWE-agent-style traces, but the paper does not establish that the same checkpoint placement remains optimal for arbitrary prompt-editing workloads. The limitation is material because recurrent states are large and only a small number of checkpoints can be retained; poor anchor placement can still cause long recomputation.

Decode execution and CUDA-graph integration

During decode, FreeToken maintains a single global LRU cache across all MoE layers. Cache entries are indexed by logical $(\text{layer},\text{expert})$ identifiers and contain the complete tensors needed for execution. This avoids static “hot expert” placement and allows residency to follow the model’s current routing distribution. Hits execute directly on the GPU. Misses are divided between cache fills and CPU execution.

The implementation places routing-dependent control on the GPU to preserve CUDA Graph execution. A device kernel deduplicates routed experts, checks residency, computes the fetch count, selects eviction victims, and rewrites logical expert IDs into physical cache-slot IDs or CPU-assignment flags. A single-pass victim-selection algorithm identifies the least-recently-used candidates without scanning the cache once per eviction. Stable work buffers, valid counts, persistent CPU task descriptors, and captured host-function nodes allow the heterogeneous CPU--GPU step to be replayed without per-token Python scheduling.

This implementation choice is important because a theoretically appropriate CPU--GPU split would lose much of its value if every layer required host synchronization. FreeToken instead turns dynamic scheduling decisions into device-resident data consumed by a statically captured execution graph. The remaining runtime overhead is consequently concentrated in memory movement and expert computation rather than control-plane coordination.

The routing-trace analysis supports the cache design. At the RTX 5090 serving capacities—37% of Qwen3.6-35B-A3B’s expert pool and 11% of DeepSeek-V4-Flash’s—FreeToken’s LRU cache misses 16% and 39% of decode-time expert reads, respectively. KTransformers’ prefill-updated placement misses 41% and 59%, while llama.cpp’s routing-blind static split misses 62% and 89%. The ordering persists across the four workloads and across cache capacities short of the full pool. The implication is that token-level recency is more predictive of useful residency than placement determined from a prior prefill or from workload-agnostic partitioning.

End-to-end evaluation

The main evaluation uses Qwen3.6-35B-A3B in BF16 and DeepSeek-V4-Flash in native MXFP4 on an RTX 5090. The workloads include AIME reasoning, OpenCode with SWE-bench issues, Claude Code with concurrent subagents and 56--65k-token sessions, and OpenClaw email/calendar interaction over 13 turns. Baselines include llama.cpp, Ollama, KTransformers, and MoE-Infinity under supported configurations.

Figure 3: FreeToken’s decode throughput and TTFT across mathematical reasoning and increasingly interactive agent workloads on the RTX 5090.

FreeToken sustains 77--83 tokens/s on Qwen3.6-35B-A3B and 22--25 tokens/s on DeepSeek-V4-Flash. Relative to the strongest baseline in each workload, these correspond to speedups of 1.8--2.3x and 1.5--1.9x, respectively. Decode throughput remains within 12% of the single-turn AIME setting across the three agent workloads. This stability is a significant result because it contradicts the assumption that single-turn decode benchmarks adequately represent agent serving. For example, KTransformers on DeepSeek-V4-Flash loses 31% of its single-turn rate already on the OpenCode workload.

MoE-Infinity serves only the single-turn AIME workload at 8.8 tokens/s in this comparison. Its per-expert prefill staging limit prevents the longer-prompt workloads, and its bundled server does not retain KV state across requests. These failures are not merely lower throughput results: they demonstrate that agent compatibility, state reuse, and long-context execution are necessary conditions for practical deployment.

TTFT shows an even larger separation. FreeToken has the lowest mean TTFT in five of six multi-turn model--workload combinations; the exception is Qwen3.6-35B-A3B with Claude Code, where KTransformers’ GPU-prefill path is faster. More consequentially, FreeToken’s worst turn remains below 44 seconds in every evaluated cell. Every baseline exceeds 150 seconds in at least one setting, with observed maxima of 232 seconds for llama.cpp, 179 seconds for Ollama, and 946 seconds for KTransformers. These values cross application-level availability thresholds: OpenClaw uses a 120-second idle watchdog, while Claude Code has a default request timeout of roughly ten minutes. The result establishes that tail TTFT is not only a performance metric; in interactive agents it determines whether a serving engine completes a request at all.

Cross-hardware capability boundary

The cross-hardware experiments evaluate Qwen3.6-35B-A3B across RTX 3090, RTX 4090, RTX 5090 server, RTX 5090 desktop, and RTX 4060 laptop systems. FreeToken outperforms the strongest baseline by 1.3x on the RTX 3090, 1.3x on the RTX 4090, 1.9x on the RTX 5090 server, 2.1x on the RTX 5090 desktop, and 1.8x on the RTX 4060 laptop.

The laptop result is particularly strong: an 8 GB RTX 4060 serves the NVFP4 model at 39.3 tokens/s, exceeding the paper’s cited 33 tokens/s median decode rate for Codex production traces. The result is not attributable solely to GPU compute. The laptop has only PCIe 4.0 x8 bandwidth, so the measured bandwidth-adaptive split is essential to avoid making either the GPU or CPU path exclusively responsible for misses.

The comparison between the two RTX 5090 systems isolates host effects. Moving from the multi-channel server to the dual-channel consumer desktop reduces FreeToken’s decode rate by only 4%. In contrast, llama.cpp retains 80% of its server rate because its CPU-resident experts become constrained by the desktop’s two DDR5 channels. This result supports the paper’s stronger systems claim: a runtime that jointly schedules PCIe and CPU execution can be less sensitive to host-memory limitations than a runtime that assigns misses statically to the CPU.

Figure 4: FreeToken maintains a throughput advantage across consumer GPUs and serves GLM-5.2 on a single RTX PRO 6000 workstation GPU.

At the workstation tier, FreeToken serves the 753B-parameter, 40B-active GLM-5.2 on one RTX PRO 6000 with 96 GB VRAM at 14.9 tokens/s, compared with 7.3 tokens/s for llama.cpp, a 2.0x improvement. Mean TTFT is comparable—7.5 seconds versus 7.8 seconds—so the gain is primarily in decode execution rather than a tradeoff against first-token latency. KTransformers has no servable configuration for this model because its GLM-5.2 methods require 753 GB--1.5 TB of host memory and its CPU kernels do not support the model’s NVFP4 layout. The experiment therefore demonstrates both performance improvement and broader model deployability.

Limitations and open questions

The evaluation is broad for a single serving system but remains bounded. It covers six machines, with three server platforms emulating constrained CPU-thread and NUMA conditions rather than representing ordinary consumer systems directly. The paper validates the emulation using one desktop and one laptop, but the diversity of PCIe topologies, operating systems, memory pressure patterns, and CPU architectures remains larger than the tested set.

The $q^\star$ policy also relies on bandwidth measurements that are sufficiently stable to predict the relative cost of CPU execution and PCIe fills. The paper profiles these quantities at deployment and adapts cache capacity at safe points, but it does not fully characterize performance under rapidly varying contention from games, browsers, other GPU applications, or simultaneous inference requests. Whether the closed-form split remains optimal under concurrent multi-request batching, nonstationary DRAM contention, or thermal throttling is left open.

The semantic state cache depends on the regularity of agent context edits and gives checkpoints only a small budget. The experiments establish strong benefits for the selected agent harnesses, but do not quantify sensitivity to anchor density, checkpoint eviction strategy, heterogeneous agent protocols, or edits that cut through semantic blocks. Similarly, the paper reports end-to-end throughput and TTFT but does not provide a complete quality evaluation under all supported quantized formats and models. Exact execution is preserved relative to the loaded weights, yet numerical parity across every model backend and CPU SIMD implementation is not exhaustively analyzed.

Finally, the paper reports per-request mean throughput and TTFT rather than cross-engine wall-clock completion time because agent trajectories diverge. This is methodologically defensible, but it leaves open how FreeToken compares on aggregate task completion time, energy per solved task, and cost under realistic concurrent workloads where request scheduling and trajectory length interact with serving speed.

Conclusion

FreeToken presents a coherent edge-serving architecture for MoE models whose sparse active computation fits within consumer GPU resources while their complete expert pools do not. Its main technical contribution is the joint treatment of prefill transfer, recurrent-state reuse, decode-time expert locality, CPU--GPU miss execution, and elastic memory management. The reported results—up to 2.3x decode speedup on the RTX 5090, sub-44-second worst-case TTFT across evaluated workloads, 39.3 tokens/s for a 35B model on an 8 GB laptop GPU, and 14.9 tokens/s for a 753B model on one RTX PRO 6000—show that serving software materially determines which MoE models are practical on local hardware. The unresolved questions concern robustness under broader contention, workload diversity, batching, and agent state-edit patterns rather than the feasibility of the demonstrated deployment model (2608.16157).