FreeToken runs frontier-scale MoE models on consumer GPUs
FreeToken, a new open-source serving system for Mixture-of-Experts models, is pushing a class of AI workloads that normally belongs in datacenters onto consumer and workstation hardware. In a paper submitted to arXiv, researchers from Berkeley and collaborators describe an edge-native runtime that combines GPU memory, host memory, CPU execution and PCIe bandwidth as one elastic inference platform rather than treating local inference as a smaller version of GPU-only serving.
The central claim is not simply that very large models can technically be loaded on a personal machine. FreeToken is designed to make them usable under real agentic workloads, where long prompts, repeated tool calls and changing contexts stress a serving stack differently from a single-turn benchmark. The authors evaluate the system with coding agents, Claude Code, OpenCode, OpenClaw and math reasoning workloads, and they compare it with llama.cpp, Ollama, KTransformers and MoE-Infinity.
Mixture-of-Experts models create the opportunity. A model may contain hundreds of billions of total parameters while activating only a small subset of experts for each token. DeepSeek-V4-Flash, for example, has a much larger full expert pool than the active parameter footprint used for one token. That makes computation sparse enough to fit the active work on a powerful consumer GPU, but the full expert weights can still exceed VRAM by a wide margin. The systems problem is therefore how to move, cache or execute the missing experts without turning memory traffic into an unusable bottleneck.
FreeToken attacks that problem with bandwidth-adaptive execution. During prefill, it double-buffers expert movement so the next layer's experts can travel over PCIe while the GPU is computing the current layer. During decode, the runtime dynamically divides cache misses between two paths: transferring experts into GPU memory and executing some expert work directly on the CPU. The split is calibrated to the actual PCIe and host-memory bandwidth of the machine rather than fixed in advance.
The system also adds semantic-aware caching for agent workloads. Tool calls and reasoning systems often modify context at meaningful boundaries instead of appending a perfectly linear conversation forever. FreeToken stores recurrent-state checkpoints around those boundaries so it can recompute only the changed suffix after an edit. For decode, it uses a shared LRU expert cache that exploits the tendency of adjacent tokens to route to overlapping experts. Together, those mechanisms are meant to reduce both repeated prefill work and expert movement.
A third piece is elastic memory management. Consumer GPUs are rarely dedicated inference appliances, so available VRAM can change while browsers, development tools or other applications are running. FreeToken can resize its GPU expert cache and rebalance memory without restarting the engine or reloading the host-resident expert pool. The released repository describes support for NVIDIA RTX 30, 40 and 50 series GPUs, OpenAI- and Anthropic-compatible APIs, and more than 20 supported MoE models.
The paper's headline results are substantial. On an RTX 5090, the authors report 77 to 83 tokens per second for Qwen3.6-35B-A3B and 22 to 25 tokens per second for DeepSeek-V4-Flash across four workloads. They report 1.8 to 2.3 times the strongest baseline throughput for Qwen3.6 and 1.5 to 1.9 times for DeepSeek-V4-Flash. Under increasingly agentic workloads, FreeToken's decode rate stays within 12 percent of its single-turn result, while some competing systems degrade more sharply.
Latency is another important part of the evaluation. The authors report that FreeToken keeps its worst observed time to first token below 44 seconds in every tested workload cell, while each baseline crosses 150 seconds in at least one case. That distinction matters for agent software because a very slow prefill can stop being a performance annoyance and become a timeout or reliability problem. The paper explicitly frames tail latency as an availability boundary for real clients.
The cross-hardware results are what make FreeToken especially interesting for local AI. The authors report serving a 35B model at 39.3 tokens per second on an 8 GB RTX 4060 laptop. On a 32 GB RTX 5090 gaming desktop, the system serves the 284B-parameter DeepSeek-V4-Flash interactively. On a workstation with a single 96 GB RTX PRO 6000, it serves the 753B GLM-5.2 model at 14.9 tokens per second, roughly twice the throughput the paper reports for llama.cpp on the same tier.
Those numbers should still be read as research results, not settled independent benchmarks. The evaluation is produced by the FreeToken authors, and the paper notes that some competing engines cannot serve every model or multi-turn workload in the test matrix. That is itself relevant to deployability, but it also means not every comparison is a symmetric race between fully supported configurations. Independent reproduction across more consumer machines, operating systems and model versions will be important before treating the reported gains as general performance guarantees.
Even with that caveat, FreeToken represents a meaningful systems shift. Open-weight model releases have made frontier-scale parameters obtainable, but practical access has remained constrained by memory capacity, bandwidth and serving software. A runtime that can treat CPU, GPU, host memory and interconnect bandwidth as a coordinated resource pool changes the economics of experimentation for developers and small teams, particularly for workloads that would otherwise require large hosted inference bills or dedicated server GPUs.
The project is also immediately inspectable rather than only described in a paper. The FreeToken GitHub repository is released under Apache 2.0, includes installation and model documentation, and exposes a CLI and compatible API server. That makes the next stage straightforward to evaluate: whether outside developers can reproduce the paper's performance and whether the runtime remains stable under the messy memory pressure and long-lived agent sessions found on real personal machines.
## Sources
Published: