FreeToken speeds up local LLMs compared with Ollama

What FreeToken actually accelerates

What caught my attention were the claimed 3–4× gains in decode speed and 6–30× gains in prefill versus Ollama. The FlashML team reports these results in the FreeToken paper, arXiv 2608.16157, and in the project documentation. These are project-published benchmarks at the time of writing, not my independent test: the actual gain depends on the model, hardware and workload shape.

FreeToken's main idea is not just another fast kernel. The system treats the CPU, GPU and system memory as one elastic platform for running large MoE models that do not fit entirely in VRAM. During execution, it measures CPU–GPU bandwidth over PCIe as well as processing speed on the CPU.

The runtime then decides what is more efficient for each expert: keep it in the GPU cache or execute it directly on the CPU. Static partitioning loses here because transfer and compute costs vary greatly between a laptop, desktop and workstation. That looks like a systems optimization rather than a flattering number from one lucky run.

The second lever is semantic-aware caching between agent turns. When requests are related and change gradually, FreeToken reuses useful state, keeps frequently needed experts warm and reduces repeated prompt processing. That is why the prefill gain is substantially larger than the gain in sequential token generation.

The project summary also claims 1.5–2.3× faster decoding than the strongest compared edge systems on an RTX 5090. One illustrative scenario runs a 35B-parameter model using a laptop GPU with 8 GB of memory.

Why this changes local inference

FreeToken looks less like a cosmetic speedup and more like an attempt to fix the core architecture of the local serving stack. Machines with limited VRAM but enough system RAM should benefit most, as should agent workflows with long sequences of related requests.

I would first examine cold starts, tail latency, RAM pressure and cache behavior after an abrupt topic shift. Semantic reuse works exceptionally well when repetition is real; with unrelated requests, its contribution will inevitably be smaller. Likewise, CPU–GPU balancing can be constrained by a specific PCIe link or a weak processor.

From an engineering perspective, the idea is sound: measure the actual machine first, then schedule execution instead of imposing the same layout everywhere. The key open question is no longer whether FreeToken is faster in published tests, but how consistently that advantage survives across models, hardware and real agent sessions.

We previously covered how confidential compute infrastructure can reshape LLM inference costs and privacy trade-offs. FreeToken adds the performance side of that equation by improving decoding throughput through CPU-GPU bandwidth adaptation.