FreeToken Runs Qwen3.6-35B-A3B on an 8 GB RTX 4060

How FreeToken works around the VRAM limit

The key point for me is this: FreeToken turns the GPU, CPU, and RAM into a shared compute pool, allowing huge Mixture-of-Experts models to run without forcing the entire model into video memory. That is how FlashML-org describes its experimental engine in the official FreeToken repository and the paper “Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution”.

The mechanism is more sophisticated than simply offloading layers to system memory. At startup, the engine measures actual bandwidth between the GPU and CPU as well as the processor’s compute capacity. It then decides which experts should remain on the GPU, which can reside in RAM, and which workloads are cheaper to execute on the CPU than shuttle across PCIe.

Frequently used experts get fast GPU access, while less common ones do not permanently occupy scarce VRAM. This is a natural fit for MoE: only a subset of experts is activated for each token, so storing the full model in video memory is not essential. That architectural trait becomes a practical way to run models that are formally much larger than the available VRAM.

At publication in August 2026, the authors reported the following results:

• Qwen3.6-35B-A3B on a laptop with an 8 GB mobile RTX 4060: 39.3 tokens/s.
• DeepSeek-V4-Flash 284B on an RTX 5090: 22–25 tokens/s.
• GLM-5.2 753B on a single RTX PRO 6000: 14.9 tokens/s.

These are figures from the paper, not my own benchmarks. The evaluation covered three models, six machines, and four agentic scenarios, comparing FreeToken with llama.cpp, Ollama, KTransformers, and MoE-Infinity. The scope is meaningful, but the claimed speeds still cannot be automatically applied to every prompt, memory configuration, or expert-routing pattern.

Local AI is no longer limited by VRAM alone

This is a genuine engineering shift: VRAM capacity is no longer the only hard barrier for local MoE models. The Qwen3.6-35B-A3B result is particularly revealing because interactive throughput was achieved on an 8 GB laptop RTX 4060 rather than a multi-GPU server.

Memory has not disappeared as a constraint; it has become a connected system. RAM capacity, PCIe bandwidth, CPU speed, and how activity is distributed among experts now matter greatly. On a poorly balanced machine, an adaptive scheduler may hit data-transfer limits long before reaching the attractive numbers in a benchmark table.

I would focus first on time to first token, sustained throughput in long sessions, and behavior under agent workloads. Average tokens per second shows potential, but it does not reveal pauses during expert switches or pressure on system memory.

FreeToken does not eliminate hardware constraints. It does something more interesting: it stops treating the GPU as the only place where local inference can live.

We previously covered Pony Alpha on OpenRouter: free access to a model with a 200K context window for prototyping and architecture validation. This case complements FreeToken by showing another way to experiment with large models without costly infrastructure.