Local Qwen: what 500 tps on an M1 Ultra really means

What the user benchmarks actually show

I would treat these figures as the output of a specific software stack, not as the M1 Ultra's headline speed. In the user discussion, the reported result is roughly 500 tps during prefill and up to 100 tps for code generation on an M1 Ultra with 64 GPU cores, MTPLX and Qwen3.8-Flash-Next. For local inference, that is a strong reference point, but only for this exact configuration.

The comparison with Nvidia is still incomplete. Another participant runs Qwen3.6 27B on an A6000, but no throughput for that setup is provided. This is not an Apple-versus-Nvidia test; it is two separate observations. Without the same model, quantization, prompt length, batch size and parallelism mode, an impressive number can easily mislead.

The mlx-serve documentation lists Qwen3.6-27B and Qwen3.8-27B among supported models, including MLX packages with speculative-decoding heads. oMLX release notes for the M3 Ultra report around 91 tps without MTP and 134 tps with MTP; after expert-offloading optimization, 32K prefill reached about 636 tps. That makes the reported range plausible, but it does not transfer newer Apple-chip results directly to the M1 Ultra.

The stack matters at least as much as the hardware. mlx-serve provides an MLX-focused server layer, oMLX adds model management and dedicated accelerations, while Strata primarily targets CUDA systems, including the A6000. As of November 8, 2026, these figures are better understood as configuration-specific outcomes than as universal device specifications.

Where local inference genuinely changes the equation

The practical benefit is real: private data can be processed locally without paying an external provider for every token. This works especially well for sequential workloads serving one user or a small number of users.

• Transcription and analytics: call recordings, transcripts, fact extraction and structured summaries can remain within the local environment.
• Medical documents: a model can normalize terminology and prepare drafts, but its output still requires clinical review.
• Agents and coding: high single-stream speed can make a local assistant feel responsive, yet it says nothing about performance under dozens of simultaneous requests.

For Apple Silicon, I would first test memory usage and long-prefill stability. For the A6000, batching behavior, concurrent requests and the quality cost of quantization matter more. The key unanswered question is not whether 100 tps is possible, but how much of it remains under real workload conditions.

We previously covered Rust LocalGPT, a local assistant in a single binary with memory and an HTTP API. This case adds practical context to Qwen and Hermes benchmarks for local inference.