What I Found in My Test
I took on a pretty down-to-earth task: I gave the models API documentation where I had intentionally planted several errors and watched to see which one could catch them without hallucinating. I then ran the answers through GPT acting as a judge. It's not a perfect benchmark, but for a quick, practical comparison, it's a solid method.
In my test, there were two contenders. On one side, a local Gemma 4 31B IT; on the other, Claude Sonnet 4.6 Extended Thinking. The final scores paint a clear picture: the first response got a 4/10, the second a 7.5/10.
There's a crucial detail here: my local Gemma setup wasn't a single variant. I tested Gemma 4 31B IT in MLX 4-bit and in Ollama 4-bit separately. And this is where the hardware and backend have just as much impact as the model itself.
Where the Infrastructure, Not the Answer, Really Grabbed My Attention
The difference in memory usage was almost staggering. Ollama ate up about 43 GB on my machine, while MLX on an M4 showed a peak memory of 19.994 GB. For the same idea of running a 31B model locally, this isn't just cosmetic; it's the difference between 'runs smoothly on the machine' and 'the machine starts to suffer'.
I love these moments because they directly impact AI architecture. On paper, you have a 'local open model,' but in practice, one stack fits within a reasonable unified memory limit, while the other turns your laptop into a space heater. If you're building an AI integration for a team, this is no longer a matter of taste but of total cost of ownership.
MLX on Apple Silicon currently looks significantly more mature for such tasks. Not because of magic, but because the stack is closer to the metal and loses less to overhead. When you can keep a 31B model running locally at around 20 GB, the conversation about private pipelines, internal code reviews, and offline documentation checks becomes tangible.
In Terms of Quality, Gemma Isn't There Yet, But It's No Longer a Toy
I'd describe the responses this way: Claude is better at maintaining the review structure, more confident in separating real defects from minor comments, and is less likely to miss the mark on priorities. In my test, Gemma 4 31B was helpful but felt a bit raw specifically as a documentation reviewer. It didn't fall apart, but it also didn't show the level of reliability you'd need to confidently hang a critical workflow on it.
Still, it's too early to write off local models. While a couple of years ago, a run like this was more of an enthusiast's hobby, now it's a viable starting point for AI automation in a private environment. This is especially true where you can't expose internal APIs, integration schemas, or proprietary documentation to the cloud.
I'll put it bluntly: Claude wins on quality right now, MLX wins on the economics of local deployment, and Gemma 4 31B is at a point where it shouldn't be discussed in a vacuum but integrated into real chains to see the results.
Who Benefits From This Right Now
The biggest winners are teams with a lot of routine engineering checks: API docs, SDK guides, changelogs, internal policies, migration notes. There, you can build multi-pass AI automation: a local model finds obvious inconsistencies, and a powerful cloud model handles the complex or ambiguous cases.
Those who are waiting for a universal silver bullet will lose out. If you just grab a local model, throw documentation at it, and expect a senior-level review, you'll be disappointed. You need solid prompts, validation stages, sometimes a judge model, and sometimes rule-based checks on top of the text.
At Nahornyi AI Lab, this is exactly what we build: not just 'here's a chatbot,' but an architecture of AI solutions tailored to your process. A local model handles privacy and a cheap first pass, while a cloud model is brought in only where its quality truly justifies the cost. This is how AI implementation stops being a toy and starts saving your team's time.
My Unvarnished Conclusion
If I need the best result on an API documentation review today, I'm choosing Claude. If I need a controlled, local environment on Apple Silicon, I'm looking very seriously at Gemma 4 31B through MLX, not through a heavy backend with an excessive appetite for memory.
I, Vadym Nahornyi from Nahornyi AI Lab, don't run these comparisons just for charts. I do it to build real-world AI solutions for businesses. If you want to discuss your use case, order AI automation, create an AI agent, or build an n8n scenario with local and cloud models, get in touch. We'll figure out what's actually worth running locally and what's better left to an API.