Technical Context
I got hooked on a very practical question: can you build a local DM assistant on Gemma 4 that generates quests, remembers a long session, and doesn't require the cloud? For this kind of AI implementation, my answer is simple: yes, but not by brute-forcing the entire history into the context.
From what I see in benchmarks and discussions, Gemma 4 26B-A4B and 31B are already running in llama.cpp on RTX 3050/3060, especially with quantization. But there's no magic: even if the MoE only activates about 4B parameters per token, the model is still heavy in memory, and a long context starts to choke the hardware.
On a 3060 with 12 GB, I'd look at a heavily compressed 26B-A4B or even smaller E2B/E4B models for a stable local scenario. On a 3050 with 8 GB, you'll have to carefully manage expectations: speed drops, some of the load spills over to RAM, and long prompts cause the very freezes that users complain about.
And this is where the popular idea of "let's just give it a 128K or 256K context" falls apart for me. It looks great on paper. In a real DnD session or any long-running game, the model either starts forgetting important details or wastes too much compute re-processing the entire history.
I would implement memory more simply. Not a full-blown agentic search for every little thing, but an external structure tailored to the specific use case: Markdown files, SQLite, an append-only event log, plus short summaries after each session. I would feed the model not the whole world, but 5-15 key facts about characters, the current arc, active quests, and the latest state changes.
If search is needed, a local FAISS or HNSW index over notes already solves half the problem. For a truly budget-friendly mode, you can even live without classic RAG by using injection rules: who is important, what has changed, and what plot points must not be broken.
What This Means for Business and Automation
My main takeaway is this: agentic search is smarter, but it's not always justified on weak hardware. For local products and AI automation on budget PCs, a dumber but more predictable memory architecture often wins.
The winners are those who design an assistant for the task, not for the hype around a long context. The losers are the teams that try to replace architecture with one big token window.
I regularly build these kinds of compromises for clients too: determining where structured memory is sufficient, where RAG is needed, and where it's genuinely time to build an AI integration with agents and tools. If you have a similar story, and your local assistant needs to work fast, be stable, and have no cloud dependency, let's break down your scenario at Nahornyi AI Lab and build an AI solution without excess compute or decorative complexity.