LLM on ESP32-S3 for $8: The Memory Trick

What Was Actually Run on the ESP32-S3

The main fact here is not the meme-worthy $8, but that a 28.9M-parameter language model is described as fully local on the ESP32-S3. The slvDev/esp32-ai repository on GitHub claims about 9.5 tokens per second end-to-end and 9.7 tokens per second for pure compute.

The hardware isn't abstract either: an ESP32-S3 N16R8 is specified with 16MB flash, 8MB PSRAM, and 512KB SRAM. For a microcontroller, this is still a very tight box, so the usual approach of loading the entire model into fast memory wouldn't have worked here.

The trick is called Per-Layer Embeddings, PLE. Essentially, about 25 million of the 28.9M parameters sit in flash as a large lookup table, while the compute core, activations, and temporary structures remain in SRAM and PSRAM. It's not magic but a careful trade-off: more accesses to slow memory, but the model fits at all.

What I like about this story is precisely the engineering roughness of the solution. There was no attempt to pretend that the ESP32 suddenly became a little server. Instead, the architecture was tailored to the real memory hierarchy of the microcontroller: flash for the bulk of parameters, PSRAM for buffers, SRAM for the hottest path.

There's an important caveat: the model is trained on TinyStories. So it generates short, simple stories—not factual QA, instruction following, or a general-purpose conversational assistant.

Why This Changes Edge AI, But Without Fantasies

This is a real step for local AI on IoT, but not a replacement for large LLMs. As of July 30, 2026, I read this project as a demonstration of memory and architecture, not as the arrival of a smart assistant in every sensor.

The practical benefit is clear: the device can generate text without the cloud, without network latency, and without sending data out. For small narrative interfaces, toys, offline demos, and private edge scenarios, this is already enough if the domain is narrow.

Where it will break is also immediately visible. Flash accesses, limited context, narrow training on TinyStories, and weak general instruction-following ability will quickly hit a ceiling. I'd look first not at tokens per second, but at the quality of degradation outside the training domain.

But the fact itself is uncomfortable for old intuition: LLM no longer necessarily means cloud, GPU, and a bulky board. Sometimes it's a lookup table in flash and a small core that does exactly as much intelligence as the physics of memory allows.

We previously discussed how AI demos on Raspberry Pi often remain myths due to a lack of well-thought-out architecture. This case with an $8 microcontroller shows that with the right approach, even minimal hardware can achieve real results.