Technical Context
I regularly see the same 'strawberry test': how many letters in a word, how many r's, why the model spouts nonsense. And yes, for prompt engineering this is a useful test — not of the model's intelligence, but of understanding its architecture and how you build AI automation around it.
The problem is very down-to-earth: an LLM doesn't read a word as a sequence of characters. It receives tokens, usually chunks of words assembled by a tokenizer like BPE. For it, strawberry doesn't have to exist as s-t-r-a-w-b-e-r-r-y. Often it's 1–2 tokens with the internal structure already 'packaged'.
In such cases I immediately slow down teams that want to measure a model's 'smartness' with everyday tasks. Counting letters, character positions, precise masks, simple string operations without a tool often break even large models. This isn't a bug like 'training fell short'—it's a fundamental limitation of text representation.
There's a nuance: sometimes the model answers better in English, sometimes worse, but the essence doesn't change. Language affects tokenization, it doesn't eliminate it. If a task requires character-level accuracy, asking the model to 'think again' is almost always worse than asking it to write and run a short script.
That's where proper AI integration starts: I don't force the LLM to count what an interpreter should count. I give the model the role of orchestrator. Let it generate Python, SQL, or JS, and let the execution environment handle the precise operation.
What This Changes for Business and Automation
The practical takeaway is simple: don't route string, arithmetic, and verifiable operations directly through the model's answer. Otherwise you'll get nice-sounding text with randomness inside.
The teams that win are those building AI solutions for business as a 'LLM + tools' combo, not as a magic chat. Those who try to stuff reasoning and precise execution into a single prompt lose.
I constantly see this pattern with clients when processing emails, contracts, catalogs, and support logs. As soon as you move exact steps into code, quality shoots up and the cost of errors drops. If you have a similar story and need not just another chatbot but coherent automation with AI, at Nahornyi AI Lab we can build such a loop for your process without unnecessary magic and with proper result verifiability.