What makes this experiment so compelling
I appreciate tests like this not for the hype, but for their uncomfortable honesty. Researchers took about 80 programming tasks and gave them to models in two wrappers: standard Python and Turing-complete esoteric languages. With Python, the success rate was around 85-95%. With the esoteric languages, it was a dismal 0-11%.
This wasn't shocking to me. On the contrary, these numbers just highlighted what I already see in my applied work with code generation. A model often doesn't "understand the task" in an abstract sense—it confidently follows the familiar tracks of syntax, patterns, and common solutions.
In short, Turing completeness itself isn't a silver bullet. A language might formally be as capable as Python, but that means nothing to the model in terms of skill transfer. This is the unpleasant part of the story about sample-efficient transfer—knowledge transfer in transformers is still quite fragile.
The source here isn't academic but rather an internet-based research dive around the idea of The Illusion of Thinking. It's important to be clear about this: I wouldn't present this as a final verdict on the entire architecture. But as a stress test for abstraction, it's very telling.
Why Python deceives us more than we'd like
I've seen it many times: a team looks at a high success rate on Python benchmarks and draws an overly bold conclusion: "Well, the model can program now." No, it excels at statistical navigation within a familiar ecosystem. This is useful, but it's not synonymous with thinking.
Python is the perfect greenhouse for LLMs. It has a massive corpus of code, predictable idioms, countless similar tasks, and a sea of documentation and discussions. The model isn't just strong there—it's at home.
But when I pull it out of its comfort zone and ask it to do something with an unusual DSL, an old internal rules engine, or a convoluted config format, the magic quickly fades. It's not because the task is fundamentally harder, but because the crutch of familiar patterns is gone. To me, this is much closer to real life than another polished Python demo.
What this means for business and automation
The business takeaway is very practical: don't confuse a successful copilot in a familiar stack with a universal reasoning engine. If your AI implementation relies on non-standard processes, internal DSLs, legacy systems, or rare data formats, the risk of underestimating the model's fragility is quite high.
I would frame the rule like this: the further your environment is from the mainstream internet, the less you should trust the "average benchmark score." You need your own evaluation sets, your own transfer tests, and your own constraints on the agent's autonomy. Otherwise, your AI automation will look great in the pilot and disappoint in production.
Who wins? Teams that build AI architecture with checks, intermediate representations, compilation into controlled steps, and strict result validation. Who loses? Those who try to just bolt a model onto a rare domain-specific language and hope it will "figure it out."
At Nahornyi AI Lab, we encounter this regularly when developing AI solutions for internal processes, not just for flashy demos. If the domain is narrow, I almost always build in a normalization layer: first, translate the task into a more stable representation, then generate, then automatically verify. It's not as romantic, but it works.
My conclusion, without the drama
I wouldn't shout that "models can't think at all." That's too cheap a formula. But I would definitely rephrase it: their ability to transfer knowledge is highly overrated, especially when we only look at familiar languages and convenient benchmarks.
If you're planning to implement artificial intelligence in your code, ops, or internal tools, keep a simple question in mind: is the model solving my problem, or is it just recognizing a familiar problem shape? The difference between these two scenarios later translates into either months of saved time or a very expensive illusion.
This analysis was written by me, Vadim Nahornyi, at Nahornyi AI Lab. I build hands-on AI integrations, agentic pipelines, and AI-powered automation in environments with legacy systems, non-standard data, and high quality-demands. If you'd like, I can help you soberly assess your use case and figure out where a model's real strength lies and where it's just a confident facade.