Technical Context
I view this user insight without any romanticism about the model's "long memory". The reality is simple: as soon as a team gets a window of around 1M tokens, they start treating it like infinite space, resulting in limits and budgets vanishing faster than expected.
I regularly see the same mistake in production: developers estimate context by percentage or visually by dialogue length, rather than by actual token count. In practice, system instructions, message history, RAG snippets, service fields, repeated templates, and sometimes multimodal data are already embedded. As a result, a "compact" request becomes heavily bloated.
Under API pricing models, providers charge linearly for input and output, but the computational cost of long context is felt non-linearly. I've analyzed such scenarios on GPT-5, Claude 4, and Gemini: closer to the upper limits of the window, latency spikes, answer steerability drops, and the "context rot" effect emerges—where the middle of the context is processed worse than the beginning and end.
That is why manual cleaning and running compact functions are not workarounds, but a rational engineering response. If the chat history isn't compressed, every new call drags along all the accumulated garbage. This hurts not just the cost, but the quality as well.
Impact on Business and Automation
For a business, this isn't an academic problem. If I am designing AI automation for a sales department, customer support, or an internal knowledge assistant, undisciplined long context almost always turns into a hidden tax on scalability.
Teams that treat tokens as an infrastructural resource rather than an abstraction win. Those who try to compensate for weak AI architecture by merely expanding the context window lose.
In Nahornyi AI Lab projects, I usually build in multiple layers of defense: strict token budgets, rule-based history clearing, intermediate summarization, semantic caching, and targeted retrieval instead of "loading everything into one prompt". This reduces costs and makes system behavior predictable.
To put it bluntly, a 1M context rarely saves bad AI architecture. More often, it masks the problem initially, leaving the company with an expensive, slow, and unstable system later. Therefore, AI implementation should start not with selecting the maximum window, but with designing the data pipeline.
Strategic Perspective and Deep Dive
My conclusion is this: the market has overestimated the sheer fact of a large context window. I won't argue that 1M is useful in specific scenarios—like auditing lengthy documents, complex correspondence analytics, or legal review of massive archives. But for most operational workflows, it is an emergency mode, not the working norm.
I increasingly recommend that clients calculate their MECW (Maximum Effective Context Window)—the actual effective window for a specific process—rather than looking at the max possible. In some cases, it's 16K; in others, 64K or 128K. I only activate anything beyond that after measuring cost, latency, and accuracy on real data.
From Nahornyi AI Lab's practice, I see a clear pattern: when a team implements compacting, context ranking, and phased prompt assembly early on, the solution's economics improve dramatically. When they don't, costs skyrocket before the benefits of AI implementation are even realized.
I see the next stage of market maturity like this: the winners won't be models with the biggest windows, but companies with the best context management logic. Victory belongs not to memory size, but to AI architecture where every token is justified.
This analysis was prepared by Vadym Nahornyi — lead expert at Nahornyi AI Lab on AI architecture, AI automation, and practical AI integration into business processes. If you want to build AI automation without token budget leaks, I invite you to discuss your project with me and the Nahornyi AI Lab team. We will design an AI integration that delivers results, not a bill for unnecessary context.