Your AI is overwhelmed, and confused
Pile on context and it forgets; pile on rules and it freezes. The most expensive mistake is reaching for a bigger model when you have an architecture problem.
When a team discovers that their carefully engineered AI product performs worse than the plain model underneath it, the instinct is to upgrade to a more expensive model.
Yet the product probably isn’t underperforming because the model is bad. It’s underperforming because someone stuffed the desk or shouted too many instructions. Those are two different problems — and I know because I caused both.
My first encounter started when I built an agent to scan the news. Setting it up meant briefing the agent the way you’d brief a sharp new hire who happens to know nothing about you. I gave it my preferences, the angles I care about, the stories to skip. By the time I finished I’d handed it about nine pages of instructions.
As the pages stacked up, I hit a wall: the information was all there, but when I asked the agent to do something with it — pull a particular subset of articles, say — it would quietly skip or omit results.
My AI was overwhelmed and confused, and I had caused both.
Piling the information on
A widely known principle of working with AI over the past few years is that AIs have a fixed attention budget — a context window — and every token draws it down. While this originally applied in the sense of ‘reduce long chat windows’, a whole discipline of curating what a model sees in a multi-threaded agentic interaction has arisen: context engineering.
When an AI’s attention budget runs out, there’s a name for the particular way it loses the plot, “lost in the middle.” The model prioritizes what sits at the beginning and end of the window, the way your eye slides down a contract to find the signature line. What sits in the center gets skimmed or skipped entirely.
Picture a mid-career white-collar worker in grey slacks and suspenders and horn-rimmed glasses, sitting behind a desk. The worker is the AI, and they look up as you walk in with a stack of books. You are working on an important project, you tell him, and you stack ream after ream on the desk until it’s holding the weight of a small library.
Now you ask the worker to keep all of that in mind while helping you on your project.
What struck me about this is that the AI suffers a very human-like response. It starts forgetting. Intuitively, this shouldn’t happen: an AI has theoretically unlimited potential to store and recall data. Practically, it’s a function of space and reasoning constraints that limit the model. The desk has edges.
That describes the attention budget, but the interesting thing is expanding the desk (upgrading to a bigger model) doesn’t resolve the issue. More desk space without curation is like a teenager’s room — it doesn’t matter how big the room is; the mess fills to the size of the room.
There’s a bill attached to the mess, and it’s easy to miss. In a single chat, you pay for that context once. In an agentic loop, you pay for it on every turn — the model re-reads your instructions from scratch. My nine pages, on a twenty-step run, were a hundred and eighty pages — re-read and re-billed, in service of results the agent was quietly dropping anyway. And the upgrade path makes that worse before it makes it better: a more expensive model raises the per-token price of exactly the material you should have deleted.
Shouting at the help
Context overload is relatively well understood by now, at least among builders. The second failure I encountered receives less attention, and it may be more costly. This was a failure of rules.
In my nine pages of information, I had structured, with good intention, a number of guidelines for the model to follow. Prioritize your search by XYZ sources. Search only for articles in the trailing 90 days. Do this instead if you encounter topics that aren’t bounded by the core search areas. And so on.
Go back to the worker behind the desk, and take off the stack of books. Instead, you tell the assistant you have 20 instructions you need him to follow: run every search two times for accuracy, double check references, filter out references if they return certain parameters, draft an original memo, be brief, be formal, adapt it to the standard company format, and do all of it while dancing on one foot.
Now the person hasn’t lost a document — the desk has no books in this example — only a set of rules. But as the rules become more complex, the person begins to lose their judgment. They quietly work towards what they perceive as the loudest rules — always do this, never do that — at the expense of overlooking softer ones. Or they create eloquent output that satisfies all conditions while failing to land the objective: a nice-looking memo that does nothing to advance the project.
When Arize AI benchmarked instruction-following across frontier models this year, they found that even the best systems held steady and then fell off a cliff. Past a few hundred simultaneous constraints, models start dropping instructions they were meant to follow. (Interestingly, they don’t ease into it. They hold, then turn erratic, quietly favoring whichever instructions came first.) MindStudio documented a related failure they call “instruction interference”: tell a model to “be concise” in one section and “always explain your reasoning fully” in another, and it resolves the ambiguity differently every time. The behavior looks inconsistent, but the inconsistency is coming from the instructions themselves.
Here, as with desk stuffing, a bigger model doesn’t solve the underlying problem. When there are too many rules being shouted at the AI, an upgraded model with stronger reasoning is a magnet for more of the same mis-use, if un-curated. Even a more capable worker can’t take direction from fifteen managers at once.
The takeaway for your team
A system can drown in context or drown in rules, and what moves the needle on production AI is knowing what to leave out. Georgia O’Keeffe, talking about paint, put it plainly: we reach the meaning of a thing “by selection, by elimination, by emphasis.”
That’s the true meaning behind context engineering, and it’s fundamentally an architecture problem. With the spread of AI beyond engineering teams, understanding the solution is more important than ever.
When context is the overload, you cut it back and reorder it, with the load-bearing facts at the front and end where the model genuinely reads them. When it’s the rules, you thin them, settle the contradictions, and be ruthless about which behaviors matter versus which ones you bolted on because something embarrassing happened once.
A bigger, more expensive model gives you two things: a larger context window and stronger reasoning. But neither solves the problem of unstructured, unruly context and a spaghetti of competing rules.
For someone running a product or a business, curating context and pruning rules requires similar judgment to prioritizing a roadmap: deciding what’s essential and dropping the rest. And the teams that come out ahead won’t be the ones spending the most on the latest models. They’ll be the ones willing to leave things out.