Context bloat: what it means in plain terms

Why this matters

• To understand why the bill for an AI system grows faster than the number of requests
• To see the cause of slower answers and lost quality without any change of model
• To ask a vendor for a request breakdown by source rather than a single project price

What context bloat means in plain terms

An employee is getting ready for a client meeting. Three pages matter: the latest invoice, the disputed line in the acceptance report, the letter from the lawyer. The whole folder goes along anyway — it feels safer that way.

The folder swells every quarter. It is heavier to carry and slower to search, and the decision still rests on three pages.

The model receives the same folder on every call. Inside are the system instruction, tool descriptions, attached documents, the dialogue history, and search results from the knowledge base. Together this is called the context, and the room for it is limited by the context window.

Context bloat is the state where the folder keeps growing while the number of decisive pages stays the same. That volume is paid for in tokens on every request.

How it works

1. The request is assembled from pieces. Overhead, history, documents and tool replies are stacked into a single package and sent to the model.
2. The whole package goes out on every call. Caching makes resending the fixed part cheaper, but the variable pieces are sent again at full price.
3. Volume grows with each turn. By the end of a long conversation, the input volume of a single turn is several times what it was at the start. Cost and response time follow.
4. Overhead grows with complaints. Every failure that gets investigated adds a rule to the instruction. Rules pile up; old ones are never removed.
5. The useful share falls. Buried in surplus text, the needed fragment gets lost, and answer quality drops before the window runs out of room. That effect is covered separately — context rot.

Agents deserve separate attention. An agent takes many steps in a row and appends each result to the shared context, so volume accumulates within a single user task.

A case from practice

The typical picture in customer support looks like this. The system works, the answers are acceptable, but the monthly bill is noticeably above plan, and long dialogues respond more slowly than short ones.

The way in is measurement. One typical operation is taken, its input volume is broken down by source, and the measurement is repeated at the first and the twentieth turn of a dialogue. Volume is counted with the tokenizer of the model in use — a token calculator per task helps gauge the order of magnitude.

From there it becomes clear what to cut first. History and the system instruction usually sit at the top: both are sent on every call and both grow without human involvement. The techniques are known — a window of recent turns, a summary instead of the full history, a cache for the fixed part, filtering of search fragments. They are gathered into a single request-assembly layer known as context compaction.

The result is checked against the same sample of dialogues before and after. Two numbers matter: input volume per operation and the share of hand-offs to a live agent. Savings only count where the second number has not got worse.

What this gives the business

• The bill returns to plan. Cost per operation stops depending on conversation length, and the monthly budget becomes a matter of multiplication.
• Answers arrive faster. Time to first response is tied directly to input volume, so trimming the surplus speeds the system up without a change of model and without new hardware.
• Quality improves alongside the savings. A short package assembled for the task is less likely to pull the model toward stale fragments of the dialogue.
• There is something concrete to discuss with the vendor. A breakdown of input volume by source is a verifiable document that shows where the money goes and what to cut first.

When this can be skipped

With rare, short requests there is nothing worth breaking down. A few hundred requests a month with no long dialogues and no attached documents produce a bill against which the reduction work costs more than the whole saving.

The topic does not apply to subscriptions with a flat per-seat fee: there the constraint is request limits, and text volume has no bearing on price. One-off tasks with large documents are a separate case too — there the long package is the work itself, and it is handled as long context rather than as bloat.

What is worth checking

First: room in the window and money are different constraints. A request can fit the window and still be expensive, so "it fits" is no substitute for measuring volume.

Second: a reduction is validated on quality, not on the bill alone. Trimming history and cutting search results removes some facts, and that shows up on a sample of real dialogues before and after.

Third: the volume breakdown is worth measuring regularly, at least once a quarter. Instructions and toolsets grow on their own, and volume returns to its former level within a few months of any clean-up.