What is a context window, and why does the model "forget"?

Why this matters

• Understand why a long conversation with an assistant starts contradicting itself
• Understand what is driving the bill up when the volume of work has not changed
• Tell apart a task that fits in a single request from one that needs a search layer over a knowledge base

What is a context window, in plain language

Picture a desk that holds exactly forty sheets of paper. While there are fewer, everything sits in plain view and connects to everything else. Add a forty-first sheet and the oldest one leaves the desk — it is out of play.

A model works at the same kind of desk. Everything is laid out on it at once: the standing instructions, the attached documents, the user's question, and the entire preceding conversation. The size of the desk is measured in tokens — the fragments a model chops all incoming text into. A Russian word typically breaks into two or three of them.

What matters is how the sheets leave the desk. There is no warning. The model does not report that some of the paperwork never reached it. It answers from what is left — with exactly the confidence it would have shown with the full set in front of it.

How it works

1. Everything is gathered into one stack. Before each request, the system assembles the instructions, the retrieved documents, the conversation history, and the new question. The model receives all of it as a single block.
2. The stack is measured in tokens. Limits differ across models by orders of magnitude — from tens of thousands of tokens to a million. A rough rule of thumb: a thousand tokens is roughly a page to a page and a half of Russian text, and the same text in English takes one and a half to two times fewer tokens. Exact counts depend on the model, so the volume of a company's own documents is measured with the token counter of the model actually in use.
3. The excess is cut. Once the stack stops fitting, the oldest material is removed — usually the start of the conversation. Compression is used less often: older messages are replaced with a short summary.
4. The model reads the whole stack and answers. Reading is billed as input tokens, the answer as output tokens, and output costs more.
5. The next request starts from zero. A model has no memory of its own between requests. Memory is created by the system around it: the conversation is fed back into the stack every time and paid for every time.

One subtlety deserves its own note: "it fits" and "it works equally well" are not the same thing. In long stacks, models hold on to the beginning and the end more reliably and skip the middle more often. Hence a simple working rule: an important condition goes at the start or the end of the stack, because it gets lost more often in the middle of a long attachment.

An example from practice

Customer support at an online delivery service: about three thousand tickets a month, eight agents, an assistant handling the first line through to resolution. On long tickets — thirty messages and up — it would start asking again for the order number the customer had given at the very beginning. A review of a hundred such conversations showed that in 18 cases the assistant lost data it had already been given, and an agent had to step in from scratch.

The fix required no change of model. After every ten messages, the system compressed the conversation into a short card: who the customer is, which order, what has already been checked. From then on, the card went into the window instead of the full transcript. A repeat review of a hundred conversations of the same length showed 2 cases of data loss. A side effect: average request length fell roughly threefold, and with it the charge for input tokens; there were no savings on output tokens, which do not depend on request length.

What this gives a business

• The bill becomes explainable. Costs rise with the length of each request: a thousand short requests come out cheaper than a hundred long ones. Cleaning up the context reduces the bill without rewriting the system, which makes it a sensible place to start any conversation about savings.
• Less rework for people. An assistant that loses information from the start of a conversation hands the task back to an agent. Every handback costs staff time and forces the customer to repeat what has already been explained.
• A clear project boundary. Window size answers straight away whether a single request will do or whether a separate document search system is needed. These are projects of different duration and different cost, and the question is better settled before a contract is signed.
• The risk becomes visible. Context is truncated silently. If the system logs such cases, data loss shows up in a report rather than in a customer complaint.

When this can be ignored

If a task fits in a single request — reviewing an invoice, rewriting an email, sorting a request into categories — window size need not be a concern. A ten-page document is a few thousand tokens; with the models in common use today it occupies a small share of the window, and there is no reason to build compression or search around it.

The question is equally moot where there is no dialogue at all: processing runs in batches, each document separately, and no history accumulates. The limit shows up in two situations — long conversations, and attempts to hand a model a large body of text in one piece. Everything else comes down to other things: data quality, how the task is framed, and the choice of model by price.

What is worth checking

The advertised window size and the working window size are two different numbers. The provider states a limit, but quality at that limit has to be verified in house: take the company's own documents, hide a specific fact at various depths inside them, and see whether the model finds it. On paper it is a million tokens; in practice, reliable performance may end considerably earlier.

Second, ask the contractor what happens on overflow. Silent truncation, compression into a summary, an explicit error: three different behaviours. Silent truncation is the most dangerous of the three, because the system keeps answering and looks perfectly healthy. A clear answer sounds like this: on overflow the system compresses the conversation, every such case goes into a report, and the frequency is visible.