What is latency and how to keep it within seconds
Why this matters
- Understand why users abandon a system even when its answers are correct
- Know which levers speed up responses without swapping the model or rewriting the system
- Ask vendors for response times on live traffic, not on a demo request
What latency is in plain words
A coffee shop takes an order in a second. The coffee takes three minutes. But what irritates people is not waiting for the cup — it is the barista who says nothing and gives no sign of having heard.
Models work the same way. Latency is the time between sending a request and receiving a finished answer. It splits into two parts: waiting for the first word, and finishing the rest of the text.
The first part carries more weight. A screen where nothing happens reads as a breakdown. The same answer, displayed as it is written, feels fast even when the total time is identical.
How it works
- Two numbers are measured, not one. Time to the first word governs the sense of a live conversation. Total response time governs whether an employee can keep up with the queue.
- The tail is what counts, not the average. Take the value that is not exceeded in 95 cases out of 100. The average hides rare slow answers, and complaints come from exactly those.
- The wait is broken down step by step. A stopwatch is put on each step of the chain separately. Usually it turns out the model accounts for the smaller share of the wait.
- Input volume is trimmed. A short request is read faster than a long one. Here latency and the token bill move in the same direction: less volume means faster and cheaper.
- Answer length is capped. A 600-word text takes roughly twice as long to write as a 300-word one. The cap is set in the request, not expressed as a wish.
- Streaming output is turned on. The answer appears on screen from the first words. Total time does not change; the perception changes a great deal.
Chains of several requests deserve separate attention. Each step an AI agent takes stacks onto the next: four requests at three seconds each add up to twelve seconds of waiting. Some steps can run in parallel, others can be replaced by an ordinary integration with no model involved.
An example from practice
The figures below are calculated benchmarks for a company of this profile, not a report on a specific deployment. Each comes with its measurement method, so it can be repeated on other data.
A wholesale company connected an assistant to incoming order processing. The substance of the answers was correct, but the operators went back to email. A measurement across 300 requests over two weeks showed this: at the tail value, the first word appeared after 6 seconds and the full answer after 11.
The wait was broken down step by step. Knowledge base search took 2 seconds, the call to the warehouse system another 1.5, and the model took the remainder. Three things were done: streaming output was turned on, the injected context was cut from eight documents to three, and the warehouse call was moved to a parallel step.
The first word began appearing in under a second, and the full answer came in under 5 seconds at the same tail value. The share of correct parses was checked on the same sample before and after — it did not change. The operators came back to the system without anyone having to order them to.
What this gives the business
- The system actually gets used. A correct answer that arrives too late plays no part in the work. Speeding up the first word brings users back without a single change to the logic.
- Throughput becomes predictable. An operator with a 5-second answer handles noticeably more requests per shift than one with an 11-second answer. That figure is calculated on live traffic, before licences are purchased.
- The levers are visible and cheap. Streaming output, short context, an answer-length cap, parallel steps — four techniques that require no change of model. The load per request can be estimated with the context window load calculator.
- The vendor conversation gets concrete. The question becomes "what response time holds up at 500 requests a day" — and it has a verifiable answer.
When this can be skipped
Overnight document processing lives by a different rule. A batch of invoices parsed by eight in the morning is equally good whether each document took one second or ten. Here volume per shift is what counts, and a cheaper model is the right pick — inference costs less during quiet hours.
The same goes for tasks where a person is waiting for an outcome rather than an answer: preparing a report, working through an archive, monthly reconciliation. Here the stopwatch gives way to a deadline. Such processes can be sped up, but it rarely saves money.
What is worth checking
First: a demo request is almost always faster than a real one. A demo sends a single question; in production that question comes with an instruction, retrieved documents, and conversation history. Ask for a measurement on traffic that matches the request volume in question.
Second: reasoning models think before answering, and that time lands inside the wait. On complex tasks the invisible part can run longer than the answer itself. This is verified on real requests before launch.
Third: latency grows along with load. A system that answers in 3 seconds with ten users may take fifteen with a hundred. A measurement under load is done once and removes the biggest post-launch surprise.
Frequently asked questions
What is latency in plain words?
It is the time from pressing a button to a finished answer. Inside, it splits into two parts: how long the first word took to appear, and how long the rest of the text took to finish. Users feel the first part far more sharply. That is why systems that display the answer as it is written feel faster even when total time is identical.
What counts as normal latency
The benchmark depends on the scenario. A conversational assistant needs to respond within 100–150 milliseconds, otherwise the conversation breaks down. For document analysis, twenty seconds can be perfectly normal: the person is busy with something else meanwhile. For overnight batch processing it does not matter at all — what counts there is volume per shift.
Why is the answer slower than it was in the demo
Demos use a short request with no surrounding context. In production, that request also carries an instruction, retrieved documents, conversation history, and calls to adjacent systems. Every component adds its own time. Break a single request down step by step with a stopwatch — usually two thirds of the wait sits in the steps around the model rather than in the model itself.
How can the wait be shortened without changing the model
Four levers work faster than the rest: turn on streaming output, shrink the input volume, cap the answer length, and remove unnecessary steps from the chain. A fifth is caching the fixed part of the request at the provider. Each lever is checked by measuring before and after on the same sample of requests.
How can latency be measured in-house
Take a week's worth of live traffic, at least 200–300 requests. For each one, record the time to the first word and the total response time. Do not calculate the average — calculate the value that is not exceeded in 95 cases out of 100. The average hides the slow tail, and complaints come from exactly that tail.
Are latency and the model bill connected
They are connected through volume. A long input request and a lengthy answer both cost more and take longer to wait for. Trimming context usually improves both figures at once. The reverse case is an expensive fast model: the wait drops, the bill rises, and the choice is made on the cost of a minute of an employee's time.
Let’s discuss your project?
Tell us about your process — we’ll suggest where AI pays off fastest.
Related articles
What Inference Is and What Makes Up the Bill for It
Inference is the moment a trained model answers a request — and it is what gets billed every month.
What is a context window, and why does the model "forget"?
A context window is the limit on how much text a model can see in a single request: instructions, documents, the question, and the conversation history all count together, and anything past the limit is dropped silently.