Assistant Response Latency

How the calculation works

The answer comes together in four beats, and each one adds seconds.

The first beat is reading. The model takes in the instruction, the dialogue history and the retrieved knowledge base fragments. It reads noticeably faster than it writes. In the calculation the reading speed is obtained by multiplying the output speed by forty, so even a large prompt costs single-digit seconds.

The second beat is generation. Tokens come out one at a time, and the time grows linearly with the length of the answer. Three hundred tokens on a mid-range model take about seven seconds. This is the largest line item in almost any configuration.

The third beat is calls to the outside. A search over a vector base, a CRM query, a stock check in the warehouse. The select field has three typical values built in: from tenths of a second to a little over two.

The fourth beat is network and queue. A fixed 0.4 seconds per call. That is what goes to the trip to the provider and waiting for a free slot under load.

The sum of the four beats is multiplied by the number of calls. The chain "refine the question → find → formulate" is three calls, not one.

What weighs more than the rest

The length of the answer. Every extra hundred tokens on a mid-range model costs roughly two seconds, and no prompt optimization compensates for that. A short answer format is the cheapest way to speed things up.

The number of calls. The multiplier applies to the whole beat, network and retrieval included. Two steps instead of three remove a third of the wait, even if the model stays the same.

The model class. The difference between a compact model and a large reasoning one is roughly fivefold in output speed. The large model is placed where an error costs more than seconds: calculating terms, legal wording, disputed cases. On routine questions it is kept in reserve.

Prompt length matters least of all. Five thousand extra tokens of context add about three seconds. Context is trimmed for the sake of cost and output accuracy, while seconds are found elsewhere.

Two limits of patience

In chat the limit is soft. With streaming output the wait ends at the first words, so what matters is not the full duration but the pause before the start. A second and a half to two seconds before the first word reads as a fast answer, even if the text keeps being written for another ten seconds.

In telephony the limit is hard. Silence longer than a second and a half is perceived as a dropped call, and the gateway breaks the session after about thirty. Between these thresholds a short filler line fits: it starts playing immediately while the model computes the main answer. The breakdown shows what share of the thirty-second window the chosen configuration takes up.

For email and requests neither limit applies. There an answer is awaited for minutes, and the model is chosen by the quality of its analysis.

How to verify the number on your own setup

The calculation gives a reference point, measurement gives a fact. The method is simple. The same request is run twenty times during working hours, not at night, and the median is taken — the best result out of twenty is deceptive. The time to the first token and the time to the end of the answer are recorded separately. The difference between them is generation.

The measurement is repeated at two prompt lengths: short and full. The time difference divided by the token difference gives the reading speed for the specific provider. The resulting numbers are substituted for the coefficients in the calculation.

The peak hour is counted separately. The provider's queue during the load hour adds anywhere from fractions of a second to several seconds per call, and on a chain of three steps that is noticeable.

When a single measurement is enough

If the assistant already works and there are no complaints about speed, the calculation is only needed before changes. Switching the model, adding a verification step, expanding the knowledge base — each such decision shifts the seconds, and it helps to see the shift in advance.

For a new scenario the order is reversed. First the acceptable latency for the channel is chosen, then the answer length and the number of calls are fitted to it. Designing this way is cheaper than speeding up something already built.

Fitting within a customer's patience happens under three conditions: the answer is short, the steps are few, streaming is on. These three coinciding is enough to stay inside a live chat even on a large model.