Error Rate at Volume

How the calculation works

The starting point is volume. The number of replies per month is multiplied by the share of wrong ones measured in testing. That yields the stream of errors at the system's current quality.

Then the stream splits in two. Part of the replies goes under review, the rest go to the customer directly. In the unreviewed part, every error gets through without exception.

Inside the sample, not everything is caught either. Review depth is set by a coefficient: quick scan — 0.7, checklist — 0.85, cross-check against the source — 0.95. These are reference points for a rough estimate, not values measured on a specific project.

The result is how many wrong replies reached the customer over a month. Alongside it, the cost of control is calculated: review hours, headcount and their cost at the employee's rate.

What moves the result most

The first lever is the error share. It acts on the whole volume at once and without people. Dropping from 4% to 2% on a flow of six thousand replies removes 120 errors from the monthly stream, regardless of who reviews and how.

The second lever is sampling depth. Going from 40% to 80% with a checklist roughly halves the number of errors that get through. The price is hours: review doubles too.

The third lever is volume. It is a multiplier for both of the previous ones. A 2% share looks calm on a thousand replies and turns into a stream at forty thousand.

The order of the levers is not accidental. Sampling costs money every month, while work on reply quality pays off once and holds afterwards.

Under what conditions an assistant is released to people

The first condition is reversibility. An error is visible and can be fixed before it becomes an obligation: a wrong date is corrected by a manager, a wrong price is caught by a price-list check on the way out.

The second is a narrow topic. On a single scenario with clear sources, the error share falls fastest, because rules can be stated briefly and checked on a sample within a day.

The third is working escalation. The assistant must be able to say that it does not know and hand the conversation to a human. Declining to answer is cheaper than a confident invention.

The fourth is sampling on an ongoing basis. Even after reaching a low error share, a few percent of replies are looked at by eye: the request flow changes, and the picture of errors changes with it.

How to measure your own error share

Real requests from the past two weeks are taken — 150–200 of them, with no cherry-picking of convenient ones. The assistant answers them separately from customers. The replies are labelled by a specialist who knows the subject area.

Labelling goes into three buckets: correct, wrong, declined to answer. Declines are counted separately and do not go into the error share — they are costly in terms of convenience but safe in terms of consequences.

The resulting share is entered into the calculator. On a sample of a hundred requests the margin of error is around two percentage points, so the number should be read as a range.

What stays outside the brackets

In the calculation, errors are equal to one another; in life they are not. A mixed-up delivery date and a mixed-up warranty condition differ in cost by orders of magnitude. Estimating the damage requires analysis by type.

Errors come in clusters. One gap in the knowledge base produces a run of similar misses back to back, and a flat monthly average does not show such a run.

The lag of control is not accounted for either. Spot-checking is often done after the fact, while the reply goes to the customer immediately — in that case the number of errors that get through does not depend on review depth at all. For the calculation to be honest, review must sit before sending, at least on risky request types.

Finally, the work on the errors themselves stays outside the brackets. Analysing cases, correcting instructions and expanding the knowledge base take hours on top of review. Usually that is between a quarter and a half of the time spent on the sampling itself.