Answer Spread and the Cost of Checking

How the calculation works

The flow splits into two risk groups. The first is answers that will diverge from one another when the same question is repeated. The second is answers containing a fact that exists neither in the knowledge base nor in reality.

The divergence share starts at 9% and is multiplied by the format coefficient. A template with mandatory fields leaves the model little freedom and halves the share. Free-form wording raises it almost twofold: the same knowledge gets retold in different ways, and some of those retellings change the conclusion.

The fabrication share is counted from 4% and depends on what the answer rests on. When the text is assembled from knowledge base excerpts, fabrication has nowhere to appear — coefficient 0.35. When the model answers from general knowledge, the share grows 2.2 times.

Next comes the work of people. The critical share of the flow is checked in full, and the time is multiplied by the hourly cost. The result is the monthly cost of review in rubles and the reviewer's workload in full-time slots.

What affects the result most

Minutes per check. This is a multiplier across the whole volume: the difference between two minutes and eight changes the bill fourfold. Cutting them is easier than it seems — a templated answer reads faster than free text.

The share checked in full. Every next 10% of the flow costs as much as the previous one. That is why the zone of full review is described as a list of cases, not as a share picked out of thin air: first list where an error costs money, then look at what share that turns out to be.

The answer format and the source are the only two fields that move not the cost of checking but the volume of questionable answers itself. The "questionable and unchecked" line shows the other half of the picture: how many divergences reach the client without a second look.

Under what conditions review pays off

The first condition is measurability. Until a divergence log exists, any improvement remains a matter of faith. The log is started before the pilot: question, answer, reviewer's verdict, reason. After a month it becomes clear which topics produce inconsistency.

The second is the gradual handover of checks to code. Numbers are matched against the knowledge base automatically, mandatory fields are checked for presence, an answer without a reference to a source is flagged. A person handles what is left.

The third is that the zone of full review narrows based on evidence. Topics with not a single divergence over three months move to 10% sampling. The freed-up hours go to topics where divergences did occur.

When one figure is enough without a calculation

At a flow of up to a few hundred answers a month, all the checking fits into a couple of hours a week. There is nothing to calculate here — it is enough to assign someone on duty and keep a log.

The opposite end of the scale is also clear without a calculator. If an error in a single answer costs more than the annual review budget, the share checked in full equals 100%, and the conversation is about architecture: matching against the accounting system, refusing to answer on incomplete data, a mandatory second loop.

The in-between case is where the figure changes the decision. A flow of thousands of answers, a different cost of error by topic, limited expert time. The calculation shows the order of magnitude and suggests which of the six fields to move first.

What is left out of scope

Building the automated checks. Number matching, format control and refusal to answer without a source are written once and then work on their own; they are not part of the monthly cost of review.

Putting the knowledge base in order. Regulations that contradict each other produce divergences that neither format nor review can cure: the model faithfully retells what it found. That work is scoped by the state of the documents.

The cost of a missed error. The "questionable and unchecked" line gives a count but not a damage figure: it depends on who was told what. There is a separate tool for that part of the calculation.