AI hallucinations: why models confabulate and how to catch it

Why this matters

• Understand what exactly is being paid for when a vendor budgets for answer verification
• Spot in advance which processes make a model's error expensive and which make it harmless
• Know how to demand a measurable accuracy figure from a contractor instead of the word "works"

What is a hallucination, in plain terms

Picture a waiter who is not allowed into the kitchen. A guest asks whether the sauce contains nuts. The waiter has worked thousands of tables, knows what goes into sauces like this one, and answers calmly and to the point. Nine times out of ten the guess is right. On the tenth, the sauce does contain nuts — and the voice sounds exactly as confident.

A language model works much the same way. It opens no reference book and checks nothing against a database — it predicts the most likely continuation of the text based on how similar phrases looked in its training data. If a fact appeared often, the continuation matches the truth. If the fact was absent or rare, the gap still gets filled: out comes a statute clause that does not exist, an invented link, or a plausible number.

Nothing in the answer itself separates the two. The model does not shift tone when it crosses from knowledge into guesswork.

How it works

1. The model computes probabilities. At each step it picks the next word from those that occurred most often in similar contexts. No step in that procedure asks whether the claim is actually true.
2. The gap always gets filled with something. A model almost always produces an answer: training rewards answering, and "I don't know" surfaces rarely unless it is explicitly permitted in the task description.
3. The format of the answer amplifies the effect. Ask for a citation to a regulation and it arrives in perfect form: number, date, article. The formatting holds up; the content may be fabricated.
4. The rarer the topic, the higher the risk. On mass-market questions described thousands of times in public texts, models err noticeably less. Your internal rules, part numbers, and specific contract terms were never in the training data at all — and that is where the invented share accumulates.
5. Supplied documents change the picture. When the right fragment of a policy is placed in front of the system before it answers, the model leans on that rather than on memory. The waiter has been let into the kitchen.

An example from practice

A service company maintaining industrial equipment: 60 staff, roughly 900 support requests a month. An assistant was set up to answer customer questions about product specifications. Before launch, 200 typical questions with answers already known from the equipment datasheets were assembled and run through the system.

In 14 of the 200 answers the model quoted parameters that appear nowhere in the datasheets: close to the truth, but not the actual values. That is seven percent, and a customer would read such an answer as the supplier's official position.

Search over the datasheets was then added, citing the specific clause was made mandatory, and the answer "not found" was permitted. A second pass over the same 200 questions produced 3 discrepancies instead of 14. The remaining three explain why the check was not removed from the process: answers about warranty terms are still read by an employee, and the time for that is budgeted into the workload.

What this gives a business

Understanding the mechanism changes three things in the numbers.

The budget becomes honest. On a project where errors carry a price, a line item appears for verification: a set of control questions, measurements after every change, a human in the chain. That work costs real money, and it is far better discovered at the budgeting stage than after launch.

Processes split into two types. Where an employee reads the answer before it goes out, the process itself closes the risk. Where the answer reaches a customer or lands in a document automatically, sources, constraints, and spot checks are required.

There is something to put in the contract. Instead of "the system answers correctly", acceptance criteria state something measurable: the share of discrepancies on an agreed set of questions stays below a given percentage, measured monthly. Wording like that can be verified, and an argument about quality turns into arithmetic.

When this can be skipped

If a human reads and edits the model's output before it is used anyway, no separate defence against hallucinations is needed. A draft email, an alternative headline, meeting notes, ideas for a brief: the error is visible immediately and costs a minute to fix.

Nor is it worth stacking up checks where no exact answer exists by the nature of the task: rephrasing a text, proposing five name options, breaking a long document into bullet points for a quick scan. Verification here costs more than the error itself.

The line is simple: if the answer goes out or lands in a document without a human reading it, a defence is needed; if it stays internal as a draft, it is not.

What to check

Ask for the number together with the method. "95% accuracy" means nothing without stating which set of questions it was measured on and when — on your wording the result will be different.

Check three things in a finished system: whether it shows the source alongside the answer, whether it can say "this is not in the documents" instead of inventing something, and whether logs are retained so a specific complaint can be traced. Without logs there is no way to settle a dispute with a customer about an error.

Agree separately on who keeps the set of control questions. If it lives only on the vendor's side, there will be no way to verify the promised accuracy after handover: the set would have to be rebuilt from scratch, with an argument about measurement methodology thrown in.