What is RAG: how to make a model answer from your documents

Why this matters

• Understanding why an AI answer can be verified through a link to the document, and what to look at when doing so
• Understanding what the monthly bill covers and why it is cheaper than fine-tuning a model
• Telling apart the cases where a vendor is solving the problem from the cases where it is selling extra work

What is RAG in plain words

An open-book exam works differently from an exam taken from memory. The candidate leafs through the textbook, finds the right section, puts a finger on the line — and only then speaks. Mistakes are still possible, but the examiner can see where the answer came from.

RAG runs on the same principle. A language model answers from the memory of its training. Company policies, contracts and price lists were never part of it. It assembles plausible text and sounds equally confident whether it is right or making things up. RAG inserts one more step before the answer: search.

The system finds several relevant passages in company documents, places them directly into the request sent to the model, and instructs it to rely on nothing else. The answer comes back in plain language, with a link beside it: clause 4.2 of a named policy, edition of a given date.

How it works

1. Documents are loaded into a store. The text is cut into passages of a few paragraphs each and converted into numbers — embeddings. The original text is kept alongside them, together with labels: document, edition, department, access level.
2. An employee's question is turned into the same kind of numbers, and the closest passages by meaning are retrieved. At this step everything the person has no access to is filtered out.
3. The best of what was found is selected — usually between three and ten passages. The list is screened once more, and anything that landed there by accident is discarded: a superseded edition, a similar clause from someone else's contract.
4. The request to the model is assembled. It contains the instruction "answer only from the text below", the selected passages and the question itself.
5. The model formulates the answer and attaches links to the passages it was built from.

All the difficulty sits in steps two and three. When the right paragraph is in front of the model, the restatement usually comes out correct. When search returns the adjacent clause, the answer will be fluent, confident and wrong — and only someone who opens the link will notice. That is why RAG quality is measured by search, not by the elegance of the wording.

It is also agreed separately what the system does when no suitable passage is found. The correct behaviour is to answer "this is not in the documents" and pass the question to a human. Without that condition, the model will fill the gap with generalities from its training, and nothing in the answer will give it away.

An example from practice

At a wholesale company, the customer service team handled questions about delivery terms: around forty documents in force and roughly 900 enquiries a month. A third of the questions repeated, yet the answer was looked up by hand each time — through email, folders and colleagues' memory.

After launch, two weeks before and two weeks after were measured. Average time to prepare an answer fell from 11 minutes to 3. The share of enquiries an employee handled without asking a colleague rose from 55 to 82 percent.

More valuable than the speed was the link to the contract clause next to every answer. The team lead stopped proofreading every outgoing message and moved to spot checks — roughly one in ten.

Some enquiries the system did not close: questions where terms are agreed with a lawyer went to a human before and still do.

What the business gets

An update costs one file upload. When a price list or policy changes, the system is not retrained — the old document is replaced with the new one, and from the next question onward answers follow the current edition. Fine-tuning a model would have to be repeated in full.

An error can be caught before it reaches the client. A link to the source turns verification into a single click. That changes the control regime: blanket proofreading gives way to spot checks, and disputed cases are settled against the text of the document.

The data sits separately from the model. When a cheaper or stronger model appears, it is swapped in and the document base stays. If the documents cannot leave the building, the store and the search are deployed inside the company perimeter.

The bill is predictable. Payment covers the model's answers and storage; both figures can be worked out in advance, from the number of enquiries and the volume of documents.

When it can be done without

If there are a dozen documents and they are short, it is cheaper to put them into the request in full. The context window of large models holds hundreds of pages. Search is not needed at all here.

If the task does not touch company documents — drafting an email, translating, parsing an incoming text — RAG adds nothing and only makes each answer more expensive.

If the data sits in an accounting system in strict fields — stock levels, prices, order statuses — an ordinary database query is what is required. Search by meaning gets in the way here: it tends to substitute something similar for the exact value.

What is worth checking

Access rights are checked at the search step, before a passage ever reaches the model. Otherwise the system will one day restate to an employee a document they were never meant to see, and the logs will hold almost no trace of it.

Duplicates and editions are checked separately. Two versions of one policy in the store mean that sooner or later an answer will come from the cancelled one. Before launch it helps to draw up a list of what has been withdrawn and what is in force — that work is almost always underestimated in the schedule.

Before the start, it is also settled who is responsible for the contents of the store: whose files go into it, who confirms that an edition is current, and how long a change takes to reach the system. Without such an owner, the base goes stale within a couple of quarters while the answers keep coming in a confident tone based on last year's rules.

And finally: if what is wanted from the system is not knowledge of documents but a different manner of answering — format, tone, structure — that is a job for fine-tuning, and it is solved differently. These two approaches are regularly confused, and they cost very different amounts.