Fine-tuning or RAG — which one fits your task
Why this matters
- Telling apart two quotes that promise the same thing on paper
- Knowing which approach updates in minutes and which needs a new training cycle
- Asking a vendor what exactly changes: the model's style or its access to documents
What are fine-tuning and RAG in plain words
A new employee joins the department. Path one: a month of training — how to talk to a customer, how to handle disputed cases, what tone to use in a letter. Path two: access to the folder of policies and a tour of where things are kept.
Fine-tuning is path one. The model is shown hundreds of examples of finished answers, its internal weights shift, and the company's style settles inside it.
Document search is path two. The model stays as it was, and an index of policies is set up beside it. For every question, a matching passage of text is slipped in.
The formula is short. The model should answer like the best employee — that is fine-tuning. The model should know what that employee knows — that is search.
How it works
- Documents are cut into passages and placed in an index. A passage is a section of a policy, a product card, a page of a manual. The index lives in a vector database and is updated separately from the model.
- Search answers the question first. It pulls three to five matching passages and slips them into the query. The model writes its answer from those passages and cites the clause.
- Examples are collected for fine-tuning. Pairs of question and correct answer, written the way it should be done. A rough order of magnitude is several hundred pairs per task.
- The training cycle is run. The weights shift, the style sets. After every policy edit, the cycle starts over.
- Measurement runs on a single sample. The same questions are run before and after, otherwise there is nothing to compare.
The update cycle is the main operational difference. A document reaches the index in minutes; a new version of a fine-tuned model takes weeks.
An example from practice
A chain of service centres keeps a policy base of roughly 900 pages. Technicians write into a shared chat: what warranty covers a given unit, what to use in place of a discontinued part, how to process a return. The figures below are an estimate for a company of this profile, not a report on an implementation. Each one comes with the method used to measure it.
The first measurement is done by hand. Two hundred questions from a month of correspondence are taken and marked by where the answer sits. Some questions are closed word for word by a clause of the policy — that is work for search. Some are closed only by a technician's experience — that is work for fine-tuning.
In a chain like this the policy is revised once a quarter, and the price list more often. Rebuilding the index after an edit costs a few hours of work; repeating the training cycle costs considerably more. Hence the usual order: search first, then fine-tuning on the errors that remain.
Quality is measured the same way in both cases. An answer counts as correct when it matches the policy in force and cites the correct clause. The measurement is repeated on the same sample of 200 questions after every change to the system.
What the business gets
- A policy edit reaches staff the same day. The document goes into the index, and the next answer already follows the new version. No waiting for a training cycle.
- The answer can be checked. A link to the clause settles the argument over where the model got it from and cuts the time spent handling complaints.
- Query length drops after fine-tuning. Part of the instruction moves inside the model, and every request becomes shorter — which shows up in the bill for tokens.
- Quotes become comparable. One vendor prices building the index, another prices labelling examples. The share of questions with an answer in the documents shows what to pay for first.
When this can be skipped
While there are few documents, a prompt is enough. Twenty or thirty pages of policy fit into the context window whole, and a separate search adds nothing. The threshold is easy to check: if the entire rulebook fits into a single query and the answers are correct, it is too early to build an index.
Fine-tuning is postponed until the answer format has settled. Retraining a model taught on an old letter template costs more than changing a couple of lines in an instruction. The format is fixed on paper first, then locked into the weights.
What to check carefully
First: the quality of the document base. Search answers with exactly what was put into it. Two versions of one policy in the index produce two different answers to the same question, and the model is not to blame. Before working out a budget, it is worth running the process through the readiness checklist.
Second: a hybrid counts as normal, not as a sign of failure. Search supplies current facts, fine-tuning supplies style and format. Order matters: fine-tuning on top of a raw document base does not remove the errors.
Third: ask the vendor to show a measurement on your own sample before signing. How many questions were taken, over what period, what counted as a correct answer. A promise of accuracy without those three numbers cannot be checked.
Frequently asked questions
What are fine-tuning and RAG in plain words?
They are two ways to adapt an off-the-shelf model to a company's work. Fine-tuning changes the weights of the model itself: it picks up the house style and learns how to handle edge cases. RAG leaves the model untouched and puts a document search alongside it, slipping the relevant passage into the query. The short version: style comes from fine-tuning, knowledge of documents comes from search.
Which is cheaper to keep up to date?
Document search. When a policy changes, the new version goes into the index and the next answer already reflects the edit. With fine-tuning, the same edit calls for a fresh set of examples and another training cycle, and until that finishes the model still answers by the old rules. So wherever documents change more often than once a quarter, search costs less to maintain.
When is fine-tuning the better fit after all?
When the answer appears in no document at all. An experienced technician carries the judgment calls in their head, and there is nothing to look up in a database. The second case is a strict answer format: a particular letter structure, a particular set of fields, a particular tone. The third is shortening the query: part of the instruction moves inside the model, and every request becomes shorter and cheaper.
Can both approaches be used at once?
Yes, and that order is common. Document search comes first — it supplies current facts and a link to the source. Then the remaining errors are examined: if they concern style and format, fine-tuning on a few hundred examples is added. The reverse order costs more, since fine-tuning on a poor document base does not remove the errors.
How can the choice be tested before signing a contract?
Collect 100–200 real questions from the past month. For each one, note whether the answer exists in company documents. If the answer is there in the text, it is a search task. If only an experienced employee knows it, it is a training task. The split between the two is the answer to the question of what to pay for first.
Let’s discuss your project?
Tell us about your process — we’ll suggest where AI pays off fastest.
Related articles
What is RAG: how to make a model answer from your documents
RAG is a way of working with AI where the system first finds the right passage in a company's own documents and then builds an answer from it.
What is model fine-tuning and when it pays off
Fine-tuning adjusts an off-the-shelf model on a company's own examples so that it answers in the required format and tone; it adds no knowledge of the company's documents.