What is model fine-tuning and when it pays off

Why know this

• To see what is actually being delivered when a vendor offers to "train a model for you"
• To tell the task "answer in a set format" apart from the task "know the company's documents": different solutions, different prices, different timelines
• To recognize the point where fine-tuning pays for itself, and the point where instructions in the prompt are enough

What fine-tuning is, in plain terms

An experienced manager joins the department. Writing letters is already a known skill, and so is drawing up documents. On the first day two things are handed over: a stack of samples showing how answers are written at this company, and a folder holding the current price list.

Fine-tuning is the stack of samples. An off-the-shelf model is shown a few hundred of your "request — correct answer" examples, and it starts reproducing the manner: the structure of the answer, the tone, the mandatory fields, the length.

The folder with the price list works differently. It is the insertion of the relevant documents into the prompt — the RAG pattern. When prices change, the contents of the folder change. A new price list is handled by swapping a file in the folder, not by retraining: nobody sends the manager back to training because the rates moved.

Hence the main point: fine-tuning changes the manner, while the company's memory lives separately, in documents.

How it works

1. Samples are collected. "Request — reference answer" pairs are pulled from the correspondence archive, support tickets and finished documents. They are cleaned up and brought to a single standard: if half the examples follow the old format, the model will learn the old format too.
2. Some examples are set aside. A control sample the model never sees during training. It is what "before" and "after" are later compared on — without it there is nothing to measure the improvement against.
3. Training is run. The model is shown your examples, and its internal settings shift slightly toward your manner. This takes hours, not weeks: the base model is not taught from scratch, only adjusted.
4. Results are compared on the control sample. Two things are checked: whether the format became more stable, and whether the model lost general skills on adjacent tasks.
5. A separate version is connected. The company ends up with a model of its own, tied to the base one. From then on it has to be maintained.

One important detail about step one. Even when specific figures and names appear in the examples, they do not become a reference book. The model memorizes them as part of the manner and will later quote an outdated price with full confidence, because "that is how it was in the samples". Reference data is not stored inside the model — it is inserted into the prompt on every request.

A case from practice

A service company maintaining retail equipment, around 1,500 support tickets a month. Every answer has to carry the ticket number, the deadline, a reference to the relevant clause of the contract and a signature. The instructions in the prompt had grown to three pages, and the model still broke the format: in a sample of 200 answers over two weeks, errors turned up in roughly one in five.

The model was fine-tuned on 900 "ticket — approved answer" pairs from a year of archives. On an identical sample of 200 answers, the share of format violations fell to roughly three percent. The money was counted separately: the instructions shrank from three pages to four lines, and processing a single ticket became about a third cheaper — the long text of the rules stopped being paid for on every request. Part of the gain was eaten by the provider's premium: calls to a fine-tuned version are billed at a higher rate than calls to the base one.

Deadlines and contract clauses were still pulled from the database. Fine-tuning did not replace that, and could not.

What the business gets out of it

• Less manual checking. A stable format means answers no longer have to be re-read and corrected before they go out. The saving is easy to compute: the share of answers edited before and after, multiplied by the time one edit takes.
• Every operation costs less. The manner moves out of the instructions and into the model, so the long text of the rules stops being sent with every request. Instructions are paid for on each request, so at a flow of thousands of tickets a day the difference shows up on the invoice. It should be counted net of the premium charged for running a fine-tuned version.
• A simpler model is enough. After fine-tuning, a narrow task is often handled by a junior model where a senior one used to be required. The price gap between junior and senior models at the providers reaches tenfold.
• Predictability. The answer stops depending on how the request happens to be worded today. Where a letter goes to the customer without proofreading, it is the customer who sees the variation in format, which makes the share of deviations from the template a measurable quality indicator.

When it is possible to do without

Most tasks are covered by instructions and three to five examples placed directly in the prompt. The order runs like this: instructions in the prompt first, then RAG, fine-tuning last. Each step costs more than the one before, and only a minority of tasks reach the third.

If the request sounds like "let the model know the company's documents", fine-tuning does not solve it at all: knowledge does not get inside that way. Which class of solution fits a given task is shown by the solution finder — it separates exactly these cases.

If documents and rules change more often than once a quarter, a fine-tuned version will go stale faster than it pays for itself.

And if there are no three to five hundred cleaned-up reference answers on hand, there is nothing to train on — the project will stall on data collection, not on technology.

What is worth checking

A fine-tuned model is tied to its base version. When the provider releases the next one, yours will not update itself, and the training has to be repeated. Put this in the budget as a recurring line, not a one-off.

Second: narrow specialization comes at the cost of general flexibility. On tasks that were absent from the examples, a fine-tuned model sometimes performs worse than the original. Testing has to run on two sets at once — your typical cases, and the adjacent ones where the model must be no worse than before. Both measurements are taken on the same sample before and after, otherwise there is nothing to compare.