What Machine Learning Is and When It Pays Off

Why this matters

• Tell which company tasks are solved by a prediction from data and which by an ordinary rule
• Know how much data has to accumulate before a project is worth discussing with a vendor
• Ask not "what is the accuracy" but "how was it measured, and on what data"

What is machine learning in plain terms

An experienced warehouse manager looks at an order and says: this customer will take the goods, that one will fall through. Ask why, and the answer does not come at once. Several thousand shipments have settled in memory, and the judgement surfaces by itself.

Machine learning does the same thing, only with records in a database. The algorithm is shown the history: here is an order, here is how it ended. It looks for what set the failed orders apart from the fulfilled ones and derives the rule itself.

There is one difference from an ordinary program. In a program, a person writes the rule: if the amount exceeds 500,000 and the customer is new, flag it for review. In machine learning, the rule is pulled out of the data, and it accounts for connections nobody ever wrote into a procedure.

Hence the central condition. Where the data exists, the task can be solved. Where it does not, the data has to be accumulated first — a separate job with a timeline of its own.

How it works

1. Frame a question with a single answer. Not "improve warehouse operations" but "will this shipment fall through or not." The answer has to be visible in past records, or there is nothing to learn from.
2. Gather examples with known outcomes. Two years of orders, each marked with how it ended. This is the longest part: the data sits across three systems and an inbox, and the fields are not filled in everywhere.
3. Split the data in two. The model learns on the larger portion; the smaller one is withheld entirely. It exists for an honest test — on its own data, any model looks excellent.
4. Train the model and test it on the held-out portion. Count how often it was right and how exactly it was wrong. Errors are broken down by type: missing a failure and raising a false alarm cost different amounts.
5. Set a confidence threshold. The model returns not a "yes" but a number between zero and one. Below the threshold, the decision goes to a person. The business sets the threshold, and it determines how many cases are handled automatically.
6. Measure again a quarter later. Data shifts, and quality declines gradually and invisibly. The testing schedule is fixed before launch.

Between steps 4 and 5 lies the most common stalling point. The model works, but nobody has decided what to do with its answer. A prediction that changes no action earns nothing.

An example from practice

A wholesaler with 4,000 shipments a month was losing money on failed deliveries. Managers reviewed risky orders by hand and got through roughly a quarter of the flow.

Two years of history were used to train a model on shipment failure. On a held-out sample of 9,800 orders the model had never seen, it caught 71% of actual failures at a 12% false-alarm rate. The threshold was set so that manual review still covered the same 25% of the flow — except that the model now picked those cases, rather than a manager going on instinct.

The figures above are indicative estimates for a company of this profile, not a report on a specific deployment. The measurement method is reproducible: the data is split into a training portion and a held-out portion, the rates are calculated on the held-out portion, and the result is compared against the current manual selection over the same period.

Here is how the arithmetic ran. Two months were measured before launch: how many failures the review caught under manual selection. Then the same two months after, at the same volume. The share of failures caught rose from 30% to 62%, without a single extra person on review. Some failures the model missed entirely — orders from new customers, for whom no history exists.

What the business gets

• Decisions come earlier. The risk is visible when the order is placed, not on the day it falls through. Early intervention costs less than sorting out the mess afterwards — the gap can be calculated from six months of a company's own data.
• People's attention goes where it is needed. Manual review stays, but the model builds the list for it. The volume of review is unchanged; the hit rate is higher.
• Rules move out of people's heads. Working through the data forces the criteria behind today's decisions into the open. Some of them turn out to be testable with an ordinary condition — and work without any model at all.
• Costs become calculable. Training, testing, retraining every six months: clear budget line items. The rough order of the sum for a given process can be worked out with the budget calculator.

When it can be skipped

If the rule fits in one sentence and does not contradict itself, it is cheaper to write it down as a condition. Amount over the limit, customer on the stop list, delivery date overdue — a model adds nothing here, and it still has to be maintained.

Machine learning is also a poor fit where examples are scarce. Ten cases in a year is not a sample; no pattern can be tested on it. The same goes for decisions made once a quarter: the savings from automation will never catch up with the cost of maintenance.

Text is a separate case: emails, contracts, enquiries. The usual approach there is a language model, already trained on language in general and then tuned to the task. There is no need to train a model from scratch, and process readiness can be checked faster.

What to check

First: ask not for accuracy but for a breakdown of errors. "92% correct answers" on a flow where one shipment in twenty fails describes a model that simply always says "everything is fine." The useful figures are how many actual failures were caught and how many false alarms were raised.

Second: insist on testing against data the model did not see during training. This is standard practice, but it is sometimes sidestepped, and then the figures from the demo fail to hold up on live traffic. The contract wording is simple: measurement on a held-out sample over a specified period.

Third: agree in advance on what happens to the model's answer. Who looks at the list, within what time, and what they do with each line. Without that, even an accurate prediction remains a report nobody opens.

Fourth: put retraining in the budget from the start. A model ages along with the data, and fine-tuning on fresh examples is planned work, not a warranty claim. Planning benchmark: a quality check once a quarter, a model rebuild every six to twelve months.