Non-determinism: what it means in plain terms
Why this matters
- Understand why a demo runs smoothly while answers drift apart in production
- Set a vendor a verifiable quality threshold instead of a promise that "the model answers correctly"
- Know which processes can be handed to a model outright and which need output checks
What is non-determinism in plain terms
A calculator returns the same number for the same sum every time. An experienced lawyer asked the same question twice will write two letters in different words. The meaning matches; the order of paragraphs and the turns of phrase do not.
Language models are closer to the second case. The answer is assembled one piece of text at a time, and at each step one option is chosen from several suitable ones, with an element of randomness. Hence different answers to the same request.
The spread is uneven. Wording changes almost always. The meaning of the answer changes noticeably less often. Specific fields such as an amount, a date, or a contract number change least often of all — and those are exactly what gets watched.
A second source of spread lies outside the request. The provider updates the model and behaviour changes without any announcement. That is why the model version is fixed in writing and quality is measured again after every update.
How it works
- The model version is fixed. The contract names a specific version, not a family name. Moving to a new one happens through a fresh measurement, not automatically.
- The response format is defined. The model is required to return fields: amount, date, counterparty, request type. Free text remains only where a person will read it.
- Temperature is lowered where consistency matters. Document processing stays at the low end of the scale; customer replies sit higher, or the text comes out lifeless.
- A reference sample is built. Between 200 and 300 typical requests, labelled by hand. The same sample serves as acceptance testing and is rerun after every change to a prompt or version.
- A threshold and an output check are put in place. For example: no more than 3% of documents processed with an error in key fields, measured on a sample of 300 documents over two weeks. Anything below the confidence threshold goes to a person.
Stability is checked separately. The same request is run 30 times in a row to see how far the key fields diverge. If the fields diverge, it is too early to automate. If only the wording diverges, the process is ready.
An example from practice
Processing incoming invoices, roughly 4,000 documents a month. The figures below are planning estimates for a process of this profile, not a report on a completed rollout; the method is spelled out so the measurement can be repeated on in-house data.
The demo showed ten invoices and all ten came out right. In production the answers drifted: on some documents the VAT amount landed in the "total" field, and the wording of the comment changed on every run.
Next, a sample of 300 invoices was labelled by hand and run through three times in a row. Only one number was counted: the share of documents where all four key fields matched the labelling on all three runs. Unstable fields were sent for automatic validation against the contract register, and documents below the confidence threshold went to manual review.
The spread in wording is still there. It never reaches the accounting system, because only verified fields go through.
What this gives a business
- Quality becomes a measurable quantity. The threshold, the sample, and the measurement period go into the contract before launch. After that, a dispute about quality is settled by rerunning the sample rather than by email.
- Updates stop being a surprise. A fixed version and a ready sample make it possible to move to a new model in a single measurement, instead of a month of complaints.
- Flexibility stays where it is useful. That same spread is what lets a model handle phrasings that no rulebook anticipated. Rigid software cannot do that.
- It becomes clear what an agent can take on outright. Steps with a verifiable result can be automated fully; steps that call for judgement need a check on the output. This readiness checklist helps gauge which steps of a process are ready.
When this can be ignored
Some tasks are harmless by nature. A draft letter, a set of headline options, a meeting summary for personal use — a person reads the result and edits it.
The conversation is also unnecessary where the model works as a suggestion inside an interface: an employee sees the proposal and makes the call. Limits become necessary the moment an answer reaches a customer or enters an accounting system without a person in the loop.
What is worth checking
First: ask for the demo sample to be run twice and for both results to be shown side by side. Divergence on key fields is visible immediately, and it is exactly what determines how much manual review will remain.
Second: work out where the line runs between spread and hallucination. Different wordings of one correct fact are a normal working situation. Different facts in answers to the same request are a signal that the model did not have enough data.
Third: find out what the vendor does when the provider updates the model. The answer "the reference sample is rerun and compared against the threshold" can be verified. The answer "quality is monitored" cannot.
Frequently asked questions
What is non-determinism in plain terms?
It is a model's tendency to answer the same request differently each time. The wording changes almost always, the meaning less often, and key fields such as amounts and dates least often of all. This is a property of the technology itself, not a defect. The job is to keep the spread inside limits set in advance.
Why does a model give different answers to the same request?
There are three reasons. The answer is assembled one word at a time, and at each step the choice is made at random among several suitable options. The order of computation on the provider's servers shifts with load, which moves the result in small ways. The third reason is model updates: the provider rolls out a new version and behaviour changes without notice. Hence the rule that the version must be fixed in the contract.
Does setting temperature to zero help?
Temperature is a setting that makes word choice more predictable. At zero the spread in wording drops sharply, but word-for-word repeatability is still not guaranteed: dependence on server load and model version remains. Zero temperature also drains the life out of the text, so support and sales teams usually raise it, while document processing keeps it low.
How can a vendor's work be accepted if the answers differ every time?
Acceptance runs on a sample, not on a single demo. Between 200 and 300 typical requests are labelled by hand, run through the system two or three times in a row, and the share of correct key fields is counted. The contract states the threshold, the sample size, and the measurement period. A test like this can be repeated on in-house data and does not depend on one lucky demo run.
Does non-determinism get in the way of automation?
It does wherever an answer reaches a customer or an accounting system unchecked. The fix is a set of limits: a strict response format, automatic field validation, a list of blocked topics, and handover to a person when confidence is low. With those limits in place the spread stays contained and never reaches the recipient. Those same limits are what separates a working setup from a demo.
Let’s discuss your project?
Tell us about your process — we’ll suggest where AI pays off fastest.
Related articles
AI hallucinations: why models confabulate and how to catch it
A hallucination is a confident answer that looks plausible but does not match reality: an invented link, a made-up number, or a clause that appears in no regulation.
What Is an AI Agent, and How Does It Differ from a Chatbot
An AI agent is a program built on a language model that receives a goal stated in plain words, chooses its own steps, calls the tools it needs, and carries the task through to a result.