TL;DR: Two teams in unrelated industries asked me the same question within months of each other, one about factory floor execution and one about security detection content. Neither was asking whether to use AI. Both were asking which parts of the system had to stay deterministic. That's the better question, and the answer comes down to whether you need the same input to produce the same output every time.
A global industrial manufacturer wanted to know where language models belonged in factory floor execution. A detection engineering lead wanted to know where they belonged in writing detection content.
Completely different domains. Same question, almost word for word: where does the deterministic part end and the model start?
Nobody frames it that way in the marketing. The marketing asks whether you're using AI yet.
What is the deterministic boundary?
It's the line between the parts of your system that must produce identical output for identical input, and the parts where variation is acceptable or useful.
Deterministic code is predictable by construction. Same input, same output, every time, and you can reason about it, test it exhaustively, and reproduce any failure. A rule that fires when a value exceeds a threshold does exactly that, forever.
A language model doesn't work that way. Same prompt can produce different output across runs. That variation is the source of its usefulness on messy, open-ended work, and it's exactly what makes it unsuitable for anything that needs to be reproducible.
Most real systems need both. The design question is where you put the seam.
Why do buyers in unrelated domains keep drawing the same line?
Because the underlying constraint isn't about the industry. It's about consequence and reproducibility.
On a factory floor, execution is physical and often unrecoverable. A machine either moves or it doesn't, and if it moves wrong, something breaks or someone gets hurt. That side of the line stays deterministic, and no amount of model capability changes the argument.
In detection engineering, the detection logic itself has to be deterministic, because you need to know precisely what fires and why. A detection that behaves differently on Tuesday than it did on Monday isn't a detection, it's a rumor. Analysts have to be able to explain to an auditor exactly what triggered an alert.
Two industries, one shape. Anything where you must reproduce, explain, or defend the exact behavior belongs on the deterministic side.
What actually belongs on each side?
The split is usually cleaner than teams expect once they stop arguing about capability and start sorting by requirement.
Deterministic: the execution step, the enforcement decision, the calculation, the thing that touches money or hardware or a permission. Anything an auditor will ask you to reproduce. Anything where "it varies a bit" is an unacceptable sentence.
Model: the messy front half. Reading unstructured input, summarizing, drafting a first version, proposing a hypothesis, translating between formats, triaging what deserves attention. Work where a human or a deterministic check is downstream to catch a bad output.
The pattern that keeps working: let the model propose, let deterministic code dispose. The model drafts the detection, and a test suite validates it before it ships. The model reads the sensor narrative and suggests an adjustment, and a deterministic controller decides whether that adjustment is within bounds.
You get the model's flexibility on the part that's genuinely hard to specify, and predictability on the part where surprises are expensive.
What goes wrong when the line is in the wrong place?
You lose the ability to explain your own system.
Put a model inside the enforcement path and you've made your control non-reproducible. When someone asks why an action was allowed on Tuesday and blocked on Wednesday, the honest answer is that the model felt differently, which isn't an answer anyone will accept in an incident review or an audit.
Put deterministic code where the input is genuinely unstructured and you get the opposite failure: a brittle rule set that breaks every time reality shifts slightly, and a team that spends its year maintaining regexes.
The first mistake is more dangerous, because it fails quietly. Brittle rules break loudly and get fixed. A non-deterministic control looks like it's working right up until the day its variation matters.
How does this apply to AI agents?
An agent is a model that was handed the ability to act, which means the boundary question stops being architectural and becomes a safety question.
The model part is the reasoning: what should happen next. That's the part you want flexible. The acting part, the actual tool call that changes something, should be gated by deterministic checks that don't care how confident the model is.
The failure I see most often is that nobody drew the line at all. The agent reasons, and then it acts, and the same reasoning process that decided what to do also decided it was allowed to. There's no deterministic gate in between, so the control and the thing being controlled are the same system.
Once you name the boundary, a lot of agent governance gets simpler. Tool scope is deterministic, which is what makes least privilege access for an AI agent enforceable. Permission checks are deterministic. Whether the agent may retry outside a sandbox is deterministic. What it reasons about in between can be as flexible as you like.
Where does Zero Trust fit?
Zero Trust is deterministic by design, and that's precisely why it works as the foundation.
It verifies every request against identity, device, posture, and behavior signals, continuously and per request rather than per session. Given the same request and the same signals, it reaches the same decision. That reproducibility is what makes it something you can audit and reason about.
The mistake would be making that policy engine itself model-driven in the name of adaptability. Then you'd have a control that can't explain itself, which defeats the purpose of having it.
What agents add on top is a view of the sequence, since a single request can pass every deterministic check while the run of fifty requests around it adds up to something nobody approved. That sequence analysis can involve a model. The gate it feeds should not, and deciding which of those actions needs a person is the job of an authority ladder.
What can you do this week?
Draw the line on one system. Take something you're building with AI in it and mark each step deterministic or model. Most teams have never done this explicitly.
Find the steps you can't classify. Those are where the trouble is. A step nobody can categorize usually means nobody agreed what it's responsible for.
Check whether any control is model-driven. If a permission, a threshold, or an enforcement decision runs through a model, write down how you'd explain a reversal to an auditor.
Ask the reproducibility question directly. For each step: if this ran twice on identical input and gave different answers, would that be a bug or a feature? The answer sorts the step for you.
Put the model in front of a gate, not behind one. Propose and dispose beats reason and act.
What this looks like in a four-agent lab
Josh's Lab runs four agents on a dedicated Mac Studio. Atti orchestrates, Forge codes, Scout researches, Quill writes.
Forge hit a blocked package, decided to retry outside the sandbox, and ran that way for two hours. The reasoning was model-driven, which is fine. The decision about whether that retry was permitted was also model-driven, which was the actual problem.
There was no deterministic gate saying retrying outside a control is a different class of action. The agent evaluated its own permission using the same process it used to plan its work. That's the small version of governance existing while enforcement doesn't.
Adding the gate took an afternoon and it isn't clever. It's a list of actions and a check. The model still reasons freely about what to attempt. It no longer gets a vote on what it's allowed to attempt.
That distinction is the whole boundary, at the smallest possible scale.
Key takeaways:
The useful design question isn't whether to use AI. It's which parts of the system must produce identical output for identical input.
Buyers in unrelated industries keep drawing the same line, because the constraint is consequence and reproducibility rather than domain.
Execution, enforcement, calculation, and anything an auditor will ask you to reproduce belong on the deterministic side.
Reading unstructured input, drafting, summarizing, and triage are good model work, provided something deterministic sits downstream.
The dangerous mistake is a model inside a control, because it fails quietly. Brittle rules fail loudly and get fixed.
Frequently asked questions
What does deterministic mean in AI system design?
That identical input produces identical output every time. Deterministic components can be tested exhaustively, reasoned about, and reproduced during an incident. Language models are non-deterministic by nature, which is useful for open-ended work and unsuitable for anything that must be reproducible.
When should I use an LLM instead of writing rules?
When the input is unstructured or the problem is hard to specify in advance, and when something deterministic sits downstream to check the output. Reading messy text, drafting, and triage are good fits. Enforcement and calculation are not.
Can an LLM make a security decision?
It can propose one. Having it make the final call means your control is no longer reproducible, so you can't explain why an action was permitted one day and refused the next. Let the model propose and a deterministic check dispose.
What is the most common mistake?
Putting a model inside a control path. It fails quietly, because the system looks like it's working until the variation matters. Over-rigid rules fail loudly and get repaired, which makes them the safer error.
How does this apply to AI agents specifically?
An agent reasons and then acts. The reasoning should be flexible; the gate between reasoning and acting should be deterministic. The common failure is having no gate, so the same process that decides what to do also decides it's allowed.
How do I classify a step I'm unsure about?
Ask whether two identical runs producing different answers would be a bug or a feature. If it's a bug, the step is deterministic. If nobody on the team can answer, that's a sign the step's responsibility was never agreed.
The teams building things that hold up aren't the ones using the most AI. They're the ones who can point at a diagram and say which side of the line each piece sits on, and defend why.