Hallucinations are a system design problem, not a model problem

Ask a vendor how to prevent AI hallucinations in a business system and you will usually hear some version of "the models have gotten much better." True, and not an answer. Language models still produce confident, fluent, wrong statements, and they always will at some rate, because a model is a machine for generating plausible text, not verified text. The teams running AI reliably in production accepted this early. They stopped treating hallucination as a model defect to wait out and started treating it as a failure mode to design around, the way engineers treat network outages or human data-entry errors.

That reframing matters because it moves the question from "which model lies least" to "what happens in my system when the model is wrong." The first question has a different answer every quarter. The second has a stable answer you can build, and it is the difference between a demo you admire and a system you trust with customers.

Why "the model improved" is not a strategy

Each model generation hallucinates less on benchmarks than the last. This helps less than you would hope, for two reasons. First, volume amplifies whatever rate remains: an assistant that answers a thousand questions a week and is wrong two percent of the time delivers twenty wrong answers a week, fluently, to employees with no way to tell. Second, progress removes the absurd errors first and keeps the dangerous ones. What survives in newer models is the subtle miss: a plausible but wrong figure, a date shifted by a year, an invented clause in an otherwise accurate contract summary.

So the model is the floor, not the plan. The plan is a small set of design moves that production systems share. None of them is research. All of them require deciding that reliability is a requirement rather than a hope.

How production systems prevent hallucinations from reaching users

The single most effective move is to stop asking the model to answer from memory. In a retrieval-augmented system, the model is handed relevant passages from your own documents and instructed to answer from them or say that it cannot. We covered the architecture in plain terms in RAG, explained for people who run companies. Grounding cuts invention sharply on its own, and citations do the rest: when every answer links to the paragraph it came from, a wrong answer becomes checkable in seconds instead of trusted by default. In our builds, citation is non-negotiable. An answer without a source is an opinion.

Grounding has failure modes of its own, and retrieval is the usual one. When a grounded assistant is wrong, the cause is more often the pipeline than the model, which is a subject of its own.

Constrain what the model is allowed to say

Free text is the hardest output to verify, and much business AI does not need it. If the task is extracting invoice fields, classifying a claim, or routing a ticket, force the model into a schema: defined fields, allowed values, formats that ordinary code can validate. A model that must answer with one of six categories cannot invent a seventh. Validation rules then catch what slips through: totals that do not sum, dates outside plausible ranges, references to entities that do not exist in your database. Those outputs get rejected and retried automatically, and the user never sees them. This unglamorous machinery is most of what makes structured extraction dependable.

Put a human where the stakes justify one

Reviewing everything quietly kills the return, and reviewing nothing is negligence. The design question is where errors are expensive. Confidence thresholds do the sorting: outputs the system is sure of flow straight through, while low-confidence cases, high-value amounts, and anything touching a regulated decision land in a review queue. A payment above a threshold, a customer-facing commitment, a medical or legal statement: a person approves. The system earns its keep by making that review fast, showing the source passage, the proposed output, and the reason it flagged the case.

Treat refusal as a feature

A system that answers every question is broken by design. "I cannot find this in the documentation, here is who to ask" is a better answer than a fluent guess, and users learn to trust an assistant that visibly knows its limits. Refusal has to be designed: the prompt must permit it, the retrieval layer must signal when nothing relevant was found, and the product must route the refusal somewhere useful. More than one assistant we audited had been forced to always answer because a stakeholder found refusals embarrassing. Those are the systems that end up quoted in a complaint.

Measure the rate, then manage it

Everything above only holds if you can see the error rate move. That takes an evaluation set: a few hundred real questions with verified answers, run against the system on every change, so that hallucination stops being an anecdote and becomes a number with a trend. It is the unglamorous work that separates systems from demos, and it is also what lets you answer the board when they ask how often the assistant is wrong. Without it, you are guessing exactly like the model is.

None of this is exotic. It is ordinary engineering applied to an unusual component, and it is buildable in weeks, not quarters. If you are deciding whether an AI system can be trusted with customer-facing or compliance-relevant work, the trustworthy part is the system around the model. Building that surrounding system, grounded, constrained, measured, and honest about what it does not know, is exactly what our custom AI practice exists to do.