Why your RAG assistant gives wrong answers, and how to fix it

When a RAG assistant starts giving wrong answers, the first instinct in most companies is to blame the model and open a fine-tuning conversation. That instinct is almost always wrong. If you want to improve RAG accuracy, the model is the last place to look. The suspects, in the order we check them: how the documents were cut up, whether the right passages were retrieved, what state the knowledge base is in, how the prompt was assembled, and whether anyone can measure any of this. This note walks that list for an executive who needs to direct the work, not do it.

A one-paragraph anatomy first, because the failure points follow from it. A RAG system takes a question, searches your document base, pulls back the passages it judges relevant, and hands them to the model with instructions to answer from them. If you want the primer, we wrote RAG, explained for people who run companies. The key fact: an error anywhere upstream looks identical to the user. A confident wrong answer caused by bad retrieval is indistinguishable from one caused by the model. Most "hallucinations" in RAG systems are retrieval failures wearing a model costume.

Chunking: the answer was cut in half

Documents are split into chunks before they are indexed, and naive splitting severs answers from their context: a table from its header row, an exception from the rule it modifies, step four from steps one through three. The symptom is answers that are half right, or right for the wrong product tier, because the qualifying sentence lived in the next chunk. The fix is structure-aware chunking that respects sections, tables and headings, and it is cheap to do well. Teams skip it because the default settings of every framework produce something that works in the demo.

Retrieval: the right passage never arrived

Semantic search finds text that means roughly the same thing as the question, and that is weaker than it sounds when your company speaks in acronyms, product codes and internal jargon that the embedding model has never seen. The symptom is answers built from a plausible but wrong document: last year’s price list, the policy for the other subsidiary. The fixes are standard: hybrid search that combines semantic matching with plain keyword matching, metadata filters (date, product, region), and a reranking step that reorders candidates before the model sees them. When we are called in on an underperforming assistant, this layer is where most of the recovered accuracy comes from.

The knowledge base: the assistant faithfully repeats what is wrong

Retrieval can work perfectly and still feed the model poison. We have seen a team ship an assistant that answered from stale documents for three months, because the export job that fed the index had silently stopped, and nobody owned freshness. Duplicated policies in four versions, obsolete pages nobody deleted, drafts sitting next to approved documents: the assistant cannot tell them apart, and it will cite the wrong one with a straight face. Curation beats volume. A smaller base of owned, current documents outperforms a full SharePoint dump every time, which is a large part of what makes an internal assistant genuinely trustworthy.

Prompt assembly and permission to say no

Two quieter failure sources sit at the end of the pipeline. One is overstuffing: shoveling twenty retrieved chunks into the prompt on the theory that more context is safer, when it mostly buries the relevant passage in noise. The other is instructions that forbid failure. An assistant told to always produce an answer will produce one even when retrieval came back empty, and that answer will be invented. Allowing refusal, and requiring citations so every claim points back to a source, converts silent errors into visible ones. That principle extends beyond RAG, and we treat it as a design rule in how production systems control hallucinations.

To improve RAG accuracy durably, measure it

Every fix above is a hypothesis until you can test it. The tool is an evaluation set: one to two hundred real questions from your users with verified correct answers, run automatically on every change. It tells you your actual accuracy instead of your impression of it, and per-stage metrics tell you where the failure lives: if the right document is retrieved for ninety percent of questions but final answers are right for seventy percent, your problem is downstream of retrieval, and the reverse points upstream. Without this, teams fix whatever was in the last screenshot a stakeholder sent. Evals are the difference between a system and a demo, and for RAG they are also the difference between a targeted fix and six months of thrashing.

If your assistant is disappointing, the practical order of operations is: build the eval set, measure retrieval and answers separately, then fix chunking, retrieval and the knowledge base before anyone says the word fine-tuning. This is a few weeks of focused engineering, not a rebuild. Diagnosing and repairing exactly this kind of system is bread-and-butter work for our custom AI team, and the fastest first step is usually the eval set you should have anyway.