Most AI assistant projects that disappoint a client are not failing because of the underlying language model. They are failing because the retrieval step is feeding that model the wrong, incomplete, or outdated information before it ever generates a response. Retrieval augmented generation, RAG, gets sold as the fix for AI hallucination, and it is, but only if the retrieval half of the system is built with the same care as the generation half. Most implementations are not, and the result is an assistant that sounds confident while quietly answering from the wrong document.
The Part Everyone Skips
The pitch for RAG is straightforward: instead of relying on a model's training data, ground every answer in your actual documents by retrieving relevant passages and feeding them into the prompt. The generation step, where the model turns retrieved passages into a coherent answer, gets most of the attention because it is the visible, demoable part. The retrieval step, which decides what those passages even are, gets built quickly and rarely revisited, even though it determines the ceiling on how good the final answer can possibly be. A model given the wrong context will produce a fluent, well-formatted, entirely wrong answer, and it will do so with the same confidence as a correct one.
Where Retrieval Quietly Fails
Chunking Strategy Determines What Gets Found
Documents get split into chunks before they are indexed for retrieval, and the chunking strategy has an outsized effect on quality that most teams never revisit after initial setup. Chunks that are too small lose surrounding context and get retrieved without the information needed to interpret them correctly. Chunks that are too large dilute the specific answer among unrelated surrounding text, making it harder for the retrieval step to rank the right passage highly. Fixed-size chunking, the default in most tutorials, is rarely the right choice for real business documents like contracts, product specs, or support histories that have their own internal structure.
Semantic Search Alone Misses Exact Matches
Vector similarity search finds passages that are conceptually related to a query, which is powerful but not sufficient on its own. A customer asking about a specific product SKU, error code, or policy number needs an exact match, not a conceptually similar one, and pure semantic search frequently ranks a related but wrong passage above the exact answer sitting elsewhere in the index. The systems that perform reliably combine semantic search with keyword-based retrieval and re-ranking, rather than relying on embeddings alone.
Stale Indexes Feeding Outdated Answers
A retrieval index is only as current as the last time it was rebuilt, and a surprising number of production RAG systems index documents once at launch and never establish a reliable pipeline for keeping that index current. An assistant confidently quoting a pricing page or policy document that was updated three months ago is a direct, measurable failure of the retrieval pipeline, not the model.
Building Retrieval That Actually Holds Up
Test Retrieval Independent of Generation
We evaluate the retrieval step on its own, before ever looking at what the language model does with it, by building a test set of real questions with known correct source passages and measuring whether retrieval surfaces the right passage in the top results. This isolates retrieval quality from generation quality and makes it possible to improve one without the other masking the problem.
Re-rank Before Generation
Initial retrieval typically pulls a wider set of candidate passages than get passed to the model, and a re-ranking step that scores those candidates against the specific query before final selection consistently improves answer quality more than almost any other single change, because it corrects for the imprecision inherent in first-pass retrieval.
Automate Index Freshness
Source documents that change need an automated pipeline that detects the change and updates the index, not a manual process someone forgets to run. We build this as a standard part of any RAG system, tied directly to the systems where the source content actually lives, whether that is a CMS, a knowledge base, or an internal database.
The Real Cost of Getting This Wrong
An AI assistant that occasionally gives a wrong answer with full confidence is worse for a business than no assistant at all, because it erodes trust in a way that is hard to win back and hard to detect early, since the wrong answers often sound just as polished as the right ones. Getting retrieval right is not a nice-to-have refinement layered on top of a working system. It is the foundation the rest of the system depends on.
MAPL TECH builds AI assistants and internal tools with retrieval pipelines engineered for accuracy, not just demos. Explore our automation and AI services or get in touch to talk through what a reliable RAG implementation looks like for your data.
