Improving RAG retrieval quality

What you’ll learn

Where RAG quality is actually won: the handful of levers, chunking, hybrid search, reranking, and evaluation, that turn a system that retrieves roughly the right thing into one that retrieves the right thing.

The one idea to hold onto

If a RAG system gives poor answers, the cause is usually retrieval, not the model. The model can only work with the passages it is given, so if the right passage is not among them, no model will save the answer. That means most of your effort should go into retrieving better passages, not into a bigger model.

Chunking

How you split documents into passages has an outsized effect. Two failures are common: chunks so large that the useful sentence is buried among irrelevant text and dilutes the embedding, and chunks so small that they lose the context needed to make sense.

Practical guidance:

• Start around a few hundred tokens per chunk, and adjust to your content.
• Add a little overlap between chunks so a point split across a boundary is not lost.
• Respect structure where you can. Splitting on headings, paragraphs, or sections keeps related text together far better than a blind character count.

Hybrid search

Pure semantic search, comparing embeddings, is powerful but has a weakness: it can miss exact terms, product codes, names, or rare keywords that matter. Keyword search has the opposite profile. Combining them, often called hybrid search, gets the best of both.

Some embedding models, such as BGE-M3, support this directly by producing both dense and sparse representations. Otherwise, many vector databases like Qdrant can combine a keyword index with vector search and merge the results.

Reranking

Retrieval is usually tuned to cast a wide net: fetch more candidate passages than you need, quickly. A reranker then does a second, more careful pass, scoring those candidates for relevance to the query and reordering them, so the best few go to the model. Retrieving, say, twenty candidates and reranking down to the top five often lifts quality noticeably for little extra cost, because the reranker looks at each candidate more closely than the first retrieval did.

Evaluation

You cannot improve what you do not measure. Build a small set of representative questions with known good answers, and check whether the right passages are being retrieved and whether the final answers are correct. This turns tuning from guesswork into something you can iterate on, and it is what tells you whether a change to chunking or search actually helped.

What can go wrong

• Blaming the model for a retrieval problem. Check what was retrieved before reaching for a bigger model.
• Retrieving too much. Stuffing many passages into the prompt can overflow the context window and bury the useful one. More is not better.
• Never measuring. Without evaluation, you are tuning blind, and it is easy to make changes that feel better but retrieve worse.

Next steps

Revisit building a local RAG system for the overall shape, and the embedding models and vector databases in the catalogue for the components that do the retrieving.