Adding a reranker to your RAG pipeline

What you’ll learn

What a reranker is, why it so often improves a RAG system more than anything else, and how to add one, with the small local models that do the job.

The problem it solves

Retrieval in a RAG system is fast but approximate. An embedding search finds passages that are broadly similar to your question, but “similar” is not the same as “answers it”. A common failure is that the right passage was retrieved but ranked too low to make the cut, so the model never sees it. Better embeddings help, but only so much.

How a reranker fixes it

A reranker is a second, more careful model. Where retrieval compares a query and a passage only through their separate embeddings, a reranker reads the query and each candidate passage together and scores how well that passage actually answers the query. It is slower per comparison, which is why you do not use it for the whole corpus, only to re-order a shortlist.

The pattern is a two-stage retrieve-then-rerank:

1. Retrieve a generous shortlist from your vector store, say the top 20 to 50 passages.
2. Score each with the reranker against the query.
3. Keep the best few, say the top 3 to 5, and pass those to the model.

Which models to use

Rerankers are small, so this costs little memory:

• Qwen3-Reranker 0.6B is a strong, current, multilingual choice under Apache 2.0, with larger 4B and 8B siblings if you want more quality.
• BGE-reranker-v2-m3 is a mature, widely-supported default that has served local RAG pipelines for a while.

How to run it

The RAG frameworks make this a small change: LlamaIndex and LangChain both have a reranking step you can drop in after retrieval. To serve the reranker itself efficiently, Text Embeddings Inference runs reranker models as well as embedding models behind a fast API.

What can go wrong

• Reranking too few candidates. If you only retrieve five passages and rerank those, there is little for the reranker to fix. Retrieve a generous shortlist first.
• Ignoring the latency. Every candidate is a model call, so reranking 50 passages adds real time. Tune the shortlist size to balance quality against speed.
• Expecting it to fix bad retrieval. A reranker re-orders what retrieval found; if the right passage was never retrieved, it cannot help. Fix chunking and embeddings too.

Next steps

Reranking is one lever among several in improving RAG retrieval quality. The models here pair naturally with a local embedding model such as BGE-M3.