Building a local RAG system
How to let a local model answer questions about your own documents, the pieces involved, and the two routes: an all-in-one app, or assembling it yourself.
What you’ll learn
What goes into a retrieval-augmented generation system, how the pieces fit together, and whether to reach for a ready-made app or build one from parts.
The concept
A model only knows what it was trained on. RAG is how you let it answer from your own documents instead, without retraining it. The idea is to find the most relevant passages at the moment you ask a question, and hand them to the model along with the question.
A local RAG system has four parts:
- Chunking. Your documents are split into passages of a few hundred words.
- Embedding. Each chunk is turned into an embedding by an embedding model such as BGE-M3, and stored.
- A vector store. A database like Chroma holds the embeddings and finds the closest ones to a query.
- Generation. The retrieved chunks and your question go to a language model, such as Qwen3 8B, which writes the answer.
Doing it
There are two honest routes, and the right one depends on how much control you want.
The quick route is an all-in-one app. AnythingLLM or Open WebUI bundle the embedder, vector store, and chat interface, so you point them at your documents and start asking. This is the sensible starting point for most people, and it runs fully locally with a backend like Ollama.
The build-it-yourself route uses a framework like LlamaIndex to wire the parts together in code. It is more work, but you control chunking, retrieval, and prompting, which is where RAG quality is won or lost. Choose this when the bundled defaults are not good enough.
What can go wrong
- Poor retrieval. If the wrong passages are retrieved, the answer will be wrong no matter how good the model is. Most of the effort in RAG goes here: chunk sizes, the embedder, and how many passages you retrieve.
- Overflowing the context. Retrieved passages plus the question have to fit in the model’s context window. Retrieve too much and you run out of room, or bury the useful passage.
- Expecting RAG to fix a weak model. RAG supplies knowledge, not reasoning. A small model still reasons like a small model over the passages it is given.
- Forgetting the documents change. Re-embed when your source material updates, or the system answers from stale content.
Next steps
Start with an all-in-one app to get a feel for it, then read about embedding models and vector databases when you want to improve retrieval quality.