Embedding models for search and RAG
Not every local model generates text. Embedding models turn text into vectors, which powers semantic search and RAG. How they work and how to choose one.
What you’ll learn
What an embedding model does, why it is a different kind of model from a chat model, and how to pick one for search or RAG.
The concept
An embedding model does not generate text. It turns a piece of text into a vector, a list of numbers that captures its meaning, so that similar texts have similar vectors. This is what lets you search by meaning rather than by keyword, and it is the retrieval half of any RAG system.
The workflow is simple: embed your documents once and store the vectors in a database like Chroma; then, for each query, embed the query and find the closest stored vectors. Because embedding models are small, this is cheap and runs comfortably on a CPU.
Choosing one
A few properties matter when choosing:
- Languages. If your content is English, a small model like Nomic Embed Text is fast and more than adequate. For multilingual content, BGE-M3 or Qwen3-Embedding handle many languages well.
- Retrieval quality. Larger embedding models generally retrieve better, at some cost in speed and storage. The Qwen3-Embedding family, for example, tops multilingual retrieval leaderboards at its larger sizes.
- Dimensions. The vector length affects storage and comparison speed. Some models let you choose it, trading a little quality for a smaller footprint.
- Licence. Most good embedding models are permissively licensed. BGE-M3 is MIT, and the others here are Apache 2.0, so commercial use is straightforward.
Doing it
- Match the model to your content, mostly on language, using the guide above.
- Keep the embedder consistent. You must query with the same model you indexed with, so if you change embedder, you re-embed everything.
- Store the vectors in a vector database suited to your scale, from Chroma for prototypes upward.
What can go wrong
- Mixing embedders. Vectors from different models are not comparable. Index and query with the same one, and re-embed if you switch.
- Chasing the top of a leaderboard. A domain-tuned or well-matched smaller model often beats a larger general one on your specific content. Test on your own data.
- Forgetting to re-embed when documents change. Stale vectors return stale results.
Next steps
To put an embedder to work, follow building a local RAG system. The embedding models and vector databases in the catalogue cover the options.