Text Embeddings Inference (TEI)

RAG frameworks Intermediate Open source Free

Hugging Face's fast serving solution for text-embedding, reranking, and sequence-classification models, the retrieval-side counterpart to a text-generation server.

Strengths

High performance via Flash Attention and dynamic token-based batching
Runs on CPU, NVIDIA GPUs, and Apple Silicon, with small images and fast startup
Production features: OpenTelemetry tracing, Prometheus metrics, HTTP and gRPC

Weaknesses

Some GPU paths are experimental (Turing, Blackwell, ROCm)
GPU use needs a recent CUDA
Scope is limited to embeddings and reranking, not general LLM serving

At a glance

Licence: Apache-2.0
Pricing: Free
Platforms: macOS, Linux, Docker
GPU required: No

Works with

BGE-M3
Qwen3-Embedding 0.6B
Qwen3-Reranker 0.6B

Related guides

Building a local RAG system
Improving RAG retrieval quality

Glossary

RAG
Embedding
Vector database
Reranker

Entry last verified 18 August 2026.