llama-swap
A small proxy that sits in front of llama.cpp and compatible servers and swaps models in and out on demand, so a single machine can serve many models through one OpenAI-compatible endpoint without holding them all in memory at once. You list the models in a config file; it loads the one a request asks for and unloads others to stay within VRAM.
Strengths
- Single Go binary, zero external dependencies, minimal overhead
- Works with llama.cpp, vLLM, whisper.cpp, and ComfyUI backends
- Automatic model unloading via a configurable idle timeout
Weaknesses
- Not an inference engine itself, so it needs a backend server to be useful
- Config-driven YAML setup for each model
- Small single-maintainer project
At a glance
- Licence
- MIT
- Pricing
- Free
- Platforms
- macOS, Windows, Linux, Docker
- GPU required
- No
Links
Related guides
Glossary
Entry last verified 18 August 2026.