Serving models at scale

What you’ll learn

What actually changes when you move from a single-user setup to serving a model to many people, why the tools are different, and how to think about throughput.

The shift from running to serving

The tools that make local AI pleasant for one person, like Ollama, are built to run a model for a single user at a time. Serving is a different problem. When several people, or an application handling many requests, hit the same model at once, the question stops being “can it run?” and becomes “how many requests per second, at what latency, on this hardware?”

Serving engines are designed around keeping the GPU busy. The key techniques:

• Continuous batching merges incoming requests so the GPU processes many at once rather than idling between them, which is the single biggest lever on throughput.
• Efficient memory management for the context of each request lets a card serve more concurrent users without running out of memory.
• An OpenAI-compatible API means your applications talk to your own server exactly as they would to a commercial API, so switching is often a change of URL.

The engines

• vLLM is the common default. It has broad model and hardware support, strong throughput, and a drop-in OpenAI-compatible API. Reach for it first when you need to serve a model to several people.
• SGLang shines when your workload reuses a lot of context, such as retrieval-augmented chat, where its cache sharing pays off.
• For NVIDIA-only infrastructure squeezing maximum performance, TensorRT-LLM goes further, at a real cost in setup complexity.

Doing it

1. Choose the model and quantisation you will serve, sized to your GPUs. A Qwen3 32B on a 48GB card, or a Llama 3.3 70B across two, are common shapes.
2. Stand up a serving engine, starting with vLLM, exposing the OpenAI-compatible endpoint.
3. Load-test with realistic traffic. Measure throughput and latency at the concurrency you actually expect, not a single request.
4. Tune batching and memory settings to the sweet spot between throughput and latency for your users.

What can go wrong

• Benchmarking with one request. Serving performance is about concurrency. A single-request test tells you almost nothing about how it behaves under load.
• Ignoring latency for throughput. Maximum throughput settings can make each user wait longer. Balance the two for your use.
• Underestimating the operational load. A served model is infrastructure to run, monitor, and keep up. Be honest about whether that is worth it versus an API, which is a fair comparison to make.

Next steps

Serving is only half of a team setup. See multi-user access and gateways for routing, authentication, and cost tracking, and planning hardware for a team for the machines behind it.