Running large open models: the server-class tier

What you’ll learn

What changes when a model is too big for a single GPU, the forms of parallelism used to spread it across many, how quantisation keeps the hardware bill down, and the software that ties it together.

The wall is memory

The defining constraint for large open models is memory, not compute. A model’s weights have to be resident in memory to run, and for a mixture-of-experts model that means all the experts, even though only a few are active per token. So the total parameter count, not the active count, sets the memory bill. When that bill exceeds what one GPU holds, you have no choice but to spread the model across several, and the way you do that is the heart of this tier.

The ladder

It helps to see server-class as a ladder rather than a single thing:

• A single 80GB GPU. gpt-oss-120b is over 100 billion parameters, yet its mixture-of-experts design and native 4-bit format let it fit on one 80GB card. This is the gentlest rung: bigger than any consumer model, but no distributed systems required.
• A multi-GPU node. DeepSeek-R1 at 671 billion parameters needs several GPUs in one machine, connected by fast interconnect, working together on each request.
• A multi-node cluster. Kimi K3, at 2.8 trillion parameters and roughly 1.4TB even in a native 4-bit format, needs many GPUs across several machines. Its maker suggests a configuration of 64 or more accelerators.

Each rung up is a real increase in difficulty. Know which one your model actually needs.

Parallelism, briefly

Spreading a model across GPUs is done in a few complementary ways, and serving engines combine them:

• Tensor parallelism splits each layer’s maths across GPUs, so they compute one layer together. It is fast but demands very high bandwidth between the GPUs, which is why it is usually kept within a single machine.
• Pipeline parallelism puts different layers on different GPUs or machines, passing activations along like a production line. It tolerates slower links, so it is used to cross between machines.
• Expert parallelism distributes a mixture-of-experts model’s experts across GPUs, which suits exactly the very large MoE models at this tier.
• Data parallelism runs whole replicas of the model in parallel to raise throughput once a single copy fits.

You rarely choose these by hand in detail; you tell a serving engine how many GPUs and nodes you have, and it applies a sensible combination. But knowing the vocabulary explains why interconnect, covered in the next guide, matters so much.

Quantisation at scale

The other lever is quantisation. At this tier it is not only about fitting a model on smaller hardware, but about fitting it on fewer GPUs, which is a direct cost saving. Formats like FP8 and MXFP4 are common, and some models, including gpt-oss and Kimi K3, ship natively in a 4-bit format, so the quantised version is the intended one rather than a compromise. As always, verify that the quality holds for your use rather than assuming.

The serving software

The engines from serving models at scale are what apply all of this. vLLM and SGLang support tensor and pipeline parallelism across GPUs and nodes, and on NVIDIA hardware TensorRT-LLM squeezes out the most performance. Larger deployments increasingly separate the prompt-processing and token-generation phases onto different hardware, since they have different bottlenecks, an approach called disaggregated serving. Managing the KV cache, the memory that holds each request’s context, well is often what decides real throughput.

What can go wrong

• Underestimating interconnect. Tensor parallelism across GPUs with slow links performs terribly. The connection between GPUs matters as much as the GPUs.
• Naive multi-node. Spreading a model across machines without fast networking between them can be slower than a smaller model on one machine.
• KV cache blow-up. Long context multiplied by high concurrency consumes enormous memory for the cache, which can dominate the model’s own footprint. Plan for it.

Next steps

The hardware and networking underneath all of this, the GPUs, the interconnect, and the rest of the stack, is covered in enterprise hardware and networking.