Enterprise hardware and networking

What you’ll learn

The hardware that server-class self-hosting actually runs on, why the connections between GPUs and machines matter as much as the GPUs themselves, and the infrastructure around it that is easy to forget until it bites.

The GPUs

At this tier you are on data-centre accelerators, not consumer cards. The common choices are NVIDIA’s H100 and its successors, the H200 and the Blackwell B200, and on the AMD side the Instinct MI300X and MI350 series. Two numbers matter most: the amount of high-bandwidth memory, typically 80GB or more per GPU, which sets how much model fits per card, and the memory bandwidth, which largely sets how fast it runs. These are also the parts in shortest supply and highest demand, which shapes both price and lead time.

For a genuinely large model you need several of them working together, which is where the connections between them become the story.

Interconnect is the bottleneck

This is the point most workstation experience does not prepare you for. When GPUs share the work on a single request, they exchange data constantly, and the speed of that exchange, not the GPUs’ own speed, often sets the ceiling.

• Within a machine, GPUs are linked by NVLink and NVSwitch on NVIDIA systems, which are far faster than the ordinary PCIe bus. This is why tensor parallelism, which needs the most bandwidth, is kept inside one machine.
• Between machines, nodes are linked by a high-speed fabric, typically InfiniBand or high-performance Ethernet with RoCE. This is slower than in-machine links, which is why crossing between machines is done with pipeline parallelism, which tolerates it.

The practical lesson: a cluster is only as good as its slowest relevant link. Fast GPUs joined by slow connections underperform badly, and the interconnect is a first-order design decision, not an afterthought.

The rest of the stack

Around the GPUs and their links sits everything that makes a cluster a running system:

• Storage and loading. Model weights are large, hundreds of gigabytes to over a terabyte, so fast storage and a sensible loading strategy matter to start-up time.
• Power and cooling. A rack of accelerators draws serious power and produces serious heat. This is a data-centre or colocation concern, measured in kilowatts, with real implications for where the hardware can even live.
• Orchestration. Kubernetes and similar tools schedule and manage the serving workloads, alongside the serving engines and observability from the earlier guides.

None of this is exotic to infrastructure teams, but it is a different discipline from running a model on a workstation, and worth being clear-eyed about before committing.

Buy, rent, or colocate

The economics at this tier deserve an honest look, because the numbers are large.

• Rent. Cloud providers and specialist GPU clouds rent this hardware by the hour. For spiky demand, for getting started, or for avoiding a large capital outlay, this is usually the right first move, and often the right permanent one.
• Colocate. Owning the hardware but housing it in a data centre gives you control of the machines without building a facility.
• Build. Running your own facility makes sense only at real, sustained scale, and brings the full weight of power, cooling, networking, and operations.

Many organisations that must keep data in-house still rent from providers that offer isolated or on-premise arrangements, which can square data control with not owning a cluster. Our cost calculator can frame the simpler single-machine trade-offs, though cluster economics are more involved than it models.

What can go wrong

• Buying GPUs, forgetting the fabric. The interconnect budget is part of the GPU budget, not separate from it.
• Underestimating power and cooling. The facility, not the hardware, is often the real constraint, and the slowest thing to change.
• Building before you have measured demand. At these costs, rent first, measure, and own only what sustained demand clearly justifies.

Next steps

This completes the server-class tier. For the software side, revisit running large open models, and for the smaller end of self-hosting, see planning hardware for a team. The hardware catalogue has the data-centre accelerators referenced here.