Planning hardware for a team

What you’ll learn

How to size hardware for serving a model to several people at once, the choice between server cards and multiple consumer cards, and how to weigh buying against renting, honestly.

What concurrency does to memory

For a single user, you size memory for the model plus a bit of context. Serving is different: each concurrent request holds its own context in VRAM at the same time. Ten people in long conversations need far more memory for context than one person does, on top of the model itself. Sizing for the model alone and forgetting the concurrent context is the most common planning mistake.

So plan around two numbers: the memory for the model at your chosen quantisation, and the memory for the combined context of the users you expect at peak.

The hardware routes

• Multiple consumer cards. Several RTX 5090s or used 3090s can pool memory to serve a mid-sized model. This is cost-effective, but multi-GPU setups add complexity in power, cooling, and configuration.
• Workstation cards. A single RTX PRO 6000 with 96GB runs large models on one card, simplifying the setup at a professional price.
• Data-centre accelerators. H100 and A100 cards offer the most memory and throughput, and are the right tool at real scale, but they are infrastructure with the power, cooling, and cost that implies.

Buy or rent

This is the decision that most affects the budget, and it deserves an honest answer.

Owning hardware makes sense for steady, predictable load, sensitive data that must stay in-house, or a workload heavy enough that renting adds up. Renting cloud GPUs by the hour makes sense for spiky or uncertain demand, for trying a configuration before committing, or when you cannot justify the capital and operational cost of your own machines. Many teams do both: owned hardware for the baseline, rented capacity for peaks. Our cost calculator can frame the trade-off, though serving costs are more involved than a single-user setup.

What can go wrong

• Sizing for the model, not the load. Concurrent contexts can dwarf the model’s own memory. Plan for peak users.
• Underestimating the operational cost. Power, cooling, failures, and maintenance are real and ongoing. A served cluster is a system to run, not a one-off purchase.
• Buying before you know your load. If demand is uncertain, rent first, measure, then decide what to own.

Next steps

Revisit serving models at scale for the software side, and use the hardware catalogue and the cost calculator to put numbers to your plan.