Self-hosting at organisational scale, from a small team on a couple of GPUs to enterprise clusters running open models beyond 100 billion parameters.
"At scale" covers a huge range, from a handful of colleagues sharing one GPU to an enterprise cluster running a trillion-parameter model. Working out which you are is the first and most important decision.
Running a model for one person and serving it to a team are different problems. What changes when you scale up, and the engines built for the job.
Once a model is served, a team needs a front door: access control, routing across models, cost tracking, and a way to see what is happening. The pieces that provide it.
Serving a model to a team needs more than a single card. How to size hardware for concurrent users, when to buy versus rent, and the running costs people forget.
Once a model no longer fits on one GPU, serving it becomes a distributed-systems problem. The ladder from a single 80GB card to a multi-node cluster, and the parallelism and quantisation that make it work.
At server-class scale the GPUs are only part of the story. The interconnect between them, the networking between machines, and the power and cooling around them decide whether a cluster runs well. An honest map of what enterprise self-hosting involves.