GPUs and Apple Silicon
The main routes into local AI hardware, and the one trade-off that runs through all of them: memory capacity versus memory bandwidth, and what each buys you.
What you’ll learn
The three common hardware routes for local AI, how they differ, and how to think about the trade-off between how much a machine can hold and how fast it runs.
The concept
Two numbers dominate local AI hardware. Capacity is how much memory you have, which decides how large a model you can run. Bandwidth is how fast that memory can be read, which largely decides how quickly a model generates text. Almost every choice is a trade between the two.
The three routes
NVIDIA GPUs. The path of least resistance. CUDA support is mature, every tool targets it first, and cards have high bandwidth, so generation is fast. The catch is capacity: consumer cards top out at 24GB to 32GB, so the largest models need quantisation or a second card. A used RTX 3090 is often the best value in local AI, giving 24GB cheaply, while the RTX 5090 is the current fast flagship.
Apple Silicon. Macs share one pool of unified memory between processor and graphics, so a Mac with plenty of memory can hold very large models, far more than a consumer GPU. The trade-off is bandwidth: even the fastest Apple chips have less than a high-end NVIDIA card, so generation is slower. It is quiet and low-power, which many people value. A Mac mini M4 Pro is an affordable entry, and the Mac Studio machines scale up the memory.
AMD and unified-memory mini PCs. AMD’s Radeon cards offer 24GB for less than NVIDIA, at the cost of a less mature software stack. A newer category, unified-memory mini PCs like the Ryzen AI Max+ 395, puts up to 96GB of usable memory in a small, low-power box, trading bandwidth for capacity much as a Mac does.
Choosing
- Fast generation on mid-sized models, widest tool support: an NVIDIA GPU.
- The largest models you can afford, quietly and at low power: Apple Silicon or a unified-memory mini PC, accepting slower generation.
- Best value for 24GB: a used 3090, if you are comfortable buying second-hand.
What can go wrong
- Buying for capacity when you need speed, or the reverse. Decide which matters for your use before you spend. A huge, slow machine is frustrating if you wanted snappy chat.
- Underestimating AMD’s software friction. It has improved, but expect more troubleshooting than NVIDIA.
- Forgetting the running costs. High-end GPUs draw a lot of power and generate heat. Our cost calculator can help you weigh that.
Next steps
Try the hardware matrix to see what specific machines can run, and the cost calculator to weigh buying hardware against paying for an API. The full hardware catalogue has the numbers for each option.