How much memory do I need?
Memory is the single biggest thing that decides what you can run locally. Here is how to estimate what a model needs, and why the headline number is only part of the story.
What you’ll learn
How to estimate the memory a model needs before you download it, why real usage is always a bit higher than the headline figure, and how much headroom to leave.
The concept
For running models, memory matters more than raw speed. A model has to fit in memory to run on the GPU at all, so the amount of VRAM on your card, or unified memory in your Mac, sets a hard ceiling on what you can run.
A useful rule of thumb, for a model at a 4-bit quantisation, is a little over half a gigabyte of memory per billion parameters:
- A 7B to 8B model needs roughly 5GB
- A 14B model needs roughly 9GB
- A 32B model needs roughly 20GB
- A 70B model needs roughly 43GB
Higher quantisations need more, and full precision needs far more. These are estimates, and the catalogue lists the approximate figure for each quantisation of each model.
The part people forget
The model’s weights are not the only thing in memory. The context, the running memory of your conversation, is also held in VRAM, and it grows as the conversation gets longer. A model that fits comfortably with a short prompt can run out of memory once you feed it a long document.
So the figures above are a floor, not a budget. Leave headroom: a couple of gigabytes at least, and more if you plan to use long context. Filling your memory exactly is how you end up with crashes or a model that slows to a crawl.
Doing it
- Find your memory. Note your GPU’s VRAM, or your Mac’s unified memory.
- Subtract headroom. Take off two gigabytes or so for the system and context.
- Match a model to what is left, using the rule of thumb above, or let the hardware matrix do it for you.
Unified memory, as on Apple Silicon and some newer mini PCs, is shared between the processor and graphics, so most of it is available to models. That is why a Mac with plenty of unified memory can run larger models than a discrete card with less VRAM, even though it is often slower.
What can go wrong
- Budgeting for the weights only. Add context to your estimate, or a long chat will break it.
- Treating “fits” as “runs well”. A model crammed into memory with no headroom will be slow and fragile. Give it room.
- Ignoring quantisation. The same model at a higher quantisation may not fit at all. Check the specific figure, not just the parameter count.
Next steps
Next, it helps to understand the different kinds of hardware and their trade-offs, in GPUs and Apple Silicon, and to try the hardware matrix with your own numbers.