How much memory a local LLM actually needs
Local LLMs are a memory problem, not a compute one. How model size and quantization set the number, discrete VRAM vs unified memory, and why a Ryzen AI Max+ 395 can run a 70B with no graphics card.
Almost every "what hardware do I need to run a local LLM" question has the same answer: enough fast memory to hold the model. Compute decides how fast the tokens come out; memory decides whether the model runs at all. Get the memory number right and the rest is detail. Here is how to get it right.
The number is model size times quantization
A model's memory footprint is roughly its parameter count times the bytes per parameter. Quantization is what sets the bytes: it compresses the weights, trading a little quality for a lot of space. Q4 (about 4 bits per weight, ~0.5 bytes) is the sweet spot most people run - close to full quality, roughly a quarter of the size. So a rule of thumb at Q4, with a bit of headroom for context:
- 7B model - about 6-8GB
- 13B - about 10-12GB
- 33B - about 20-24GB
- 70B - about 40-48GB
- 120B+ - 80GB and up
Run a heavier quant (Q6/Q8) for better quality and the number climbs; run Q3 to squeeze a bigger model into less memory and quality drops. But Q4 is the honest planning figure, and it is what the "Local LLM" tiers on this site assume.
Two ways to get the memory on a small machine
Historically the memory that mattered was discrete-GPU VRAM - a graphics card with its own fast memory. That is still the fastest option per gigabyte, and it is how the accelerators here are ranked (sort by VRAM). A 16GB card runs a 13B comfortably; a 24GB card reaches a 33B; a 48GB card gets you to a 70B. The catch on a small machine is that most mini-PCs have no PCIe slot for a card at all.
The new option is unified memory. Modern AMD APUs let you hand a large slice of system RAM to the integrated GPU as graphics memory. On a Ryzen AI Max+ 395 with 128GB, that can be roughly 96GB allocated to the iGPU - enough to fit a 70B locally with no graphics card at all, in a box that idles quietly on a desk. The Ryzen AI 9 HX 370 does the same on a smaller scale. This is why boxes that used to be irrelevant for LLMs are suddenly interesting, and why the best mini PC for local AI list now ranks unified-memory boxes alongside GPU boxes.
Unified memory is about capacity, not speed
One honest caveat. A discrete GPU and a unified-memory APU that both "fit" a 70B are not the same experience. The GPU has far more memory bandwidth and compute, so it generates tokens several times faster. Unified memory is how you fit the biggest models cheaply and quietly; a discrete card is how you run a given model fast. For a chat assistant or overnight batch work, a 395-class box running a 70B at a handful of tokens per second is genuinely useful. For low-latency, high-throughput serving, a real GPU still wins. Pick the path that matches how you will actually use it.
How to read the tiers on this site
Every accelerator and every big-APU mini-PC here carries a "Local LLM" line - "up to ~13B (Q4)", "up to ~70B (Q4)", and so on. That figure is the largest Q4 model its memory can hold, computed from discrete VRAM for cards and from allocatable unified memory for APU boxes. Treat it as the fit ceiling: it tells you the model will load and run, not that it will run at GPU speed. Buy for the model size you actually want to run, leave a little headroom for context, and you will not be surprised.
Start with the accelerator comparison if you want a card, or the best mini PC for local AI if you want a single quiet box that does it on unified memory.