Local LLM Memory Explained: VRAM, RAM, Quantization, and Context

“Will this model fit?” sounds like one question. It is really several questions:

Will the weights fit?
Will the selected context fit?
Will the runtime have enough working memory?
Will the operating system still have room?
Will it fit with the intended number of concurrent sequences?

A download-size comparison answers only the first part.

Weight Memory Sets the Floor

Every model parameter needs storage. FP16 weights use about two bytes per parameter. Quantization represents most weights with fewer bits and adds scale information, which lowers the total.

A Q4_K_M planning ratio is around 0.58 bytes per parameter in the FindUtils calculator. The actual GGUF file can differ because formats use mixed tensor types and include metadata.

The LLM Requirements Calculator starts with this weight floor, then adds cache and runtime headroom.

KV Cache Makes Context a Memory Choice

During generation, the runtime keeps attention keys and values for the current context. More retained tokens mean a larger cache.

The cache also grows with concurrent sequences. A context that fits for one user can fail when several requests run together.

Model architecture changes the exact rate. Grouped-query and multi-query attention can reduce cache dimensions. Some runtimes support lower-precision cache formats. This is why two estimators can show different results for the same model.

The reliable conclusion is simple: longer context costs more memory. The exact amount belongs to the model and runtime combination.

Quantization Does Not Shrink Every Memory Use Equally

Quantizing the weights makes the model file and resident weights smaller. It does not automatically shrink all temporary buffers or the KV cache.

This creates a practical pattern. Moving from FP16 to Q4 can produce a large reduction. Moving from Q4 to Q3 can save less total memory than expected once cache and overhead become a larger share.

Choose quantization for the full workload, not only the model file.

MoE: Compute Is Not Storage

Mixture-of-Experts models activate only some experts for each token. Marketing and model cards often publish both total and active parameter counts.

Active parameters describe the compute path. They do not mean the inactive experts vanish from memory.

A normal resident load still needs all expert weights in VRAM or system RAM. Expert offloading can move weights elsewhere, but then memory bandwidth and data movement affect speed.

For a safe fit estimate, use total parameters for model weight memory. Use active parameters to understand per-token compute.

Headroom Prevents Fragile Fits

A calculated requirement of 23.8 GB does not make a 24 GB GPU comfortable.

The runtime needs buffers. The display system may use VRAM. Context can grow. A different backend may allocate larger temporary tensors. Parallel requests multiply state.

Leave meaningful headroom. The right amount depends on the runtime and workload, but zero is never a robust plan.

Unified Memory Changes the Boundary

On a unified-memory system, the GPU and CPU share one pool. This lets local inference use more than a discrete GPU's fixed VRAM in some configurations.

The operating system and other applications use that pool too. Advertised memory is not fully available to the model. Bandwidth, thermal limits, and runtime support also affect performance.

Unified memory makes larger models possible. It does not make memory free.

Offload Trades Memory for Speed

If a model does not fit in VRAM, a runtime can keep some layers in system RAM. This turns a hard GPU limit into a performance tradeoff.

The model can run, but token speed may fall because data crosses a slower link. Storage-based expert offload can expand capacity again, with another speed cost.

The useful question becomes: “Can this model run at an acceptable speed?” A memory calculator answers the capacity part, not the final throughput.

A Practical Selection Process

Choose the task and a realistic context.
Select several model sizes that meet the task.
Compare available quantizations.
Add cache and runtime headroom.
Decide whether offload is acceptable.
Confirm the exact published file size.
Test the intended runtime with the real workload.

This process prevents a common waste: downloading a large checkpoint before basic capacity planning.

Try It

Open the LLM Requirements Calculator. Compare one model at Q8, Q5, and Q4. Then change only the context length and watch the cache estimate move.

Read the Local LLM Hardware Guide for a complete planning checklist.