Best GPU for Local LLMs: VRAM for Llama, Qwen, DeepSeek
By Rohit, Founder ·
Quick answer: The VRAM a local LLM needs is the size of its quantized weights plus the KV cache for your context length, plus some headroom. Using download sizes published by Ollama, a 7B to 8B model at Q4_K_M is 4.4 to 5.2 GB, a 14B is 9.0 GB, a 32B is 20 GB and a 70B is 43 GB. That points to roughly an 8 GB card for 7B to 8B, 16 GB for 14B, 24 GB for 32B and 48 GB for 70B. Longer context, parallel users and higher-precision quantization all add to the total, so size from your own settings, not only the model name.
This guide is for developers, IT managers, researchers and business owners in India who want to run Llama, Qwen, DeepSeek or Mistral models on their own hardware and need to choose a GPU. Nothing here is a ProStation benchmark: ProStation Systems publishes no tokens-per-second figures, so this post gives memory arithmetic from vendor documentation and says where a figure is derived. Last updated: 19 October 2026.
How Do You Estimate the VRAM a Local LLM Needs?
VRAM is the memory on the graphics card. A model must sit in it, or the part that does not fit runs from system memory and slows down. Three things use it:
- Weights. These are the model itself. Their size depends on the parameter count and the quantization. Quantization stores each weight in fewer bits: Q4_K_M is a 4-bit-class format, Q8_0 is 8-bit-class and FP16 is the 16-bit original. Fewer bits means a smaller file and some loss of precision.
- KV cache. While the model reads your prompt and writes its answer, it stores attention keys and values for every token so far. This grows with context length and with every parallel user.
- Runtime overhead. The inference software needs working buffers. This varies by software and version, so leave headroom and check the real figure with
ollama psafter loading.
The weights part is the easy one, because the model's published download size is a good guide. The KV cache can be worked out from the model's configuration: bytes per token = 2 (keys and values) x layers x KV heads x head size x bytes per value. At the default f16 setting each value takes 2 bytes.
How Much VRAM Do Popular Models Need?
This table lists download sizes shown on the Ollama library tag pages for each model family. These are weights only. Add the KV cache from the next section and some headroom before choosing a card.
| Model (Ollama tag family) | Q4_K_M | Q8_0 | FP16 | Smallest single-GPU class for Q4_K_M weights + modest context |
|---|---|---|---|---|
| Mistral 7B Instruct | 4.4 GB | 7.7 GB | 14 GB | 8 GB card or larger |
| Qwen2.5 7B Instruct | 4.7 GB | 8.1 GB | 15 GB | 8 GB card or larger |
| DeepSeek-R1 distill 8B (Llama or Qwen3 base) | 4.9 to 5.2 GB | 8.5 to 8.9 GB | 16 GB | 8 GB card or larger |
| Qwen2.5 14B / DeepSeek-R1 distill 14B | 9.0 GB | 16 GB | 30 GB | 16 GB card (Q4_K_M); Q8_0 needs more than 16 GB once context is added |
| Qwen2.5 32B / DeepSeek-R1 distill 32B | 20 GB | 35 GB | 66 GB | 24 GB card (Q4_K_M, short context); 48 GB for Q8_0 |
| Llama 3.3 70B / DeepSeek-R1 distill 70B | 43 GB | 75 GB | 141 GB | 48 GB card (tight); two 48 GB cards for Q8_0 |
| Qwen2.5 72B Instruct | 47 GB | 77 GB | 145 GB | Above 48 GB once context is added: plan for two GPUs |
| DeepSeek-R1 (full 671B, mixture of experts) | 404 GB | 713 GB | 1.3 TB | Multi-GPU server territory; not a single-card model |
Two points on reading it. First, the models Ollama lists as deepseek-r1:7b, 8b, 14b, 32b and 70b are not the full DeepSeek-R1. DeepSeek's model card says the distilled models are fine-tuned from open-source models, using samples generated by DeepSeek-R1, with Qwen and Llama bases. The real DeepSeek-R1 is a mixture-of-experts model with 671 billion total parameters, 37 billion activated per token, which is why its Q4_K_M download is 404 GB. When someone says they run DeepSeek on a laptop, they mean a distilled model. Second, the download size is not the whole VRAM bill: it excludes context.
How Much Does Context Length Add?
Ollama's FAQ states that by default it uses a context window of 4,096 tokens, changed with OLLAMA_CONTEXT_LENGTH or num_ctx. Many models advertise far more: the Llama 3.3 tags on Ollama list 128K and the DeepSeek-R1 tags list 128K to 160K. Using each model's published configuration (Llama 3.3 70B: 80 layers, 8 KV heads, head size 128; Qwen2.5 32B: 64 layers, 8 KV heads, hidden size 5,120 over 40 attention heads, which gives head size 128), the arithmetic below applies the formula above at f16. These are derived figures, not measurements.
| Context length | Llama 3.3 70B (80 layers, 8 KV heads, head size 128) | Qwen2.5 32B (64 layers, 8 KV heads, head size 128) |
|---|---|---|
| 4,096 tokens (Ollama default) | 1.25 GiB | 1 GiB |
| 8,192 tokens | 2.5 GiB | 2 GiB |
| 32,768 tokens | 10 GiB | 8 GiB (the model's stated maximum) |
| 131,072 tokens | 40 GiB | Beyond the model's 32,768 configured maximum |
The lesson is that a long context can cost as much memory as a small model. Running Llama 3.3 70B at its full 131,072-token window would need about 40 GiB for the cache alone, on top of 43 GB of weights. For most chat and document tasks a context of 8,192 to 32,768 tokens is enough, so set it deliberately rather than paying for a window you do not use.
Ollama offers a way to shrink this. Its FAQ says OLLAMA_KV_CACHE_TYPE can be set to q8_0, which uses about half the memory of the default f16 cache with very little quality loss, or q4_0, about a quarter with more quality loss at larger contexts. This needs Flash Attention, which Ollama enables automatically when the backend and devices support it. The setting is global, so all models run with it.
What Happens When Several People Use the Model?
Every simultaneous request needs its own context. Ollama's documentation says that parallel request processing for a model increases the context size by the number of parallel requests, and OLLAMA_NUM_PARALLEL defaults to 1. As illustrative arithmetic from the table above: Qwen2.5 32B at 8,192 tokens needs about 2 GiB of cache per request, so four parallel users need about 8 GiB for cache plus 20 GB of weights, which is more than a 24 GB card comfortably holds. If your use is a shared team assistant rather than one developer, size VRAM for the parallel load, or lower the context. We cover the wider private-AI decision in our guide to on-prem AI servers versus cloud GPUs.
Single GPU or Multiple GPUs?
Ollama's FAQ states that if a model fits entirely on any single GPU it loads there; if not, it is spread across all available GPUs, and it notes that single-GPU placement typically performs better because it reduces data transfer over the PCI bus. So the practical rule is to buy the smallest single card that holds your model with its context, and move to several cards only when the model cannot fit on one. Two 24 GB cards offer 48 GB in total but behave differently from one 48 GB card, and the split needs spare PCIe lanes, power and cooling in the chassis. Multi-GPU builds also change what the power supply, UPS and airflow must handle; see our notes on UPS sizing, redundant power supplies and why heavy GPU loads throttle without good cooling.
Does System RAM Help?
It helps as overflow, not as a substitute. Ollama's documentation says ollama ps reports where a model sits: 100% GPU means fully on the GPU, 100% CPU means fully in system memory, and a split such as 48%/52% CPU/GPU means partial offload. Anything running from system RAM is limited by the CPU and memory path, not GPU memory, so expect the offloaded part to be slower. ProStation's own AI/ML guidance is to keep system RAM at about twice total VRAM as a floor, and every ProStation tier uses ECC DDR4; our ECC versus non-ECC guide explains why that matters for long-running jobs. We do not publish a speed figure for offloaded models because we have not measured one.
Which NVIDIA Cards Fit Which Models?
Using NVIDIA's own specification pages, here are four cards of different memory sizes. This is a memory-fit guide, not a performance ranking, and not a statement of which cards are in stock.
| NVIDIA card | Memory (per NVIDIA) | Max power (per NVIDIA) | Form factor | Largest model class it holds at Q4_K_M |
|---|---|---|---|---|
| RTX A4000 | 16 GB GDDR6 with ECC | 140 W | Single slot | Up to 14B |
| RTX A5000 | 24 GB GDDR6 with ECC | 230 W | Dual slot, active cooling | Up to 32B with short context |
| GeForce RTX 4090 | 24 GB GDDR6X (ECC not mentioned) | 450 W total graphics power; 850 W minimum system PSU | Consumer card | Up to 32B with short context |
| RTX A6000 | 48 GB GDDR6 with ECC | 300 W | Dual slot; 2-way NVLink bridge | 70B at Q4_K_M, with little headroom |
For inference, memory size is usually the first filter, because a model that does not fit cannot run at full speed at all. Compare the "largest model class" column with your Q4_K_M row above, then check headroom for context. A 24 GB card holds a 20 GB 32B model with about 4 GB left, which covers an 8,192-token context (about 2 GiB) but not 32,768 tokens (about 8 GiB). Power also differs: the 140 W single-slot A4000 suits a compact build, while the RTX 4090 page lists 450 W and an 850 W minimum system power supply. The A-series pages list ECC on the card memory; the GeForce page does not mention it. If you also train or fine-tune, see our fine-tuning VRAM guide, since training needs more memory than inference for the same model.
Which ProStation Build Fits a Local LLM?
ProStation Systems builds brand-new custom workstations and tower servers for many workloads, with local AI being one of them. These are the tiers as published on the servers page:
| ProStation tier | CPU | ECC DDR4 RAM | GPU (as published on the site) | Fits this local-LLM use |
|---|---|---|---|---|
| Starter | Intel Xeon E | 16 to 64 GB | Optional | Small models on CPU for testing; a GPU is better chosen on a higher tier |
| Pro | Intel Xeon Scalable | 64 to 256 GB | Optional NVIDIA RTX | One GPU: 7B to 32B models for a small team |
| Ultra | Dual Intel Xeon Scalable | 256 to 512 GB | NVIDIA RTX / A-series / Tesla | Larger VRAM, multi-GPU builds, 70B-class models |
All tiers use ECC DDR4 memory with Intel Xeon processors, ship in about 4 working days after configuration, and are burn-in tested for 48 hours or more (see how every build is stress-tested). Warranty is one year on hardware, with extended cover available as a paid option; the warranty page has the terms. The exact GPU model for a given quote is confirmed at order time, so tell us the model, quantization, context length and number of users, and we match the card to that. You can describe it through free consulting or build a configuration in the configurator. For the wider picture of AI hardware, read the AI and machine learning server buying guide, best GPUs for AI/ML workstations in India, how to spec a GPU server for LLM training and the AI and ML use-case page. Teams planning a first AI setup in phases may prefer the startup AI lab roadmap, and the general VRAM guide for AI and rendering covers rendering workloads. For image generation instead of text, see the Stable Diffusion sizing guide.
If you only need a GPU temporarily or on a tight budget, ProStation's sister brand Serverwale offers tested refurbished GPU servers. Compare both routes on the new versus refurbished page and ask for warranty terms in writing. For research groups, our AI and ML labs page lists typical needs, and the CPU versus GPU balance guide helps decide how much to spend on each. Budget questions are covered in how much a custom server costs in India.
Checklist Before You Choose a GPU
- Write down the exact models and the quantization you will use (for example Qwen2.5 32B at Q4_K_M).
- Take the weights size from the model's tag page and add the KV cache for your context length.
- Multiply the cache by the number of parallel users you plan to serve.
- Add headroom, load the model, and read the real figure from
ollama psor your runtime's own report. - Decide whether quantizing the KV cache or lowering context is acceptable before buying more VRAM.
- Check power supply, UPS capacity and case airflow for the cards you pick.
- Keep system RAM at about twice total VRAM as a floor, ECC if the job runs for days.
- Ask your supplier to confirm the exact GPU model, warranty and delivery time in writing.
Frequently Asked Questions
Q1. How much VRAM do I need to run Llama 3.3 70B locally?
The Q4_K_M download is 43 GB on Ollama, so the weights alone need about 43 GB of GPU memory before any context is added. At an 8,192-token context the KV cache adds roughly 2.5 GiB at the default f16 setting, so a single 48 GB card is possible but tight, and a 24 GB card cannot hold it without spilling into system RAM. Q8_0 is 75 GB and FP16 is 141 GB, which need multiple GPUs.
Q2. Can I run DeepSeek-R1 on one GPU?
Only the distilled versions. The models Ollama lists as deepseek-r1:7b, 8b, 14b, 32b and 70b are smaller Qwen- and Llama-based models fine-tuned on DeepSeek-R1 outputs. The 7B at Q4_K_M is 4.7 GB and the 32B is 20 GB, so they fit common single GPUs. The full 671-billion-parameter DeepSeek-R1 is 404 GB at Q4_K_M and is not a single-GPU model.
Q3. Is system RAM useful for running LLMs?
Yes, as overflow. Ollama checks whether a model fits on one GPU, and if not it spreads it across GPUs or partly onto the CPU, using system memory for the part that runs on the CPU. The ollama ps command shows the split. Anything that runs from system RAM is slower than GPU memory, so treat RAM as a safety margin, not a substitute for VRAM. ProStation's AI/ML guidance is system RAM of at least about twice total VRAM.
Q4. Should I choose a GeForce RTX card or an RTX A-series card?
Both run the same inference software. Based on NVIDIA's own pages, the RTX A4000, A5000 and A6000 carry 16, 24 and 48 GB of GDDR6 with ECC, and the A4000 is a single-slot 140 W card. The GeForce RTX 4090 has 24 GB of GDDR6X at 450 W total graphics power and its page does not mention ECC. VRAM size usually matters more for inference than the product line, and ProStation confirms the exact card in your quote.
Q5. Can several people use one local LLM at the same time?
Yes, but each parallel request needs its own context. Ollama's documentation says parallel processing increases the context size by the number of parallel requests, and OLLAMA_NUM_PARALLEL defaults to 1. For Qwen2.5 32B at 8,192 tokens the KV cache is about 2 GiB per request at f16, so four parallel users need about 8 GiB for the cache alone, on top of the 20 GB of weights.
Q6. How can I reduce the VRAM a model needs?
Pick a smaller quantization, lower the context length, or quantize the KV cache. Ollama's FAQ says OLLAMA_KV_CACHE_TYPE can be f16 (default), q8_0 (about half the memory of f16 with very little quality loss) or q4_0 (about a quarter, with more quality loss at larger contexts), and it needs Flash Attention enabled. Each step trades some quality or context for memory.
Final Recommendation
Choose the GPU from the model you will actually run: about 8 GB for 7B to 8B models, 16 GB for 14B, 24 GB for a 32B model at short context, and 48 GB or more for a 70B model at Q4_K_M, then add memory for context and parallel users. Buy the smallest single card that fits, and move to multiple cards only when one cannot hold the model. For a quote built around your model and team size, contact ProStation Systems or call +91 87962 44410 (WhatsApp: wa.me/918796244410) for free consulting.
Sources: Ollama library tag pages for Llama 3.3, DeepSeek-R1, Qwen2.5 and Mistral (download sizes and context windows, retrieved 19 October 2026); Ollama FAQ (default context, KV cache types, parallel requests, GPU and CPU placement); Hugging Face model configurations for Llama 3.3 70B Instruct (a mirror of Meta's, whose own repository is gated) and Qwen2.5 32B Instruct; model cards for DeepSeek-R1 and DeepSeek-R1-Distill-Qwen-32B; NVIDIA product pages for the RTX A4000, RTX A5000, RTX A6000 and GeForce RTX 4090. KV cache figures are our arithmetic from those configurations. Model sizes and software behaviour change between releases, so re-check the current pages before purchase.