Qwen2.5-Coder infrastructure

Qwen2.5-Coder Hardware Requirements: RAM and VRAM by Model Size

Qwen2.5-Coder comes in six dense sizes, from 0.5B for editor autocomplete to 32B for a local coding assistant. Coding workloads are context-heavy — whole files, diffs and repository snippets — so the KV cache matters as much as the weights. This page works through AIG's three layers — parametric estimate → actual artifact fit → workload planning — for every size.

Sizing framework: three layers

1. Parametric estimate: parameter count × bits per weight ÷ 8. The fastest calculation, but it ignores quantization overhead and the runtime.
2. Actual model artifact fit: the exact GGUF, AWQ, GPTQ or safetensors file you will load. Ollama's Q4_K_M files run about 22–28% above the 4-bit floor from 1.5B up, and more for 0.5B (Layer 2).
3. Conservative workload planning: artifact + KV cache for your configured context + runtime overhead + OS reserve + one cache per concurrent request. For 14B and 32B at long context this layer adds more than a third to the artifact.

Layer 1: Parametric weight estimates

CALCULATED. Parameter count × bit width ÷ 8, using the totals reported by each repository's safetensors metadata (the 32B model card rounds its count to 32.5B). Real quantized files are larger — see Layer 2.

Layer 2: Actual model artifact sizes

Ollama's default Qwen2.5-Coder tags are Q4_K_M quantizations of the instruct models. From 3B upward they work out to just under 5 bits per weight. The two smallest sizes come out higher: their embedding tables — a 151,936-token vocabulary — are a large share of their parameters (over a quarter of 0.5B), and quantization formats typically keep those tensors at higher precision.

DOCUMENTED (sizes, quantization and context) and CALCULATED (bits per weight = size × 8 ÷ parameter count). Ollama also lists -base tags of each size at the same file sizes, plus other quantizations on its tags page.

Layer 3: Workload planning — KV cache by model and context

Every Qwen2.5-Coder size uses grouped-query attention, but with very different key/value head counts — 2 heads for 0.5B–3B, 4 for 7B and 8 for 14B and 32B. At the default 16-bit cache precision, the cache per token is 2 (K and V) × layers × KV heads × head dimension × 2 bytes. The jump from 7B to 14B is the one to notice: 14B stores more than three times as much cache per token as 7B.

CALCULATED from config.json. "Loaded" = Ollama Q4_K_M artifact + f16 KV cache at that context. Add 10–20% headroom for runtime allocations, which are not modelled separately. 128K is only available for 7B, 14B and 32B, and only with YaRN enabled (see below).

• Context actually configured: Ollama's context-length documentation picks the default from available VRAM — 4K below 24 GiB, 32K from 24 to 48 GiB, and 256K at 48 GiB and above (its FAQ still says 4,096) — and recommends at least 64,000 tokens for coding tools, which is beyond Qwen2.5-Coder's native 32K and needs the YaRN setup below. Set OLLAMA_CONTEXT_LENGTH or num_ctx explicitly, check what your editor or agent sets, and size for that value. The documentation does not say how Apple-silicon unified memory is counted, so set it on a Mac too.
• 128K needs YaRN: Qwen's model cards state that the shipped config.json supports 32,768 tokens and that longer inputs require adding a YaRN rope_scaling entry. Qwen also notes that vLLM's static YaRN can reduce quality on shorter texts, and recommends enabling it only when you need long context.
• Concurrency: Ollama documents that memory scales with OLLAMA_NUM_PARALLEL × OLLAMA_CONTEXT_LENGTH. Two parallel 32K requests on 14B add about 12.9 GB of cache on top of the 9 GB of weights.
• KV cache quantization: OLLAMA_KV_CACHE_TYPE=q8_0 roughly halves cache memory, taking 32B at 32K from about 28.6 GB to about 24.3 GB. Ollama requires Flash Attention for quantized cache and applies the setting to every loaded model.
• OS and application reserve: on Apple silicon the model shares unified memory with your editor, browser and build tools, which matter more on a development machine than on a server.

Can Qwen2.5-Coder 32B run on a 24 GB GPU?

Yes, with a shorter context or a quantized KV cache. On a card exposing 24 GiB (25.8 decimal GB), Ollama's 32K default with an f16 cache needs about 28.6 decimal GB before runtime overhead, so it cannot fit entirely on the GPU. With a q8_0 cache, the subtotal is about 24.3 decimal GB (22.6 GiB), leaving about 1.4 GiB for runtime buffers, display use and other allocations. That is tight: whether it stays fully on the GPU needs measurement, rather than following from the subtotal alone. A conservative starting point is 4K with num_ctx 4096 (or OLLAMA_CONTEXT_LENGTH=4096): about 21.1 decimal GB with an f16 cache, before runtime overhead. Check the effective context and GPU offload with ollama ps. For more headroom at whole-file or repository-scale context, plan a 32 GB or 48 GB GPU, two 24 GB GPUs, or a Mac with 48 GB or more of unified memory. Compare rental options in the GPU cloud providers guide.

Which Qwen2.5-Coder size fits 8 GB, 12 GB or 16 GB?

8 GB: 7B at up to about 16K context, or any smaller size at the full 32K. 12 GB: 7B at the full 32K, or 14B at short context with about 2 GB of headroom. 16 GB: 14B at up to 32K — about 15.4 GB loaded, so tight with an f16 cache and comfortable with q8_0. 24 GB: 14B with room for concurrency — it fits at Ollama's 32K default for this size — or 32B at a shorter context, with 4K as a conservative starting point.

Which size for autocomplete vs a coding assistant?

Autocomplete and chat are different workloads. Qwen recommends the base models — not the instruct models — for fill-in-the-middle completion, and editor completion has to answer within a keystroke or two, so small base models (0.5B–3B) are the usual choice. They need about 0.5–3 GB, so they can run alongside a larger chat model. Chat, code review and agentic edits benefit from the 14B or 32B instruct models, and are where the context and concurrency costs above apply. For hosted alternatives, see the best AI coding agents guide.

Can Qwen2.5-Coder run without a GPU?

Yes, with Ollama or llama.cpp on the CPU, using the same memory figures from system RAM. The small sizes are practical on a CPU; 14B and 32B are usable for occasional chat but slow for interactive completion, because every generated token reads the full 9–20 GB of weights from memory. See the local AI RAM requirements guide for CPU-only sizing.