Qwen infrastructure

Qwen Hardware Requirements: RAM and VRAM by Model and Quantization

Qwen hardware requirements depend on the exact model, precision or quantization, context length, inference runtime, batch size, and whether the workload runs entirely on GPU, unified memory, or with CPU offload. This page uses a three-layer sizing framework: parametric estimate → actual model artifact fit → conservative workload planning.

Sizing framework: three layers

Do not plan hardware from a single number. Each layer adds information the previous one does not have:

1. Parametric estimate: Parameter count × bits per weight ÷ 8 = approximate weight floor in bytes. This is the fastest calculation but ignores model file metadata, quantization overhead, and runtime requirements.
2. Actual model artifact fit: The real file size of the quantized or full-precision model on Hugging Face or Ollama. This is more accurate than the parametric estimate because it accounts for quantization format overhead, embedding tables, and metadata. Check the specific GGUF, AWQ, GPTQ, or safetensors file you plan to deploy.
3. Conservative workload planning: Model artifact size + context/KV cache + runtime/framework overhead + OS/application reserve + concurrency headroom = total memory planning target. This is the number that determines whether the hardware actually works.

Layer 1: Parametric weight estimates

CALCULATED. These are parameter count × bit width ÷ 8. Real quantized files include additional data and overhead — see Layer 2.

Layer 2: Actual model artifact sizes

The parametric estimate is a starting point. The actual model file you deploy is typically larger than the floor because quantization formats include scales, offsets, embedding tables, and metadata. Check the specific file on Hugging Face or the Ollama library for the exact size.

For example, the Qwen3.5-27B BF16 safetensors repository is approximately 55.6 GB — slightly larger than the 54 GB parametric estimate. GGUF quantizations of the same model will vary by quantization level: Q4_K_M files are substantially smaller than Q8 files.

Use the VRAM calculator for parametric estimates, and Can I Run This Model? to check whether a specific GPU can accommodate the workload.

Layer 3: Workload planning — what adds to the weight floor

A model that appears to fit exactly into available memory can still fail once runtime overhead is included. Plan for all of these components:

• Quantization format overhead: The actual model artifact is typically larger than parameter count × bits ÷ 8 because quantization formats store additional data (scales, zero points, metadata).
• Context / KV cache: Memory grows with context length. For Qwen3.5/3.8 hybrid architectures, the KV cache behavior depends on the mix of attention types — a generic formula is not reliable. Measure with the actual runtime.
• Runtime and framework overhead: The inference engine (Ollama, vLLM, SGLang, llama.cpp, MLX) consumes memory beyond the model and KV cache. This varies by engine.
• OS and application reserve: On a GPU server, reserve system RAM for the OS and other services. On Apple silicon unified memory, the same pool serves the OS, apps, and model.
• Concurrency: Each concurrent inference slot allocates its own KV cache. Ollama documents that memory scales with OLLAMA_NUM_PARALLEL × context length.
• Multimodal components: Qwen3.8 includes a vision encoder. Image inputs require additional memory beyond text-only inference.
• Planning headroom: Budget at least 10–20% headroom beyond the calculated total. Running at maximum capacity leaves no room for spikes, framework updates, or configuration changes.

Which Qwen models can run in 16 GB of VRAM?

Qwen3 models up to 8B parameters at most quantizations fit within 16 GB. Larger models at Q4 quantization may fit by weight alone, but leaving room for context cache and runtime overhead is critical. Plan around the full runtime requirement — not the model file size alone.

How much RAM does Qwen need?

System RAM and GPU VRAM answer different questions. A GPU-first deployment needs enough VRAM for the model artifact plus caches and runtime allocations, while system RAM still supports the operating system, inference engine, model loading, CPU offload, and other services. CPU-only and Apple-silicon deployments rely much more heavily on system or unified memory. Do not use the model's download size as the machine's total RAM requirement.

Qwen minimum requirements vs recommended capacity

A minimum requirement only answers whether a configuration may load. A useful deployment also needs room for the context window, runtime, concurrent requests, and the rest of the machine. For purchasing or production planning, size from the actual artifact and add explicit workload reserves instead of targeting a configuration that only barely fits.

Can Qwen run locally without a GPU?

Yes. CPU-only inference is possible with Ollama, llama.cpp, and other runtimes, particularly for smaller and quantized models. Apple silicon Macs can use unified memory through MLX or Ollama. CPU-only inference is slower than GPU-accelerated inference — how much depends on the model size and hardware. For practical CPU/unified-memory sizing, see the local AI RAM requirements guide.

MoE models: Qwen3-235B-A22B

The Qwen3-235B-A22B model uses a sparse mixture-of-experts architecture: 235B total parameters but only ~22B active parameters per token. The full model must still be loaded into memory (all expert weights), even though only a subset activates during inference. This means the VRAM requirement is closer to the total parameter count than the active count. Multi-GPU deployment is typically required.