AIG Measurement Lab
Measured local LLM benchmarks: speed, memory and task outcomes
Real Ollama runs on an Apple M4 Pro (12-core CPU, 16-core GPU) with 24 GiB of unified memory. Compare the exact model, quantization, allocated context and workload; inspect the measurements behind every row. This is a shared desktop, running macOS 15.5 and Ollama 0.35.0, rather than a dedicated test host.
Explore the measurements interactively
Filter by model and workload, compare up to three measured runs side by side, and open each source record. The explorer uses this page’s public benchmark index.
15
distinct measured runs
75
measured repetitions; warmups excluded
1
machine; no cross-hardware ranking
Each run is counted once. The data snapshot was generated on 2026-10-30; test dates describe historical measurements, not a claim about the latest runtime.
What you can use these results for
Use the measurements as historical throughput and process-memory observations for these exact configurations. They can inform a capacity-planning baseline, with additional memory headroom and a test of your own workload. They do not establish minimum required RAM, a best model, long-prompt speed or performance on another machine.
Measured results
Decode speed is the median of each repetition's median request rate; the range spans repetitions. Peak RSS is the largest sampled value across measured repetitions. Client concurrency and the runtime's serving slots are separate: two clients with one slot queue requests. Success means the workload's deterministic checker passed, not that the model is generally reliable at that task.
| Model / machine | Workload | Context / clients / slots | Decode tokens/s | Peak RSS (GB) | Task executions passed | Confidence / data |
|---|---|---|---|---|---|---|
| Gemma 4 12BQ4_K_M · Apple M4 Pro · 24 GiB | automation | 8,192 tokens1 client(s) · 1 slot(s) | 25.425.1–25.6 | 8.86 | 0/150%15 · Checker failed | medium · detailsMeasurement JSON |
| Gemma 4 12BQ4_K_M · Apple M4 Pro · 24 GiB | browser_tool | 8,192 tokens1 client(s) · 1 slot(s) | 26.926.7–27.2 | 11.38 | 10/10100% | medium · detailsMeasurement JSON |
| Gemma 4 12BQ4_K_M · Apple M4 Pro · 24 GiB | coding | 8,192 tokens1 client(s) · 1 slot(s) | 24.624.4–25.0 | 10.26 | 0/150%15 · Output limit reached | medium · detailsMeasurement JSON |
| Gemma 4 12BQ4_K_M · Apple M4 Pro · 24 GiB | rag | 8,192 tokens1 client(s) · 1 slot(s) | 25.323.1–26.1 | 10.59 | 15/15100% | high · detailsMeasurement JSON |
| Qwen2.5 7B InstructQ4_K_M · Apple M4 Pro · 24 GiB | automation | 8,192 tokens1 client(s) · 1 slot(s) | 43.342.8–45.1 | 5.53 | 10/1567%5 · Checker failed | medium · detailsMeasurement JSON |
| Qwen2.5 7B InstructQ4_K_M · Apple M4 Pro · 24 GiB | browser_tool | 8,192 tokens1 client(s) · 1 slot(s) | 44.242.9–45.1 | 5.44 | 5/1050%5 · Tool-turn limit reached | medium · detailsMeasurement JSON |
| Qwen2.5 7B InstructQ4_K_M · Apple M4 Pro · 24 GiB | coding | 8,192 tokens1 client(s) · 1 slot(s) | 42.742.4–44.2 | 5.40 | 10/1567%5 · Checker failed | medium · detailsMeasurement JSON |
| Qwen2.5 7B InstructQ4_K_M · Apple M4 Pro · 24 GiB | rag | 8,192 tokens1 client(s) · 1 slot(s) | 45.240.9–46.0 | 5.37 | 15/15100% | medium · detailsMeasurement JSON |
| Qwen2.5 7B InstructQ4_K_M · Apple M4 Pro · 24 GiB | research | 8,192 tokens1 client(s) · 1 slot(s) | 45.740.9–46.0 | 5.34 | 10/10100% | medium · detailsMeasurement JSON |
| Qwen2.5 7B InstructQ4_K_M · Apple M4 Pro · 24 GiB | research | 8,192 tokens2 client(s) · 1 slot(s) | 43.041.6–45.1 | 5.34 | 20/20100% | medium · detailsMeasurement JSON |
| Qwen2.5 7B InstructQ4_K_M · Apple M4 Pro · 24 GiB | rag | 32,768 tokens1 client(s) · 1 slot(s) | 45.240.9–45.6 | 6.75 | 15/15100% | medium · detailsMeasurement JSON |
| Qwen2.5 Coder 1.5BQ4_K_M · Apple M4 Pro · 24 GiB | coding | 4,096 tokens1 client(s) · 1 slot(s) | 154.1150.5–154.8 | 1.42 | 5/1533%10 · Checker failed | medium · detailsMeasurement JSON |
| Qwen2.5 Coder 1.5BQ4_K_M · Apple M4 Pro · 24 GiB | coding | 4,096 tokens1 client(s) · 1 slot(s) | 152.5149.2–155.9 | 1.41 | 5/1533%10 · Checker failed | medium · detailsMeasurement JSON |
| Qwen2.5 Coder 3BQ4_K_M · Apple M4 Pro · 24 GiB | coding | 4,096 tokens1 client(s) · 1 slot(s) | 37.336.1–41.7 | 2.26 | 10/1567%5 · Checker failed | medium · detailsMeasurement JSON |
| Qwen2.5 Coder 7BQ4_K_M · Apple M4 Pro · 24 GiB | coding | 4,096 tokens1 client(s) · 1 slot(s) | 23.121.3–27.9 | 5.04 | 10/1567%5 · Checker failed | medium · detailsMeasurement JSON |
Gemma 4's 0/15 coding result consists of 15 output-limit truncations at 1,024 tokens, with empty recorded answer content. These are failures to finish within that budget, rather than 15 completed solutions rejected by the checker. The records do not capture reasoning text, so they do not establish why the budget was exhausted.
Qwen2.5-Coder 1.5B, 3B and 7B: measured coding tasks
October 19, 2026 measurements on an Apple M4 Pro with 24 GiB unified memory, Q4_K_M quantization, 4,096 allocated context tokens and one concurrent client. All three runs passed the runtime and manifest integrity checks. Each model attempted 15 deterministic coding tasks across five measured repetitions.
These results describe this fixture workload and configuration only. Higher tokens/s does not imply higher task success, and three differently sized models are not interchangeable. Do not use these runs as a cross-hardware ranking.
Qwen2.5 7B: allocating 8K versus 32K context
With Q4_K_M weights and one client, the RAG workload's sampled peak RSS was 5.37 GB at 8,192 allocated tokens and 6.75 GB at 32,768: an observed difference of 1.38 GB. Decode rates were 45.2 and 45.2 tokens/s.
The longest prompts were only 188 and 190 tokens. This comparison concerns allocation, not processing full 8K or 32K prompts. Both runs began with a memory-pressure warning (level 2); free memory was 30% at 8K and 35% at 32K, with 1-minute load averages of 8.12 and 4.32. These two runs do not isolate context as the sole cause of the memory difference or validate a general memory formula.
Methodology and audit data
- One warmup and five measured repetitions per run. Warmups stay in the downloads and are excluded from headline statistics. Outliers are flagged and retained.
- RSS is sampled every 0.25 seconds from the serving process. A sampling interval can miss a higher transient peak. Memory is decimal GB (1 billion bytes); hardware capacity is GiB (2³⁰ bytes).
- The secondary physical-footprint measurement excludes clean mapped weights. It is not a direct measurement of KV-cache size and is not used as the table's peak-memory figure.
- Workloads use short fictional research sources, a small retrieval corpus, a local browser fixture, three coding functions and structured automation tasks. Task counts include repeated executions of the same small task set. They do not represent unrestricted browsing, repository-scale coding, or general model quality.
- Prompts vary by session to reduce cross-repetition cache reuse. Within-session caching can remain; cached token counts are retained. Concurrency is not a claim of simultaneous GPU execution.
- Downloads contain per-request timings, token counts, checker outcomes, per-repetition metrics, memory samples, model identity, host conditions and harness/workload versions. Generated text, local paths, endpoints, process IDs and free-text runtime errors are omitted.
- These downloads are measurement extracts with their own SHA-256 checksums. The original sealed artifacts remain unchanged internally; their seal digest is recorded for provenance and does not authenticate the edited export.
- The public reproduction package contains the historical measurement harness, complete workload prompts and grading definitions, run manifests, tests and checksum-verified public extracts. All 15 published workload-definition digests match the released source.
- A fresh replication still requires a matching model artifact, Ollama version, context, sampling options and compatible hardware. The archived private sealed/raw run directories and model blobs are not redistributed; a new run is not guaranteed to reproduce identical throughput on a shared machine.
The public source package supports an independent rerun of these workloads, but does not establish third-party replication or publish the original private raw artifacts. Power, per-task cost and GPU allocations not attributed to the process remain unavailable.
Run conditions and confidence
Gemma 4 12B · automation · 8,192 tokens · 1 client(s)
2026-10-02 · Apple M4 Pro · 24 GiB unified memory · macOS 15.5 · ollama 0.35.0
5 measured repetitions · 1 warmup · medium confidence · runtime completion: completed · methodology 1.0.0 · workload 1.0.0
Temperature 0; seed 42; max output 1024 tokens. Longest prompt: 159 tokens. Median client TTFT: 25382 ms.
Host memory pressure: normal (level 1); free memory 60%; 1-minute load 4.48.
- 15 task execution(s): Checker failed.
- peak_rss_bytes has flagged outliers [3, 4]
- ttft_ms has flagged outliers [3]
Run: 20261002T213036Z-gemma4-12b-q4km-m4pro24-ollama-ctx8k-c1-automati-08bc6b5c
Download measurements and provenance →Gemma 4 12B · browser_tool · 8,192 tokens · 1 client(s)
2026-10-02 · Apple M4 Pro · 24 GiB unified memory · macOS 15.5 · ollama 0.35.0
5 measured repetitions · 1 warmup · medium confidence · runtime completion: completed · methodology 1.0.0 · workload 1.0.0
Temperature 0; seed 42; max output 1024 tokens. Longest prompt: 880 tokens. Median client TTFT: 583 ms.
Host memory pressure: normal (level 1); free memory 78%; 1-minute load 7.32.
All graded task executions passed.
- ttft_ms has flagged outliers [0]
Run: 20261002T213905Z-gemma4-12b-q4km-m4pro24-ollama-ctx8k-c1-browser_-aba2d06b
Download measurements and provenance →Gemma 4 12B · coding · 8,192 tokens · 1 client(s)
2026-10-02 · Apple M4 Pro · 24 GiB unified memory · macOS 15.5 · ollama 0.35.0
5 measured repetitions · 1 warmup · medium confidence · runtime completion: completed · methodology 1.0.0 · workload 1.0.0
Temperature 0; seed 42; max output 1024 tokens. Longest prompt: 172 tokens. Median client TTFT: Unavailable ms.
Host memory pressure: normal (level 1); free memory 82%; 1-minute load 4.17.
- 15 task execution(s): Output limit reached.
- peak_rss_bytes has flagged outliers [0]
Run: 20261002T214309Z-gemma4-12b-q4km-m4pro24-ollama-ctx8k-c1-coding-1a30b2cd
Download measurements and provenance →Gemma 4 12B · rag · 8,192 tokens · 1 client(s)
2026-10-02 · Apple M4 Pro · 24 GiB unified memory · macOS 15.5 · ollama 0.35.0
5 measured repetitions · 1 warmup · high confidence · runtime completion: completed · methodology 1.0.0 · workload 1.0.0
Temperature 0; seed 42; max output 1024 tokens. Longest prompt: 201 tokens. Median client TTFT: 11346 ms.
Host memory pressure: normal (level 1); free memory 78%; 1-minute load 9.4.
All graded task executions passed.
No confidence downgrade was recorded by the lab.
Run: 20261002T215556Z-gemma4-12b-q4km-m4pro24-ollama-ctx8k-c1-rag-2a97de43
Download measurements and provenance →Qwen2.5 7B Instruct · automation · 8,192 tokens · 1 client(s)
2026-10-02 · Apple M4 Pro · 24 GiB unified memory · macOS 15.5 · ollama 0.35.0
5 measured repetitions · 1 warmup · medium confidence · runtime completion: completed · methodology 1.0.0 · workload 1.0.0
Temperature 0; seed 42; max output 1024 tokens. Longest prompt: 151 tokens. Median client TTFT: 201 ms.
Host memory pressure: warning (level 2); free memory 27%; 1-minute load 5.86.
- 5 task execution(s): Checker failed.
- Host memory pressure: warning (level 2); free memory 27%; 1-minute load 5.86.
Run: 20261002T212142Z-qwen2.5-7b-q4km-m4pro24-ollama-ctx8k-c1-automati-abf63481
Download measurements and provenance →Qwen2.5 7B Instruct · browser_tool · 8,192 tokens · 1 client(s)
2026-10-02 · Apple M4 Pro · 24 GiB unified memory · macOS 15.5 · ollama 0.35.0
5 measured repetitions · 1 warmup · medium confidence · runtime completion: completed · methodology 1.0.0 · workload 1.0.0
Temperature 0; seed 42; max output 1024 tokens. Longest prompt: 997 tokens. Median client TTFT: 272 ms.
Host memory pressure: warning (level 2); free memory 37%; 1-minute load 6.52.
- 5 task execution(s): Tool-turn limit reached.
- decode_tokens_per_s has flagged outliers [0]
- Host memory pressure: warning (level 2); free memory 37%; 1-minute load 6.52.
Run: 20261002T212216Z-qwen2.5-7b-q4km-m4pro24-ollama-ctx8k-c1-browser_-b1c3430b
Download measurements and provenance →Qwen2.5 7B Instruct · coding · 8,192 tokens · 1 client(s)
2026-10-02 · Apple M4 Pro · 24 GiB unified memory · macOS 15.5 · ollama 0.35.0
5 measured repetitions · 1 warmup · medium confidence · runtime completion: completed · methodology 1.0.0 · workload 1.0.0
Temperature 0; seed 42; max output 1024 tokens. Longest prompt: 164 tokens. Median client TTFT: 351 ms.
Host memory pressure: warning (level 2); free memory 31%; 1-minute load 10.54.
- 5 task execution(s): Checker failed.
- decode_tokens_per_s has flagged outliers [1]
- ttft_ms has flagged outliers [2]
- Host memory pressure: warning (level 2); free memory 31%; 1-minute load 10.54.
Run: 20261002T212636Z-qwen2.5-7b-q4km-m4pro24-ollama-ctx8k-c1-coding-36f843b8
Download measurements and provenance →Qwen2.5 7B Instruct · rag · 8,192 tokens · 1 client(s)
2026-10-02 · Apple M4 Pro · 24 GiB unified memory · macOS 15.5 · ollama 0.35.0
5 measured repetitions · 1 warmup · medium confidence · runtime completion: completed · methodology 1.0.0 · workload 1.0.0
Temperature 0; seed 42; max output 1024 tokens. Longest prompt: 188 tokens. Median client TTFT: 421 ms.
Host memory pressure: warning (level 2); free memory 30%; 1-minute load 8.12.
All graded task executions passed.
- Host memory pressure: warning (level 2); free memory 30%; 1-minute load 8.12.
Run: 20261002T212724Z-qwen2.5-7b-q4km-m4pro24-ollama-ctx8k-c1-rag-970bc01f
Download measurements and provenance →Qwen2.5 7B Instruct · research · 8,192 tokens · 1 client(s)
2026-10-02 · Apple M4 Pro · 24 GiB unified memory · macOS 15.5 · ollama 0.35.0
5 measured repetitions · 1 warmup · medium confidence · runtime completion: completed · methodology 1.0.0 · workload 1.0.0
Temperature 0; seed 42; max output 1024 tokens. Longest prompt: 243 tokens. Median client TTFT: 551 ms.
Host memory pressure: warning (level 2); free memory 31%; 1-minute load 7.57.
All graded task executions passed.
- decode_tokens_per_s has flagged outliers [1, 4]
- peak_rss_bytes has flagged outliers [0, 1]
- Host memory pressure: warning (level 2); free memory 31%; 1-minute load 7.57.
Run: 20261002T212746Z-qwen2.5-7b-q4km-m4pro24-ollama-ctx8k-c1-research-cb7b577c
Download measurements and provenance →Qwen2.5 7B Instruct · research · 8,192 tokens · 2 client(s)
2026-10-02 · Apple M4 Pro · 24 GiB unified memory · macOS 15.5 · ollama 0.35.0
5 measured repetitions · 1 warmup · medium confidence · runtime completion: completed · methodology 1.0.0 · workload 1.0.0
Temperature 0; seed 42; max output 1024 tokens. Longest prompt: 242 tokens. Median client TTFT: 2748 ms.
Host memory pressure: warning (level 2); free memory 28%; 1-minute load 8.18.
All graded task executions passed.
- decode_tokens_per_s has flagged outliers [3]
- Host memory pressure: warning (level 2); free memory 28%; 1-minute load 8.18.
Run: 20261002T212814Z-qwen2.5-7b-q4km-m4pro24-ollama-ctx8k-c2-research-6fbb1fd1
Download measurements and provenance →Qwen2.5 7B Instruct · rag · 32,768 tokens · 1 client(s)
2026-10-02 · Apple M4 Pro · 24 GiB unified memory · macOS 15.5 · ollama 0.35.0
5 measured repetitions · 1 warmup · medium confidence · runtime completion: completed · methodology 1.0.0 · workload 1.0.0
Temperature 0; seed 42; max output 1024 tokens. Longest prompt: 190 tokens. Median client TTFT: 423 ms.
Host memory pressure: warning (level 2); free memory 35%; 1-minute load 4.32.
All graded task executions passed.
- decode_tokens_per_s has flagged outliers [3]
- Host memory pressure: warning (level 2); free memory 35%; 1-minute load 4.32.
Run: 20261002T212119Z-qwen2.5-7b-q4km-m4pro24-ollama-ctx32k-c1-rag-06579dcd
Download measurements and provenance →Qwen2.5 Coder 1.5B · coding · 4,096 tokens · 1 client(s)
2026-10-19 · Apple M4 Pro · 24 GiB unified memory · macOS 15.5 · ollama 0.40.3
5 measured repetitions · 1 warmup · medium confidence · runtime completion: completed · methodology 1.0.0 · workload 1.0.0
Temperature 0; seed 42; max output 1024 tokens. Longest prompt: 166 tokens. Median client TTFT: 78 ms.
Host memory pressure: warning (level 2); free memory 45%; 1-minute load 5.85.
- 10 task execution(s): Checker failed.
- decode_tokens_per_s has flagged outliers [2]
- Host memory pressure: warning (level 2); free memory 45%; 1-minute load 5.85.
Run: 20261019T205345Z-qwen2.5-coder-1.5b-q4km-m4pro24-ollama-ctx4k-c1--a65ffba1
Download measurements and provenance →Qwen2.5 Coder 1.5B · coding · 4,096 tokens · 1 client(s)
2026-10-02 · Apple M4 Pro · 24 GiB unified memory · macOS 15.5 · ollama 0.35.0
5 measured repetitions · 1 warmup · medium confidence · runtime completion: completed · methodology 1.0.0 · workload 1.0.0
Temperature 0; seed 42; max output 1024 tokens. Longest prompt: 165 tokens. Median client TTFT: 77 ms.
Host memory pressure: normal (level 1); free memory 76%; 1-minute load 6.25.
- 10 task execution(s): Checker failed.
- ttft_ms has flagged outliers [0, 4]
Run: 20261002T215940Z-qwen2.5-coder-1.5b-q4km-m4pro24-ollama-ctx4k-c1--a65ffba1
Download measurements and provenance →Qwen2.5 Coder 3B · coding · 4,096 tokens · 1 client(s)
2026-10-19 · Apple M4 Pro · 24 GiB unified memory · macOS 15.5 · ollama 0.40.3
5 measured repetitions · 1 warmup · medium confidence · runtime completion: completed · methodology 1.0.0 · workload 1.0.0
Temperature 0; seed 42; max output 1024 tokens. Longest prompt: 166 tokens. Median client TTFT: 241 ms.
Host memory pressure: normal (level 1); free memory 62%; 1-minute load 5.06.
- 5 task execution(s): Checker failed.
- decode_tokens_per_s has flagged outliers [4]
Run: 20261019T205840Z-qwen2.5-coder-3b-q4km-m4pro24-ollama-ctx4k-c1-co-16315a05
Download measurements and provenance →Qwen2.5 Coder 7B · coding · 4,096 tokens · 1 client(s)
2026-10-19 · Apple M4 Pro · 24 GiB unified memory · macOS 15.5 · ollama 0.40.3
5 measured repetitions · 1 warmup · medium confidence · runtime completion: completed · methodology 1.0.0 · workload 1.0.0
Temperature 0; seed 42; max output 1024 tokens. Longest prompt: 164 tokens. Median client TTFT: 591 ms.
Host memory pressure: normal (level 1); free memory 61%; 1-minute load 11.55.
- 5 task execution(s): Checker failed.
- decode tokens/s CV 0.11773956207866021 exceeds 0.1
Run: 20261019T205941Z-qwen2.5-coder-7b-q4km-m4pro24-ollama-ctx4k-c1-co-8d518f80
Download measurements and provenance →Publication policy
High and medium are internal repeatability labels, not statistical confidence intervals or measures of model quality. For admitted runs, high means at least five measured repetitions, no flagged outliers in decode rate, peak RSS or TTFT, decode-rate variation (standard deviation divided by mean) at most 10%, and no host-memory-pressure warning. Medium means the run remains eligible but carries a recorded repeatability or host-condition caveat. Outliers stay in the results.
Public results require an indexed, executed run with a valid raw-artifact seal, passing runtime identity checks, a supported methodology, at least five measured repetitions, no operator exclusions, and high or medium confidence. Headline statistics must agree with the sealed repetitions. Medium confidence is displayed with its recorded caveats. Task failures remain visible.
Unindexed attempts, low-confidence runs and runs below the repetition threshold are withheld. Invalid seals or conflicting statistics stop the build. A retraction removes the run from the current table and downloads, records the date and reason below, and is reviewed before a corrected release. Old downloaded copies cannot be recalled; consult the current index before citing a result.
Retractions and corrections
The following historical claims are withdrawn. They are excluded from the measured results above.
Retracted 2026-09-17: qwen2.5-7b-mac-mini-m4
- No raw evidence capture exists for this measurement.
- 620 t/s exceeds the memory-bandwidth ceiling for a ~4.7 GB Q4 artifact on a Mac mini M4 (~120 GB/s → ~25 t/s) by roughly 25x.
- 45 t/s also exceeds that ceiling.
- 'tokens/sec (first token)' is a dimension error: time-to-first-token is a latency, not a rate.
- The only environment capture in the repository (research/tests/shared/local-baseline.txt) records an Apple M4 Pro with 24 GiB, not a Mac mini M4 16 GB.
Deleted, not repaired. The correct values are unknown and reconstructing them would be fabrication.
Agent Infrastructure Recommendation Kit
Carry the evidence into an infrastructure decision
The Recommendation Kit helps you document requirements, compare local and hosted options, and record the evidence behind your decision. Benchmark data and downloads remain free to inspect.
Get the Recommendation Kit — $19Editorial methodology →Qwen hardware planning →Estimate RAM and VRAM →