I spent today testing six open-weights models on my 2021 MacBook Pro. M1 Max, 32 GB of unified memory. I wanted to find a model and inference engine I could use for a fully local agent.
This is preparation for my talk, “Wonderful World of Open Weights AI”, at GDG Melbourne DevFest on 3 October 2026. I compared llama.cpp and MLX through mlx-lm, using builds labelled 4-bit for the same six models.
One setting changed how I read the results. Normally, a language model generates one token at a time. A token is a small piece of text, often part of a word. Multi-token prediction (MTP) uses an extra prediction head to propose several upcoming tokens. The main model checks the proposals together, accepts a valid continuation and discards the rest. This is a form of speculative decoding. Accept enough proposals, and generation gets faster.
With MTP enabled in llama-server, Qwen3.6-35B-A3B averaged 81.0 tokens per second. That is the model I am keeping for my local agent experiments. Getting there involved a power setting, a separate server setting, and a limitation I only found by trying it.
Check Low Power Mode first
My early numbers were well below what I expected from this chip. The Mac was in Low Power Mode.
Switching from Low Power to High Power roughly doubled prompt-processing speed on every model I checked. Qwen3.8-27B went from 68 to 136 tokens per second. Gemma 4 31B went from 64 to 130.
Generation speed changed unevenly: Gemma 4 31B went from 7.9 to 17.0 tokens per second, while Qwen3.6-35B-A3B went from 49.0 to 57.5. These were separate power-mode checks, not the MTP comparison below.
pmset -g | grep powermodeOn this machine, 1 meant Low Power, 0 Automatic and 2 High Power. Apple describes the modes here. I measured Low versus High, so I cannot claim a separate gain from High over Automatic. Check the setting before blaming the model.
MTP is a llama-server setting
Downloading a model with an MTP head does not enable speculative decoding by itself. In my tested llama.cpp build, I had to load the matching draft-head file and select the mode in llama-server:
llama-server -m model.gguf -ngl 99 \
--spec-type draft-mtp -md matching-mtp-head.ggufThe command shows the relevant setting; the filenames stand for the main model and its matching head. For the on/off test, I kept the other server settings unchanged and removed the two speculation options for the off run. llama-bench, which I used for the original comparison, does not exercise this path.
Which models supported MTP?
I found and tested draft heads for Gemma 4 E4B, Gemma 4 12B, Qwen3.8-27B, Gemma 4 31B and Qwen3.6-35B-A3B. They added about 0.06–1.68 GB of weights. Qwen3.5-9B was the exception: I had no published MTP head to use for it.
The six-model comparison
These are generation speeds in tokens per second. The five MTP-capable models each ran through llama-server twice per setting, on the same coding prompt, with a 300-token output limit, temperature 0 and thinking disabled.
| Model | llama.cpp, MTP off | llama.cpp, MTP on | MTP gain | MLX, speculation off |
|---|---|---|---|---|
| Gemma 4 E4B | 43.5 | 55.7 | 1.28× | 52.4 |
| Qwen3.5-9B | 37.1* | Not tested: no head | — | 57.1 |
| Gemma 4 12B | 27.3 | 37.4 | 1.37× | 33.1 |
| Qwen3.8-27B | 12.0 | 19.1 | 1.59× | 19.0 |
| Gemma 4 31B | 13.6 | 18.1 | 1.34× | 14.9 |
| Qwen3.6-35B-A3B | 53.3 | 81.0 | 1.52× | 67.8 |
The on/off pairs measure MTP’s effect within llama.cpp. The MLX column and the starred Qwen 9B value come from the separate 512-token prompt / 128-token generation benchmark. Cross-engine comparisons therefore differ in workload, harness and quantization. All figures are two-run averages.
MLX supports draft-model speculation, but it was blocked by the Qwen cache implementation tested here. Gemma speculation was untested.
MTP improved generation by 28–59% across the five eligible models. In the configurations measured, llama.cpp with MTP was ahead on four models and effectively tied with MLX on Qwen 27B. A 19.1-versus-19.0 result is too close to call a meaningful win. Qwen 9B remained faster on MLX.
There is also prompt processing, or prefill: how quickly the model reads its input before writing an answer. llama.cpp led on five of six models in the original benchmark; MLX led on Gemma E4B. Both speeds matter as an agent accumulates conversation and tool results.
Original prompt and generation results, with speculation off
| Model | Prompt: llama.cpp | Prompt: MLX | Generation: llama.cpp | Generation: MLX |
|---|---|---|---|---|
| Gemma 4 E4B | 877.5 | 1072.7 | 63.6 | 52.4 |
| Qwen3.5-9B | 455.8 | 376.8 | 37.1 | 57.1 |
| Gemma 4 12B | 344.7 | 242.9 | 36.7 | 33.1 |
| Qwen3.8-27B | 134.5 | 107.3 | 11.8 | 19.0 |
| Gemma 4 31B | 127.5 | 91.6 | 16.5 | 14.9 |
| Qwen3.6-35B-A3B | 891.7 | 636.3 | 54.5 | 67.8 |
This benchmark used a 512-token prompt, 128-token generation and two repetitions. Its figures should not be used as the off half of the server MTP test. For example, Gemma E4B’s matched server comparison is 43.5 → 55.7, even though its separate llama-bench result was 63.6.
Qwen 27B also exposed a precision mismatch
The latest server test puts Qwen 27B at 19.1 tokens per second with MTP, up from 12.0 without it, with 78% draft acceptance. This is the result I would use for my serving setup. MLX’s separate baseline measured about 19.0.
But “4-bit” does not mean the same thing in both files. The Qwen 27B GGUF contains 7.16 GB of 8-bit tensors, including Gated-DeltaNet and attention projections, plus 1.46 GB of 6-bit attention-output tensors. MLX uses 4-bit quantization for the corresponding projections. The GGUF main model is 18.97 GB; the MLX download is 16.05 GB, including a vision tower unused for text generation.
Reading fewer bytes helps explain part of MLX’s raw speed advantage. Implementation differences on the Gated-DeltaNet path matter too, including support for rollback. The six-model family split alone cannot tell us which engine is better.
I have not measured perplexity or answer accuracy. Higher precision may preserve quality, but a coherent answer to one prompt cannot establish that. These are speed measurements at different precision choices.
Why the 35B model is my pick
Qwen3.6-35B-A3B has roughly 35 billion parameters, with about 3 billion active per token. Its full weights still occupy about 20.4 GB, plus a 1.06 GB MTP head, but each generated token uses a subset of the model.
At 81.0 tokens per second with MTP, it is the large model I want to keep testing on this 32 GB machine. Qwen3.5-9B on MLX is another useful option: around 6 GB of weights and 57.1 tokens per second, leaving more memory for everything else. This is a choice based on speed and memory, not a model-quality ranking.
The setup and the limits
Measurements were taken on 15–16 September 2026, one model at a time, with one stream and High Power Mode enabled for the engine comparisons.
| Component | Tested configuration |
|---|---|
| Hardware | M1 Max, 10 CPU cores, 32 GPU cores, 32 GB unified memory |
| System | macOS 26.6, build 25G5065a; AC power |
| llama.cpp | 0.4.1, build 10964, commit b29c606e2; Metal backend |
| MLX | mlx 0.32.2; mlx-lm 0.32.0 from git, commit 872ae88d1fac |
| Python | 3.13.12 |
| Quantization | Qwen GGUF: Q4_K_M; Gemma GGUF: QAT q4_0; MLX: affine 4-bit, group size 64 |
The PyPI mlx-lm 0.31.3 build I tried could not load Gemma 12B or E4B. Both loaded with the git build above. All support and cache observations here refer to these tested versions.
There are limits: two repetitions, one machine, unequal precision and no answer-quality evaluation. The Gemma MLX results came from an earlier run under the same conditions and build, before I removed those weights to reclaim disk space. Gemma E4B also has a substantial file-size mismatch between formats. I did not benchmark long-context performance at 8k–32k or complete agent tasks. Gemma’s untested MLX draft-model speculation could change its comparison.
What I am taking to DevFest
I am keeping llama.cpp and Qwen3.6-35B-A3B for my local agent experiments. The prompt-processing results, MTP gains and working rollback support make that a practical choice on this machine. The lesson for my demo is to test the server configuration I will actually use, including speculation, and check what “4-bit” contains.
The next step is to connect these models to a local agent harness, with its tools, memory and inference all running on your machine, and build your own fully local agent.