# Six local AI models on my M1 Max: llama.cpp vs MLX

Author: Bishal Sapkota
Published: 2026-09-16
Canonical: https://bishal.io/blog/six-local-models-m1-max-llama-cpp-vs-mlx/

Six open-weights models on a 32 GB M1 Max. Power settings, speculative decoding and what the benchmark numbers leave out.

I spent today testing six open-weights models on my 2021 MacBook Pro. M1 Max, 32 GB of unified memory. I wanted to find a model and inference engine I could use for a fully local agent.

This is preparation for my talk, **“Wonderful World of Open Weights AI”**, at GDG Melbourne DevFest on **3 October 2026**. I compared [llama.cpp](https://github.com/ggml-org/llama.cpp) and [MLX through mlx-lm](https://github.com/ml-explore/mlx-lm), using builds labelled 4-bit for the same six models.

One setting changed how I read the results. Normally, a language model generates one token at a time. A token is a small piece of text, often part of a word. **Multi-token prediction (MTP)** uses an extra prediction head to propose several upcoming tokens. The main model checks the proposals together, accepts a valid continuation and discards the rest. This is a form of **speculative decoding**. Accept enough proposals, and generation gets faster.

With MTP enabled in llama-server, **Qwen3.6-35B-A3B averaged 81.0 tokens per second**. That is the model I am keeping for my local agent experiments. Getting there involved a power setting, a separate server setting, and a limitation I only found by trying it.

## Check Low Power Mode first

My early numbers were well below what I expected from this chip. The Mac was in Low Power Mode.

Switching from Low Power to High Power roughly doubled prompt-processing speed on every model I checked. Qwen3.8-27B went from **68 to 136 tokens per second**. Gemma 4 31B went from **64 to 130**.

Generation speed changed unevenly: Gemma 4 31B went from 7.9 to 17.0 tokens per second, while Qwen3.6-35B-A3B went from 49.0 to 57.5. These were separate power-mode checks, not the MTP comparison below.

```bash
pmset -g | grep powermode
```

On this machine, `1` meant Low Power, `0` Automatic and `2` High Power. [Apple describes the modes here](https://support.apple.com/en-us/101613). I measured Low versus High, so I cannot claim a separate gain from High over Automatic. Check the setting before blaming the model.

## MTP is a llama-server setting

Downloading a model with an MTP head does not enable speculative decoding by itself. In my tested llama.cpp build, I had to load the matching draft-head file and select the mode in **llama-server**:

```bash
llama-server -m model.gguf -ngl 99 \
  --spec-type draft-mtp -md matching-mtp-head.gguf
```

The command shows the relevant setting; the filenames stand for the main model and its matching head. For the on/off test, I kept the other server settings unchanged and removed the two speculation options for the off run. `llama-bench`, which I used for the original comparison, does not exercise this path.

### Which models supported MTP?

I found and tested draft heads for **Gemma 4 E4B, Gemma 4 12B, Qwen3.8-27B, Gemma 4 31B and Qwen3.6-35B-A3B**. They added about 0.06–1.68 GB of weights. Qwen3.5-9B was the exception: I had no published MTP head to use for it.

## The six-model comparison

These are generation speeds in tokens per second. The five MTP-capable models each ran through llama-server twice per setting, on the same coding prompt, with a 300-token output limit, temperature 0 and thinking disabled.

| Model | llama.cpp, MTP off | llama.cpp, MTP on | MTP gain | MLX, speculation off |
|---|---:|---:|---:|---:|
| Gemma 4 E4B | 43.5 | **55.7** | 1.28× | 52.4 |
| Qwen3.5-9B | 37.1* | Not tested: no head | — | **57.1** |
| Gemma 4 12B | 27.3 | **37.4** | 1.37× | 33.1 |
| Qwen3.8-27B | 12.0 | **19.1** | 1.59× | 19.0 |
| Gemma 4 31B | 13.6 | **18.1** | 1.34× | 14.9 |
| Qwen3.6-35B-A3B | 53.3 | **81.0** | 1.52× | 67.8 |

**The on/off pairs measure MTP's effect within llama.cpp.** The MLX column and the starred Qwen 9B value come from the separate 512-token prompt / 128-token generation benchmark. Cross-engine comparisons therefore differ in workload, harness and quantization. All figures are two-run averages.

<span id="mlx-supports-speculation-these-qwen-caches-block-it"></span>

*MLX supports draft-model speculation, but it was blocked by the Qwen cache implementation tested here. Gemma speculation was untested.*

MTP improved generation by **28–59%** across the five eligible models. In the configurations measured, llama.cpp with MTP was ahead on four models and effectively tied with MLX on Qwen 27B. A 19.1-versus-19.0 result is too close to call a meaningful win. Qwen 9B remained faster on MLX.

There is also **prompt processing**, or prefill: how quickly the model reads its input before writing an answer. llama.cpp led on five of six models in the original benchmark; MLX led on Gemma E4B. Both speeds matter as an agent accumulates conversation and tool results.

<details>
<summary>Original prompt and generation results, with speculation off</summary>

| Model | Prompt: llama.cpp | Prompt: MLX | Generation: llama.cpp | Generation: MLX |
|---|---:|---:|---:|---:|
| Gemma 4 E4B | 877.5 | 1072.7 | 63.6 | 52.4 |
| Qwen3.5-9B | 455.8 | 376.8 | 37.1 | 57.1 |
| Gemma 4 12B | 344.7 | 242.9 | 36.7 | 33.1 |
| Qwen3.8-27B | 134.5 | 107.3 | 11.8 | 19.0 |
| Gemma 4 31B | 127.5 | 91.6 | 16.5 | 14.9 |
| Qwen3.6-35B-A3B | 891.7 | 636.3 | 54.5 | 67.8 |

This benchmark used a 512-token prompt, 128-token generation and two repetitions. Its figures should not be used as the off half of the server MTP test. For example, Gemma E4B's matched server comparison is 43.5 → 55.7, even though its separate llama-bench result was 63.6.

</details>

<span id="the-biggest-mlx-win-needs-a-footnote"></span>
<span id="qwen-27b-reaches-157-tokens-per-second-with-mtp"></span>
<span id="qwen-27b-172-tokens-per-second-with-mtp"></span>

## Qwen 27B also exposed a precision mismatch

The latest server test puts Qwen 27B at **19.1 tokens per second with MTP**, up from **12.0 without it**, with **78% draft acceptance**. This is the result I would use for my serving setup. MLX's separate baseline measured about 19.0.

But “4-bit” does not mean the same thing in both files. The Qwen 27B GGUF contains **7.16 GB of 8-bit tensors**, including Gated-DeltaNet and attention projections, plus **1.46 GB of 6-bit attention-output tensors**. MLX uses 4-bit quantization for the corresponding projections. The GGUF main model is 18.97 GB; the MLX download is 16.05 GB, including a vision tower unused for text generation.

Reading fewer bytes helps explain part of MLX's raw speed advantage. Implementation differences on the Gated-DeltaNet path matter too, including support for rollback. The six-model family split alone cannot tell us which engine is better.

I have not measured perplexity or answer accuracy. Higher precision may preserve quality, but a coherent answer to one prompt cannot establish that. These are speed measurements at different precision choices.

## Why the 35B model is my pick

[Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) has roughly 35 billion parameters, with about 3 billion active per token. Its full weights still occupy about 20.4 GB, plus a 1.06 GB MTP head, but each generated token uses a subset of the model.

At **81.0 tokens per second with MTP**, it is the large model I want to keep testing on this 32 GB machine. Qwen3.5-9B on MLX is another useful option: around 6 GB of weights and 57.1 tokens per second, leaving more memory for everything else. This is a choice based on speed and memory, not a model-quality ranking.

## The setup and the limits

Measurements were taken on 15–16 September 2026, one model at a time, with one stream and High Power Mode enabled for the engine comparisons.

| Component | Tested configuration |
|---|---|
| Hardware | M1 Max, 10 CPU cores, 32 GPU cores, 32 GB unified memory |
| System | macOS 26.6, build 25G5065a; AC power |
| llama.cpp | 0.4.1, build 10964, commit `b29c606e2`; Metal backend |
| MLX | `mlx` 0.32.2; `mlx-lm` 0.32.0 from git, commit `872ae88d1fac` |
| Python | 3.13.12 |
| Quantization | Qwen GGUF: `Q4_K_M`; Gemma GGUF: QAT `q4_0`; MLX: affine 4-bit, group size 64 |

The PyPI mlx-lm 0.31.3 build I tried could not load Gemma 12B or E4B. Both loaded with the git build above. All support and cache observations here refer to these tested versions.

There are limits: two repetitions, one machine, unequal precision and no answer-quality evaluation. The Gemma MLX results came from an earlier run under the same conditions and build, before I removed those weights to reclaim disk space. Gemma E4B also has a substantial file-size mismatch between formats. I did not benchmark long-context performance at 8k–32k or complete agent tasks. Gemma's untested MLX draft-model speculation could change its comparison.

## What I am taking to DevFest

I am keeping llama.cpp and Qwen3.6-35B-A3B for my local agent experiments. The prompt-processing results, MTP gains and working rollback support make that a practical choice on this machine. The lesson for my demo is to test the server configuration I will actually use, including speculation, and check what “4-bit” contains.

The next step is to connect these models to a local agent harness, with its tools, memory and inference all running on your machine, and build your own fully local agent.