What five weeks of local coding agents taught me

In September I benchmarked six open-weights models on my M1 Max and picked one for a local agent. That post ended with “the next step is to connect these models to a local agent harness”. This post is what happened next.

Since then I have run an overnight bake-off between three models, watched a 27B model spend 42 minutes building a voxel zen temple, demoed a fully offline agent at GDG Melbourne DevFest, moved to a new machine, and tested speculative decoding at more depths than I care to admit. Most of what I learned had little to do with which model is “best”.

The setup, for context: llama.cpp↗’s llama-server serving an OpenAI-compatible API on localhost, and pi↗ as the coding agent harness. pi is small. It gives the model a few tools (read, write, edit, bash), loads project skills, and gets out of the way. Nothing leaves the machine.

Check the machine before you blame the model

My first lesson from September still holds: Low Power Mode halved every number on the M1 Max. On 11 October I moved to an M2 Max with 96 GB, and the machine found new ways to mislead me.

In a warm room, the M2 throttled under sustained GPU load. After about two minutes in Automatic mode, or three in High Power, the thermal pressure reading went to “heavy” and long benchmarks dropped 10 to 27 percent. I moved to a colder room and ran the same sequence. Qwen3.8-27B generation went from 11.1 to 15.5 tokens per second. Same laptop, same files, different room.

Memory caught me out too. llama-server keeps a host-RAM prompt cache that defaults to 8 GiB. On a nearly full disk, during long batch runs, that cache quietly became 8 GB of swap. Generation speed never changed, so the only symptom was free disk space falling. --cache-ram 1024 fixed it.

Before timing anything now, I check three things: the energy mode, the charger wattage, and swap.

pmset -g | grep powermode        # 2 = High Power
notifyutil -g com.apple.system.thermalpressurelevel
sysctl vm.swapusage

Speculative decoding is the biggest free speedup

A model normally generates one token at a time. Speculative decoding lets something cheaper guess several tokens ahead, and the main model checks all the guesses in a single pass. Correct guesses are kept, wrong ones are thrown away. The output is checked against the main model either way, so you get the same quality, faster.

I tested three kinds of guesser:

• MTP heads: a small extra prediction head trained alongside the model. Qwen and Gemma ship them as separate files.
• DFlash 2: a separate drafter model, about 1.1 GB for Qwen3.8-27B.
• ngram-mod: no model at all. It looks for text already in the context and proposes the continuation it saw before.

That last one turned out to matter most for coding agents. When an agent edits a file, it re-types a lot of code it has just read. ngram-mod predicts those spans almost perfectly. On the M1 Max, Qwen3.6-35B-A3B went from 94 to 219–281 tokens per second on file edits. Qwen3.8-27B went from 23 to 59. New code and prose barely changed. I now run MTP and ngram-mod together by default.

The best depth changed with a llama.cpp update

How many tokens should the guesser propose each step? On the M1, three was the sweet spot. Longer drafts were rejected often enough that checking them cost more than they saved.

Then llama.cpp 0.7.0 shipped faster Metal kernels for small verify batches. On the M2, checking 3 to 8 tokens now costs about 1.5 times a single step on the dense 27B model. The optimum moved:

Qwen3.8-27B on M2 Max, llama.cpp 0.7.0	Tokens per second
No speculation	16.0
MTP, depth 3	26.7
MTP, depth 6	29.8
DFlash 2, up to 8 drafts	33.4

These are averages over 28 cold prompts across seven kinds of task, at temperature 0.6. The kernel works in tiles of 8, so a batch of 9 or more tokens costs about twice as much. At temperature 0, MTP on Qwen3.8 ran at 30.9 tokens per second with 6 drafts and fell to 22.7 with 8.

The mixture-of-experts Qwen3.6-35B-A3B behaves differently. Its verify cost still grows about 20 percent per token, so depth 3 stays best: 70 tokens per second without speculation, 99.5 with MTP at depth 3, and slower again at depth 6.

Speculation does not always pay. Three cases where it lost:

• Ornith-1.5-122B, a 122B MoE I tried the day after release, has an MTP head built in. It made things slower. ngram-mod alone gave 37 tokens per second against 36.3 with nothing.
• Bonsai 2 27B, a ternary-weight model, ran 1.9 times slower with an MTP head, even at 65 percent acceptance. Its bottleneck is unpacking weights, not reading them, and speculation only helps with the second.
• A tiny 0.8B draft model made Qwen3.6 slower (0.71 times).

The lesson is that speculative decoding settings belong to a model, a chip and an engine version together. Advice written for an older llama.cpp build, or for a different Mac, may be pointing the wrong way.

A benchmark trap

ngram-mod keeps its cache between requests. If you send the same prompt twice at temperature 0, the second run replays the first answer and “decodes” prose at 50 to 60 tokens per second. My first test matrix averaged a cold run and a warm one and showed a 2 to 3 times gain on everything. It was fake. Only cold, unique prompts give honest numbers.

Context depth costs more than the spec sheet says

Agents spend most of their time at 10k to 60k tokens of context. My early benchmarks used short prompts, and they hid a big problem.

I had been storing the KV cache (the model’s working memory of the conversation) in 8-bit to save memory. At depth, that halved generation:

Qwen3.8-27B, no speculation	2k context	20k	44k
KV cache q8_0	11.8	7.1	4.7
KV cache f16	12.7	11.9	10.8

It was not memory pressure. The 8-bit path in Metal flash attention is slow on this hardware. On the 32 GB M1, f16 meant giving up the MTP head to fit. On the 96 GB M2, f16 plus MTP fits, and Qwen3.8 holds 23.3 tokens per second at 2k and 16.3 at 60k. If you only benchmark with a short prompt, you will never see this.

Give the model a thinking budget

Reasoning models think before they answer. Left alone, a local model will sometimes think forever.

While building my DevFest slides, Qwen3.6 spent its entire 24,000-token output limit re-drafting one Vue file inside its reasoning. It never answered. The odd part was the speed: 125 tokens per second, about twice normal, because ngram-mod was predicting its own repeats. An unusually fast decode is now my sign that a model is looping.

llama-server accepts a per-request thinking_budget_tokens field. When the model reaches the budget, the server appends a short message and forces it to answer. I send 8k for agent turns and lower for simple ones. pi can send this field for you (config below).

Out of curiosity, I also tried the opposite. I gave Qwen3.8-27B unlimited reasoning at its highest effort level and a 256k context, and asked it to build the same zen temple scene again. It took 205 minutes and 176k output tokens, against 98 minutes and 59k tokens with an 8k cap. The result was better in places (a grand stepped platform, better night lighting) and worse in others (the pagoda, the trees, a bridge that looked like a red pyramid). More thinking did not fix spatial design. Better feedback did, which brings me to the harness.

The harness matters more than the model

In June I wrote that a frontier model is a guest, and the harness is what you keep. Running local models made that concrete. A weaker model gains far more from good tools and instructions than a strong one does.

Some changes that made a measurable difference:

• Report file and line numbers. Bonsai spent eight turns failing to find a missing brace from the error “x is not defined”. I added the first stack frame to the check script’s output. With that line added, it found the bug in two turns.
• Give numeric targets for visual work. A text-only model cannot see the scene it built. Qwen3.6’s first temple had heavy haze and a pitch-black night. I extended the check script to print brightness, contrast and colourfulness for each screenshot, and gave targets. It fixed all four complaints in one polish round.
• Tell it to run code instead of doing arithmetic in its head. For a daylight-saving demo, Qwen3.6 spent two full 8k thinking turns working out time zones and got them wrong. One sentence (“check your reasoning by running small Python snippets”) brought the run from 9 minutes 41 seconds to between 1.5 and 3.5 minutes, with correct answers.
• Force small steps. My small-steps skill asks for a PLAN.md checklist, one verified change at a time, and a rule to stop and re-read the file after two failed fixes. With the skill and a thinking budget, Qwen3.8 finished its first temple in 30 turns with no runaway reasoning. The same prompt had earlier died on Bonsai, which is built from the same base model, with reasoning that ran out of tokens before it wrote anything.
• Clean up tool arguments. Qwen3.6 fills unused optional parameters with the string "None". That string is truthy, so my code took the wrong branch. One session produced 14 identical failures and then a retry storm against a rate-limited API. I now strip "None", "null", empty strings and friends in pi’s prepareArguments hook before validation.

None of this is clever. It is the same thing you would do for a new teammate: clear errors, small tasks, a checklist, and a way to see their own work.

Pick the model for the job

On 1 October I ran an overnight bake-off. Three models, the same pi harness, samplers, thinking budget and context, a 40-minute cap, sandboxed agents and hidden graders.

	Qwen3.8-27B	Bonsai 2 27B	Qwen3.6-35B-A3B
Website task (score out of 18)	18, 18	17, 18	16, 18, 16
Website time	~28 min	40 min (hit cap)	2–3 min
Bug fix (out of 24)	24, 24	24, 24	24, 24, 24
Bug fix time	8–18 min	13–21 min	under 1 min
Memory used	24.6 GB	12.0 GB	23.4 GB

The tasks were too easy. Every model nearly maxed them, so the useful signal is time. Qwen3.6 is roughly ten times faster than the dense 27B models, because only about 3 billion of its 35 billion parameters run for each token. Qwen3.8 produced the best visual work in every temple comparison. Bonsai used half the memory.

So my everyday default is Qwen3.6-35B-A3B for speed. I switch to Qwen3.8-27B when quality matters more than waiting.

The 96 GB machine also fits something much bigger. Ornith-1.5-122B is a mixture-of-experts model with about 10 billion parameters active per token, and it uses 70 GB of memory, so it has to run alone. Here it is in pi building a landing page, with llama-server’s log above.

Two things in the log are worth pointing out. Generation holds at about 29 tokens per second at 26k tokens of context, down from 37 on a short prompt. And each new turn only processed the new tokens (99 of 25,813 in one request), because the server reuses the cached prefix of the conversation. That prompt cache is what makes long agent sessions bearable on a Mac. Without it, every turn would re-read the whole conversation.

Sandbox your agent from the first run

During that bake-off, Bonsai was fixing a Python package and ran find / -name parse.py, looking for where its import resolved. That search would have reached the reference solution and the other models’ finished work. I had moved the task folders out of the experiment tree and thought that was enough. It was not.

I killed the run and wrapped every agent in macOS sandbox-exec, denying reads outside its own task folder. pi, Python and headless Chrome all work inside it. A tool-using model will look outside its folder without being asked, and you will not notice unless you read the tool calls.

The same profile with outbound network denied (except localhost) is how I proved the DevFest demo needed no internet.

My pi config for local coding

This is what I run day to day. First, the server. My serve.sh script boils down to this for Qwen3.6:

llama-server \
  -m Qwen3.6-35B-A3B-Q4_K_M.gguf --alias qwen3.6-35b-a3b \
  --host 127.0.0.1 --port 8080 \
  -ngl 99 -fa on -ctk f16 -ctv f16 \
  -c 262144 -np 1 --jinja --metrics \
  --spec-type draft-mtp,ngram-mod -md mtp-Qwen3.6-35B-A3B.gguf \
  --spec-draft-n-max 3 --spec-draft-sampling probabilistic

--jinja uses the model’s own chat template, which is what makes tool calls and reasoning tags work. For Qwen3.8 the script swaps in the DFlash 2 drafter with --spec-type draft-dflash,ngram-mod and --spec-draft-n-max 8, and a 131k context.

Then ~/.pi/agent/models.json, trimmed to one model:

{
  "providers": {
    "llama_server": {
      "api": "openai-completions",
      "baseUrl": "http://127.0.0.1:8080/v1",
      "apiKey": "LOCAL_LLM_KEY",
      "models": [
        {
          "id": "qwen3.6-35b-a3b",
          "contextWindow": 262144,
          "maxTokens": 32768,
          "samplingParams": {
            "temperature": 0.6, "top_p": 0.95, "top_k": 20,
            "min_p": 0.0, "presence_penalty": 0.0
          },
          "reasoning": true,
          "compat": {
            "thinkingFormat": "chat-template",
            "chatTemplateKwargs": {
              "enable_thinking": { "$var": "thinking.enabled" },
              "preserve_thinking": false
            },
            "thinkingTokenBudgetField": "thinking_budget_tokens"
          }
        }
      ]
    }
  }
}

And ~/.pi/agent/settings.json:

{
  "defaultProvider": "llama_server",
  "defaultModel": "qwen3.6-35b-a3b",
  "compaction": { "reserveTokens": 32768 }
}

A few details here cost me time to learn:

• contextWindow must match the server’s -c. If pi thinks it has more room than the server, long sessions get truncated or fail.
• Keep maxTokens equal to reserveTokens. pi lets the context grow to the window minus the reserve before it compacts. Any output budget above the reserve is unusable near that point.
• The server ignores the model name you ask for. llama-server answers with whatever it loaded. Point pi at the wrong id and you get the wrong model under the wrong context limit, with no error. Restart the matching server when you switch.
• preserve_thinking: false. The Qwen template default replays past reasoning into the context, which fills it fast.
• thinkingTokenBudgetField is what sends the thinking budget on each request. For long runs I set budgets per level in the project’s .pi/settings.json, for example "thinkingBudgets": { "low": 8192 }.
• Set every sampler explicitly. Server defaults differ from the model card.
• Check after updating pi. pi 1.3.0 started requiring "$var" in those template arguments. My old file failed validation, pi dropped the whole provider, and the error said “No API key found”. It had nothing to do with keys. pi --list-models shows the real warning.

apiKey is a dummy string. pi needs a provider to look authenticated before it will use it as the default.

On top of that sit a few project extensions: a webfetch extension with a web_search tool (DuckDuckGo, then Brave, then DuckDuckGo Lite, because local models otherwise guess search URLs and decide things do not exist), and a draft-only extension for postme.live↗ that has no way to publish or schedule. My web experiments folder has six skills: page checks with headless Chrome, small steps, web research, three.js scenes, landing pages and image generation. Skills are where most of the improvement came from.

Limitations

I want to be clear about what all this does not show.

• One person, one kind of machine. Every number here is from Apple Silicon, mostly a 96 GB M2 Max that is far from a typical laptop. The rankings could change on NVIDIA hardware or a newer Mac.
• Small samples. Two or three runs per bake-off cell, 28 prompts in the speculation matrix, tasks easy enough that every model nearly maxed them. That is enough to spot large differences, not small ones.
• Mostly speed, little quality. I measured tokens per second carefully and answer quality loosely. Ornith-1.5-122B fits and runs at 37 tokens per second, but I have not tested whether it codes better than Qwen3.8. Its published benchmarks come from its makers.
• Dense models are slow to read. Qwen3.8 prefills at 100 to 170 tokens per second. Re-reading a large session from cold can take tens of minutes. This is why my default is the MoE model, not the best one.
• Text-only models are blind. Without vision, the model cannot see the page or chart it made. Numeric checks help, but my live demo charts broke in four of six bare runs until a skill drew them safely.
• Local models are still behind the frontier. They handled my bug fixes and small builds well. I would not hand them an unfamiliar large codebase and walk away.
• Settings go stale quickly. One llama.cpp release moved the best draft depth from 3 to 6 or more. One pi release broke my config. Anything in this post about flags and depths has a short shelf life.
• Speculation is not perfectly identical output. At MTP depth 2 or more on Qwen3.8, greedy output matched the non-speculative run in only 18 of 28 prompts. The differences come from numerics in the batched kernel, not from the guesser, but it means you cannot use exact output matching to test it.
• I had help measuring. I ran most of these experiments with Claude Code writing the benchmark scripts and keeping the notes. That is a large part of why I could test this much in five weeks, and the local models are not yet at the point where they could have done that work themselves.

The short version: a local coding agent on a Mac is real and useful today for focused tasks. Most of the speed comes from server settings you have to test yourself, and most of the reliability comes from the harness you build around the model.