Quantization Insights

For Qwen3.5 MoE models (35B-A3B, 122B-A10B), I tested quants from both AesSedai (MoE-aware quantization) and Unsloth (Dynamic 2.0). The differences are significant and measurable.

I ran the full 7-benchmark suite on Gemma-4-26B-A4B at both UD-Q4_K_XL (16 GiB) and Q8_0 (25 GiB). The MoE architecture activates 3.8B parameters per token regardless of quant level, so I expected the differences to be negligible. They weren't.

Q8 improved creative writing by 5 points (23 to 28 out of 30). The Q4 run produced competent prose with decent character voices but relied on familiar beats. The Q8 run produced richer sensory detail, sharper dialogue, and a more specific emotional arc. The characters felt less like templates and more like people. That gap matters for a writing benchmark.

Polyglot coding gained 5 points (10 to 15 out of 65). The Q8 model solved challenges that Q4 couldn't, including two additional bash pipeline tasks. Both quants scored identically on LRU (10/10), FastAPI (8/8), and LeetCode (59/59), confirming that well-saturated coding benchmarks can't tell quants apart.

PostgreSQL moved from 44 to 45, a single point from T1 complex SQL. Cassandra dropped from 38 to 29. That 9-point regression came entirely from T4 procedural challenges (14 to 8), where the model writes CQL schemas and executes them against a live cluster. I suspect run-to-run variance rather than a real quality regression. These T4 challenges are sensitive to exact token sequences and a different code path can cascade into multiple test failures. One bad CREATE TABLE propagates through every subsequent test in that challenge.

The performance tradeoff is measurable. Token generation dropped 10% (52.9 to 47.7 t/s on RADV) while prompt processing improved 9% (1196 to 1303 t/s). Disk usage went from 16 to 25 GiB. For a model I run interactively, 48 t/s is still fast enough and the writing quality gain justified the switch. I deleted the Q4 from disk.

Google released official quantization-aware-trained Q4_0 GGUFs for the Gemma 4 line on June 5. I pulled the 31B and the 26B-A4B the same day and ran both through the tooling benchmark against the quants already on the board. QAT models are trained with quantization in the loop, so the q4_0 file is supposed to behave like the checkpoint rather than like a compressed copy of it. The marketing usually claims near full-precision quality at a quarter the size. This was a chance to test that claim against same-day baselines on identical hardware.

The 31B result backs the claim. At nearly the same file size as the Unsloth dynamic quant, the QAT build scored 59.5 against 54.5, ran all 65 calls machine-usable against 96.9 percent, and posted 98.5 percent strict parsing, the best zero-recovery rate of any run so far. It also passed the email format contract on all three trials where the Unsloth quant managed one of three. Most of the point gap is the chronic Gemma habit of writing summaries just under a stated word minimum, which the QAT build did three fewer times.

The 26B-A4B result cuts the other way. The Q8_0 outscored the QAT build 59 to 54, and every lost point was a summary in the 129 to 147 word range against a 150 word minimum. The QAT build still kept 100 percent of its calls usable and averaged 4.7 seconds per call, the fastest run the benchmark has produced, at roughly half the Q8 file size. One oddity worth recording with caution. The Q8_0 fell for an embedded prompt injection, emitting the exact demanded payload in its summary field, while the QAT build resisted the same fixture. That is one trial per quant at temperature 0.7, so it may be sampling noise rather than a real hardening effect.

These are tooling-only numbers from single runs. The full quality suite (writing, polyglot, databases) still needs to run on the 26B-A4B QAT build, but I have now put the 31B QAT through all of it, and the addendum below records where it landed. For automation serving specifically, the QAT 26B-A4B already looks like the practical choice, since the points it gives up are word-count preferences no consuming parser checks.

Addendum, June 8. I ran the 31B QAT through the entire quality suite to settle whether it can replace the Unsloth dynamic quant outright, not just serve automation. It holds. Writing, LRU, FastAPI, and LeetCode all match the Unsloth quant cell for cell, and the two only part ways on the databases and polyglot.

On the recorded totals the QAT edges the reference quant, 241 Combined to 238 and 59.5 tooling to 54.5, at about a gigabyte less on disk. I don't bank the polyglot jump from 44 to 54, since the Unsloth 44 predates an extraction fix and I haven't rerun it on the current build, so I read that row as a wash rather than a real gain. Set it aside and the QAT gives back exactly two things, one Postgres point and six Cassandra points, and every one of the Cassandra losses sits in the T4 procedural tier, the TWCS schemas and lightweight-transaction locks and materialized views. Everywhere else the two quants are the same model.

So the 31B QAT matches the Unsloth quant on quality, wins tooling reliability, and runs a touch smaller. Unless you specifically lean on Cassandra schema design, it's the better of the two to keep. I also put it through the agentic Sokoban build three times, where it cleared the full cumulative game in two of three runs, edging the Unsloth quant's one of three on the same boundary-phase coin flip. That settled it, so the QAT is now the 31B I keep on disk and the Unsloth quant is gone.

I also ran Kimi-Linear-48B-A3B at Q8_0 (49 GiB) against the primary Q4_K_M (28 GiB). The linear attention architecture activates 3B parameters per token. The results were less interesting than Gemma.

Writing quality was identical. Both runs scored 30/30 (the maximum). LRU stayed at 10/10. LeetCode improved from 57 to 59 (two more extraction successes, not model quality). Polyglot dropped from 22 to 15, likely from different runtime availability rather than quant regression. PostgreSQL gained 2 (26 to 28). Cassandra gained 1 (24 to 25). FastAPI dropped from 8 to 2, which I attribute to run variance.

Token generation took a 26% hit (72 to 53 t/s). Disk usage nearly doubled (28 to 49 GiB). The net Combined difference was zero after accounting for the API regression. I deleted the Q8 and kept Q4_K_M as the primary quant.

MXFP4 (Microscaling FP4) quantization behaves very differently depending on how it was applied.

Qwen3.5 dense models (4B, 9B, 27B) exhibit a token generation speed bottleneck that is consistent across all backends and quants. This appears to be baked into the architecture itself rather than caused by quantization or driver choice.

Qwen3.5 models require specific sampling parameters recommended by Unsloth. Default llama.cpp params cause dramatic failures. 0/10 coding on the 27B, prose loops on the 35B, and excessive repetition across all sizes.

Previously reported as an "architecture-level issue" affecting all quants. Root cause was missing sampling parameters. With Unsloth-recommended presence_penalty=1.5, the loop is completely eliminated.