Quantization Insights

For Qwen3.5 MoE models (35B-A3B, 122B-A10B), I tested quants from both AesSedai (MoE-aware quantization) and Unsloth (Dynamic 2.0). The differences are significant and measurable.

I ran the full 7-benchmark suite on Gemma-4-26B-A4B at both UD-Q4_K_XL (16 GiB) and Q8_0 (25 GiB). The MoE architecture activates 3.8B parameters per token regardless of quant level, so I expected the differences to be negligible. They weren't.

Q8 improved creative writing by 5 points (23 to 28 out of 30). The Q4 run produced competent prose with decent character voices but relied on familiar beats. The Q8 run produced richer sensory detail, sharper dialogue, and a more specific emotional arc. The characters felt less like templates and more like people. That gap matters for a writing benchmark.

Polyglot coding gained 5 points (10 to 15 out of 65). The Q8 model solved challenges that Q4 couldn't, including two additional bash pipeline tasks. Both quants scored identically on LRU (10/10), FastAPI (8/8), and LeetCode (59/59), confirming that well-saturated coding benchmarks can't tell quants apart.

PostgreSQL moved from 44 to 45, a single point from T1 complex SQL. Cassandra dropped from 38 to 29. That 9-point regression came entirely from T4 procedural challenges (14 to 8), where the model writes CQL schemas and executes them against a live cluster. I suspect run-to-run variance rather than a real quality regression. These T4 challenges are sensitive to exact token sequences and a different code path can cascade into multiple test failures. One bad CREATE TABLE propagates through every subsequent test in that challenge.

The performance tradeoff is real. Token generation dropped 10% (52.9 to 47.7 t/s on RADV) while prompt processing actually improved 9% (1196 to 1303 t/s). Disk usage went from 16 to 25 GiB. For a model I run interactively, 48 t/s is still fast enough and the writing quality gain justified the switch. I deleted the Q4 from disk.

I also ran Kimi-Linear-48B-A3B at Q8_0 (49 GiB) against the primary Q4_K_M (28 GiB). The linear attention architecture activates 3B parameters per token. The results were less interesting than Gemma.

Writing quality was identical. Both runs scored 30/30 (the maximum). LRU stayed at 10/10. LeetCode improved from 57 to 59 (two more extraction successes, not model quality). Polyglot dropped from 22 to 15, likely from different runtime availability rather than quant regression. PostgreSQL gained 2 (26 to 28). Cassandra gained 1 (24 to 25). FastAPI dropped from 8 to 2, which I attribute to run variance.

Token generation took a 26% hit (72 to 53 t/s). Disk usage nearly doubled (28 to 49 GiB). The net Combined difference was zero after accounting for the API regression. I deleted the Q8 and kept Q4_K_M as the primary quant.

MXFP4 (Microscaling FP4) quantization behaves very differently depending on how it was applied.

Qwen3.5 dense models (4B, 9B, 27B) exhibit a token generation speed bottleneck that is consistent across all backends and quants. This appears to be baked into the architecture itself rather than caused by quantization or driver choice.

Qwen3.5 models require specific sampling parameters recommended by Unsloth. Default llama.cpp params cause dramatic failures. 0/10 coding on the 27B, prose loops on the 35B, and excessive repetition across all sizes.

Previously reported as an "architecture-level issue" affecting all quants. Root cause was missing sampling parameters. With Unsloth-recommended presence_penalty=1.5, the loop is completely eliminated.