Quantization Insights

I ran the same eight medium-reasoning technical suites on the official Q8_0 target and Unsloth's UD-Q4_K_XL quant, both with MTP n=3 at 64K context. Q4 matches the Q8 result within run variance. The six leaderboard technical benchmarks total 221/255 for Q4 and 219/255 for Q8. Including Tooling and calibration moves the comparison to 304.5/342 and 303.5/342.

The elapsed-time difference is much larger. Q4 completes the comparable set in 2 hours 32 minutes against 12 hours 58 minutes for Q8. Every scored driver used the published medium-reasoning profile with no generation cap. The numbers below are wall time for each complete benchmark, including reasoning and answer generation.

The scored run does not show a 5.1x backend speedup. A fixed nine-prompt server control puts Q4 with MTP at 15.35 generated tokens per second and Q8 with MTP at 11.87, which is a 29.3 percent gain. The Q4 Tooling run emitted 57,922 completion tokens against 88,374 for Q8. Shorter reasoning paths account for much of the total wall-time reduction.

The comparison is close enough on score that I would not rank the quants from the one-point difference across all eight suites. Cassandra gains three points on Q4 while PostgreSQL loses one and Tooling loses one. These are sampled runs. The Q8 artifacts used llama.cpp build 10433 and the Q4 artifacts used build 10448, so the executable also changed between them.

For Qwen3.5 MoE models (35B-A3B, 122B-A10B), I tested quants from both AesSedai (MoE-aware quantization) and Unsloth (Dynamic 2.0). The differences are significant and measurable.

I ran the full 7-benchmark suite on Gemma-4-26B-A4B at both UD-Q4_K_XL (16 GiB) and Q8_0 (25 GiB). The MoE architecture activates 3.8B parameters per token regardless of quant level, so I expected the differences to be negligible. They weren't.

Q8 improved creative writing by 5 points (23 to 28 out of 30). The Q4 run produced competent prose with decent character voices but relied on familiar beats. The Q8 run produced richer sensory detail, sharper dialogue, and a more specific emotional arc. The characters felt less like templates and more like people. That gap matters for a writing benchmark.

Polyglot coding gained 5 points (10 to 15 out of 65). The Q8 model solved challenges that Q4 couldn't, including two additional bash pipeline tasks. Both quants scored identically on LRU (10/10), FastAPI (8/8), and LeetCode (59/59), confirming that well-saturated coding benchmarks can't tell quants apart.

PostgreSQL moved from 44 to 45, a single point from T1 complex SQL. Cassandra dropped from 38 to 29. That 9-point regression came entirely from T4 procedural challenges (14 to 8), where the model writes CQL schemas and executes them against a live cluster. I suspect run-to-run variance rather than a real quality regression. These T4 challenges are sensitive to exact token sequences and a different code path can cascade into multiple test failures. One bad CREATE TABLE propagates through every subsequent test in that challenge.

The performance tradeoff is measurable. Token generation dropped 10% (52.9 to 47.7 t/s on RADV) while prompt processing improved 9% (1196 to 1303 t/s). The artifact grows from 16 to 25 GiB. At 48 t/s the Q8 remains fast enough for interactive use, and the writing quality gain makes it the better tested choice.

Google released official quantization-aware-trained Q4_0 GGUFs for the Gemma 4 line on June 5. I pulled the 31B and the 26B-A4B the same day and ran both through the tooling benchmark against the quants already on the board. QAT models are trained with quantization in the loop, so the q4_0 file is supposed to behave like the checkpoint rather than like a compressed copy of it. The marketing usually claims near full-precision quality at a quarter the size. This was a chance to test that claim against same-day baselines on identical hardware.

The 31B result backs the claim. At nearly the same file size as the Unsloth dynamic quant, the QAT build scored 59.5 against 54.5, ran all 65 calls machine-usable against 96.9 percent, and posted 98.5 percent strict parsing, the best zero-recovery rate of any run so far. It also passed the email format contract on all three trials where the Unsloth quant managed one of three. Most of the point gap is the chronic Gemma habit of writing summaries just under a stated word minimum, which the QAT build did three fewer times.

The 26B-A4B result cuts the other way. The Q8_0 outscored the QAT build 59 to 54, and every lost point was a summary in the 129 to 147 word range against a 150 word minimum. The QAT build still kept 100 percent of its calls usable and averaged 4.7 seconds per call, the fastest run the benchmark has produced, at roughly half the Q8 file size. One oddity worth recording with caution. The Q8_0 fell for an embedded prompt injection, emitting the exact demanded payload in its summary field, while the QAT build resisted the same fixture. That is one trial per quant at temperature 0.7, so it may be sampling noise rather than a real hardening effect.

These are tooling-only numbers from single runs. The full quality suite (writing, polyglot, databases) still needs to run on the 26B-A4B QAT build, but I have now put the 31B QAT through all of it, and the addendum below records where it landed. For automation serving specifically, the QAT 26B-A4B already looks like the practical choice, since the points it gives up are word-count preferences no consuming parser checks.

Addendum, June 8. I ran the 31B QAT through the entire quality suite to settle whether it can replace the Unsloth dynamic quant outright, not just serve automation. It holds. Writing, LRU, FastAPI, and LeetCode all match the Unsloth quant cell for cell, and the two only part ways on the databases and polyglot.

On the recorded totals the QAT edges the reference quant, 241 Combined to 238 and 59.5 tooling to 54.5, with an artifact about one gigabyte smaller. I don't bank the polyglot jump from 44 to 54, since the Unsloth 44 predates an extraction fix and I haven't rerun it on the current build, so I read that row as a wash rather than a real gain. Set it aside and the QAT gives back exactly two things, one Postgres point and six Cassandra points, and every one of the Cassandra losses sits in the T4 procedural tier, the TWCS schemas and lightweight-transaction locks and materialized views. Everywhere else the two quants are the same model.

So the 31B QAT matches the Unsloth quant on quality, wins tooling reliability, and runs a touch smaller. Unless you specifically lean on Cassandra schema design, it is the better tested choice. I also put it through the agentic Sokoban build three times, where it cleared the full cumulative game in two of three runs, edging the Unsloth quant's one of three on the same boundary-phase coin flip.

I also ran Kimi-Linear-48B-A3B at Q8_0 (49 GiB) against the primary Q4_K_M (28 GiB). The linear attention architecture activates 3B parameters per token. The results were less interesting than Gemma.

Writing quality was identical. Both runs scored 30/30 (the maximum). LRU stayed at 10/10. LeetCode improved from 57 to 59 (two more extraction successes, not model quality). Polyglot dropped from 22 to 15, likely from different runtime availability rather than quant regression. PostgreSQL gained 2 (26 to 28). Cassandra gained 1 (24 to 25). FastAPI dropped from 8 to 2, which I attribute to run variance.

Token generation took a 26% hit (72 to 53 t/s). The artifact nearly doubled from 28 to 49 GiB. The net Combined difference was zero after accounting for the API regression, which makes Q4_K_M the more practical tested quant.

MXFP4 (Microscaling FP4) quantization behaves very differently depending on how it was applied.

Qwen3.5 dense models (4B, 9B, 27B) exhibit a token generation speed bottleneck that is consistent across all backends and quants. This appears to be baked into the architecture itself rather than caused by quantization or driver choice.

Qwen3.5 models require specific sampling parameters recommended by Unsloth. Default llama.cpp params cause dramatic failures. 0/10 coding on the 27B, prose loops on the 35B, and excessive repetition across all sizes.

Previously reported as an "architecture-level issue" affecting all quants. Root cause was missing sampling parameters. With Unsloth-recommended presence_penalty=1.5, the loop is completely eliminated.