Model Comparison

Performance

Quality

The Tooling column is new and tracks automation reliability out of 65. It is scored separately and does not count toward the Combined total. Models marked with a hyphen have not run it yet.

Key Findings

• RADV dominates prompt processing across all model families (50-280% faster than AMDVLK on pp). Token generation is typically tied between drivers since it's bandwidth-bound. AMDVLK only wins on GPT-OSS-120B pp (661 vs 596 RADV).
• Kimi-Linear-48B is the new polyglot leader at 22/65, dethroning the GPT-OSS/Gemma trio (14/65). It's one of only two models to score on the Go rule engine (10/10) alongside Qwen3-30B (3/10). Linear attention gives it 72 t/s at 28 GiB.
• Devstral-Small-2 punches above its weight. A 24B dense coding model hitting 29/57 Postgres and 31/56 Cassandra while scoring 10/10 LRU and 59/59 LeetCode. The 15 t/s generation speed hurts but the quality-per-parameter is impressive.
• Nemotron-3-Nano collapses without thinking mode. 0/57 Postgres, 0/56 Cassandra, 0/59 LeetCode. The Mamba-2 hybrid architecture appears to need reasoning tokens enabled to produce structured output. Fast (1106 pp, 68 tg) but unusable for coding or database tasks with no-think.
• Magistral-Small has the highest Cassandra score (35/56) of any non-Gemma model, beating Devstral (31) and Kimi (24). But it scores 0/10 LRU, 12/57 Postgres, and leaked its reasoning scaffold into the creative-writing test output. A specialist, not a generalist.
• GLM-4.7-Flash can't code. 0/10 LRU, 0/8 FastAPI, 16/59 LeetCode (extraction failures), 0/65 polyglot. But it handles database work (23/57 PG, 27/56 Cass) and generates at 73 t/s. A fast model with a very narrow skill set.
• MiniMax-M2.5 went from 0/10 coding to 10/10 after switching to Unsloth's UD-Q3_K_XL quant and their recommended sampling params (temp 1.0, top_p 0.95, min_p 0.01, top_k 40). At 94 GiB it's the largest model on disk, but 185/285 Combined puts it 6th overall. Perfect 10/10 on both PostgreSQL T2 optimization and Cassandra T2 anti-pattern detection. AMDVLK is its best backend at 32 t/s (vs 22 on RADV).
• MiniMax-M2.7 is the 229B successor with an UD-IQ4_XS quant that squeezes into 101 GiB. Writing improved to 28/30 and all coding benchmarks max out (10/10 LRU, 59/59 LeetCode, 8/8 FastAPI). The model has an unusual no-think quirk. Its M2.5 predecessor required a custom Jinja template to suppress thinking. M2.7 reverses this entirely. The standard --reasoning-budget 0 flag works, but the custom template causes the model to reason inline in plain text without ever producing code. Combined 179/285 puts it just behind M2.5 at 185. PostgreSQL T2 optimization and T3 diagnosis are both perfect (9/10 and 6/6). Cassandra T4 procedural challenges benefit from leaving thinking on (7/30 vs 2/30 without). RADV is the right backend here, unlike M2.5 which preferred AMDVLK.
• Qwen3.6-35B-A3B is the hybrid-attention successor to the 3.5 MoE. Combined 205/285, writing 29/30, and polyglot 35/65 put it near the top, but the 35 polyglot score is best-of-5 rather than stable single-run behavior. Database regressions come from Tier 4 procedural SQL and CQL, while raw writing output appends internal planning notes even with enable_thinking:false.
• Qwen3.6-27B is the dense companion to the 35B-A3B MoE. Combined 213/285 and writing 30/30 make it stronger on quality, while 12 tg makes it roughly five times slower than the MoE sibling. Polyglot averages 30/65 with Go variance, Postgres reaches 44/57, and the raw writing output has the same planning-note leak.
• Granite-4.1-30B ties MiniMax-M2.7 for the highest non-Qwen writing score at 28/30 and lands Combined 172/285. It also hits 10/10 LRU, 59/59 LeetCode, and perfect Postgres T2, but FastAPI stays at 2/8 and generation is slow at 11.76 tg. The 8B is notable for a perfect 22/22 hallucination-calibration run, while the 3B remains a speed-first model with LRU and FastAPI failures.
• Mistral-Medium-3.5-128B lands Combined 207/285 and writes 30/30, but coding needs the model's high-effort reasoning interface. Default no-reasoning LRU scored 2/10, while reasoning_effort: high raised it to 7/10 with a different expiry-map bug. ROCm is the right backend for prompt processing, and token generation stays around 3 t/s across all backends because the 125B dense workload is memory-bandwidth-bound.
• Nemotron-3-Nano-Omni-30B-A3B-Reasoning adds multimodal training but does not improve the text-only writing score. Combined 148/285 puts it below the working field, and task-specific sampling is required. Instruct-mode params raise LRU from 0/10 to 8/10 but lower polyglot from 6/65 to 2/65, while FastAPI and Cassandra T4 still fail on endpoint design and CQL syntax.
• Sampling params transform quality results. Unsloth-recommended params (presence_penalty 1.5, top_k 20, thinking mode via chat-template-kwargs) made the difference between 0/10 and 10/10 on several models.
• T2 (cron), T5 (FastAPI), and T6 (Sinatra) remain unsolved at 0/10 across all model runs. These single-shot challenges exceed current local model capability.
• Gemma 4 leads both database benchmarks. Dense 31B tops PostgreSQL (46/57) and Cassandra (39/56). The 26B MoE follows closely. No other model family breaks 35/57 PG or 35/56 Cass.
• Mistral-Small-4-119B is the first Mistral mixture of experts on the board, 119B total with 6B active. At RADV it runs 363 pp and 40 tg, A6B-class generation for a 119B model and far quicker than the dense Mistral-Medium-3.5 at 3 tg. Combined 174/285 rests on strong pure coding (10/10 LRU, 59/59 LeetCode, 24/65 polyglot) against a mid-pack 21/30 on creative writing. Reasoning is binary, either none or high, and high is token-hungry. The first LRU run scored 0/10 because the reasoning trace filled the entire 4096-token budget before any code reached the response, so the suite runs at --max-tokens 20480. The creative-writing output ends with an assistant sign-off offering to revise the story, which the scorecard treats as a deduction.
• The Cassandra JSON extractor now parses with strict=False, so anti-pattern answers that pretty-print multi-line CQL inside a JSON string value still count. Models often write a CREATE TABLE across several lines with literal newlines, which strict JSON rejects even when the answer is correct. The change raised Mistral-Small-4 Tier 2 from 5/10 to 7/10. Cassandra scores recorded before 2026-06-05 predate it and may understate any model that emitted the same multi-line JSON.

Partial Results

Models evaluated before the full benchmark suite was established. These ran writing and LRU cache tests but not the complete battery. No longer on disk. Listed here for historical reference.

Performance

Quality