Model Comparison

Performance

Flash-Next recorded result

I completed Qwen3.8-Flash-Next IQ4_NL-PROJFIX on pwilkin's custom ROCm stack on 14 September 2026. All ten suites finished in 7 hours 37 minutes. The recorded Combined total is 222/285. I keep it outside the ranked quality table because the audit found verifier and evidence-retention limits. I have neither removed failed tests from the denominator nor awarded unverified replacement points.

Tooling is 63/65 with 100 percent usable reliability and 96.9 percent strict reliability. Calibration is 22/22. Both are supplemental. The writing samples finish in 110.6 and 116.1 seconds. Manual review finds excessive explanation and weak consistency, with the separate prose sample behind Qwen3.8-27B and Muse Glimmer. I do not treat this single-sample comparison as a model ceiling.

The target weights occupy 93.16 GiB, with a separate 2.6 GiB MTP sidecar. The scored setup uses 131,072 context, FP16 K/V, one slot, batch and microbatch 16,384, MTP width 3, disabled RAM cache, and no context shifting. Technical generation uses temperature 1.0, top-p 0.95, and xhigh reasoning. Writing uses temperature 0.7, top-p 0.8, presence penalty 1.5, and thinking off. Both profiles use top-k 20, min-p 0, repetition penalty 1, unlimited generation, and a 3,600-second request timeout.

I pinned llama.cpp to d67d58836b4987fa9dbc03b3d87c95eae6ceaddf and the custom rocm-systems tree to 7dda3ac6cfe6bbe0b7f08c23a67cfa118d8641a1. The model revision is ba5b0d696d6997d82fcc55ba3f8e6128db6e0311. The performance table uses a separate reference runtime.

Fresh-prompt context sweep

Each context gets one fresh-server request at temperature zero with thinking off and 512 output tokens. TTFT measures request start to the first nonempty text event over loopback and excludes model loading. Generation rates cover decode after prefill. All input tokens were processed without prompt-cache reuse, and process swap stayed at zero. These are long-input measurements, separate from the pp512 and tg128 reference table. Q8 KV is unsupported on this pinned optimized path.

Audit limits

Polyglot scores 43/65. The log-parser and cron tasks score zero after no code was extracted. Server logs record 39,828 generated tokens for the former and 130,606 for the latter, which filled its context. The saved artifacts omit their raw responses, so I cannot distinguish every model failure from an extraction failure. A fresh draw would be a separate attempt.

PostgreSQL records 45/57. Its covering-index prompt does not prohibit CONCURRENTLY, but the verifier executes the answer inside a transaction and rejects that statement. Cassandra records 41/56. Its schema test loses three points when the verifier's metadata queries require filtering, and the five-point materialized-view task stops because that feature is disabled. A separate review answer fails the keyword matcher, with insufficient retained text to settle the semantic result. These scores include benchmark limitations and cannot support attributing every lost point to the model. Corrected scores require replaying retained submissions through repaired verifiers.

The run-owned server and database containers were removed after completion. The original results remain unchanged. My next useful comparison would exercise Flash-Next and 27B through the same coding agent, with repeated tasks and completion time alongside success rate.

September rerun against DeepSeek-V4-Flash

On 21 and 22 September I reran Flash-Next on the same weights and pinned ROCm build at 196,608 context with FP16 K/V, one slot, batch and microbatch 16,384, MTP width 3, disabled RAM cache, and no context shifting. The Sharp v22.5.0 chat template replaced the model's own. Polyglot and Tooling used the xhigh technical profile, and creative writing and prose used the nonthinking writing profile. DeepSeek-V4-Flash UD-IQ3_XXS ran on llama.cpp b11093 with Vulkan at 64K on its published card. Only Flash-Next got the Sharp template, so the comparison sets each model at its best configuration.

These are category reruns. They do not replace the recorded 222/285 above or DeepSeek's Combined 246 from its full August suite, where it scored 63 on Polyglot, 53.5 on Tooling, and 23 on creative writing. Flash-Next's creative-writing score stays close to its recorded 16. The home page covers the Vulkan speed comparison, and the methodology page describes the app-building test.

Quality

The Tooling column is new and tracks automation reliability out of 65. It is scored separately and does not count toward the Combined total. Models marked with a hyphen have not run it yet. Scores here are for each model's reference quant. The 31B row keeps its Unsloth quant numbers as the stable reference. Google's QAT Q4_0 matches it within noise on quality and pulls ahead on tooling, artifact size, and speed. Its full-suite breakdown and the complete comparison are on the quantization page.

Qwen3.8-27B also keeps its official Q8_0 row as the reference. The Unsloth UD-Q4_K_XL follow-up scored 221/255 on the six medium-reasoning technical benchmarks against 219/255 for Q8, then completed all eight comparable technical suites in 2 hours 32 minutes instead of 12 hours 58 minutes. I have not assigned Q4 a Combined score because its nonthinking writing profile has not run. The benchmark table and the controlled speed comparison are on the quantization page.

Key Findings

• I prefer Ornith-1.5-35B-A3B among the two new models. I tested the verified official 35.20 GiB Q8_0 with thinking enabled and the published sampler. It reaches 226/285 Combined with 10/10 LRU, 8/8 FastAPI, 59/59 LeetCode, and 64/65 Polyglot. Tooling adds 62.5/65 with every call machine usable, and calibration is 22/22. Database review and diagnosis are strong while procedural authoring remains weak. PostgreSQL reaches 32/57 and Cassandra reaches 31/56. RADV gives 804.19 prompt tokens per second and 49.93 generation tokens per second. The full suite takes 129.7 minutes. Several correct coding answers still run for four to twelve minutes, which makes the model more verbose than I want from a routine local assistant. Creative writing reaches 22/30. Its separate prose sample ranks second in a blind comparison against the 9B and a human reference, with better control than the smaller model but less character depth than the reference.
• Ornith-1.5-9B reaches 175/285 Combined from the verified official 8.87 GiB Q8_0. I used its precise coding sampler for technical work and its separate general sampler for writing. Bounded tasks remain competitive at 10/10 LRU, 57/59 LeetCode, 59/65 Tooling, and 22/22 calibration. Longer tasks reveal poor stopping behavior. Polyglot falls to 24/65 and takes 170.9 minutes. Several Polyglot and database prompts fill most of the 64K context without producing a usable final answer. The technical suite takes 558 minutes even though generation runs at 23.75 tokens per second. Creative writing reaches 15/30, with broad continuity and pacing problems. The prose sample ranks third in the same blind comparison.
• I started Qwen3.8-27B on release day with the official 26.63 GiB Q8_0 target and 2.95 GiB Q8_0 MTP drafter. It finishes at 242/285 Combined, second behind DeepSeek-V4-Flash at 246 and one point ahead of Gemma-4-31B QAT at 241. LRU, FastAPI, LeetCode, and Polyglot are all perfect, with the first 65/65 Polyglot score on the board. PostgreSQL reaches 42/57 and Cassandra reaches 35/56. Both database runs sweep review, optimization, and diagnosis, then lose most of their points on procedural authoring. Tooling reaches 62.5/65 at 98.5 percent usable reliability and calibration reaches 22/22. Medium reasoning gives the technical work plenty of room, though the full suite takes 14.83 hours and the creative writing request reaches the 60 minute cap without a final answer. I retained that failed mode qualification and scored the official nonthinking run instead. It finishes 2083 words in 269 seconds for 23/30. Dialogue and sensory control are strong, while over-explanation, overly symmetrical character work, and a continuity miss cost points. The separate prose run finishes 2443 words in 312 seconds with all seven beats and exact mechanics. It ranks fourth in the five-sample comparison, held back by safe phrasing, familiar imagery, and a setting-consistency error. The fixed server control gives MTP a clean comparison. Generation rises from 5.10 to 11.87 tokens per second for a 2.33x gain at 60.84 percent acceptance. Two retained rescoring artifacts document the only score corrections. The Polyglot Apache fixture had the wrong expected status count, and the calibration matcher missed a typographic apostrophe. The generated answers were unchanged.
• Muse Glimmer 30B is the first Meta release I have benched since the Llama line went quiet, and it lands at 240/285 Combined the day after release. That puts it third, behind DeepSeek-V4-Flash at 246 and Gemma-4-31B QAT at 241, a strong showing for a dense 30B against a 284B mixture of experts. It takes two board bests on the way. Polyglot reaches 64/65, one ahead of DeepSeek, and Tooling reaches 64.5/65 with every call machine usable and the prompt injection resisted. The base weights are bfloat16 at 59.6 GB with no quantization config. Meta's own GGUF repo ships 19.65 GB and 16.76 GB variants, while the tested Unsloth Q8_K_XL is 30.07 GiB and preserves finer precision. Across both database benchmarks it scores a perfect 10/10 on review and optimization and a perfect 6/6 on diagnosis, then falls to 14/31 and 13/30 on procedural authoring. It understands databases better than it writes them. Creative writing reaches 26/30 with board-leading sensory work and dialogue, then loses points for labeling emotional beats instead of dramatizing them. The prose test ranks ahead of DeepSeek-V4-Flash and Agents-A1 and leads on voice risk, but it finishes last on genre fulfillment and stops at 1512 words against a 2400 to 3400 target. Plain generation runs 7.25 tokens per second at 234 prompt processing on RADV. Meta's DFlash drafter raises generation to 13.53 tokens per second for a 1.89x gain at 59 percent draft acceptance. The uncapped Tooling run remains comparable because the old ceiling never bound on an earlier row.
• The Ruby polyglot challenge was broken for its entire history and I have now fixed it, which moved six models. Across 104 recorded runs not one model had ever scored a single point on the Sinatra webhook task, which I had been reading as a hard challenge. It was two harness bugs stacked. Sinatra 4.1 added host authorization, which answers 403 and the text "Host not permitted" to the test client's default host header before any route code runs, so every request failed no matter what the model wrote. Underneath that, the result parser only recognised minitest 4 wording, "N tests", while minitest 5 emits "N runs", so Ruby results parsed as zero even when the suite passed. I confirmed both by writing a correct reference app to the published spec and watching it score zero. The fix lives in the verifier rather than the prompt, because the task is about webhook logic and not about Sinatra deployment hardening. I then re-ran every stored Ruby submission through the corrected harness without re-querying any model, and 29 runs gained points. This was not a uniform lift. DeepSeek-V4-Flash, Gemma-4-12B and Mistral-Small-4 had been writing perfect Ruby all along, Ornith was at 9 of 10, and Agents-A1 genuinely fails on its own merits. A second challenge was wrong in the same spirit. The FastAPI rate limiter resets module state between tests, but its skip list ignored any variable whose name began with an underscore, which is the ordinary way to mark module-private state in Python. A model that called its store _store had rate limit counts leak between tests and lost four points for the naming choice alone. The identical implementation scores 5 with an underscore and 9 without. Nineteen more runs gained points once that was fixed. Between the two bugs, polyglot has been scored out of roughly 55 while presented as out of 65. Original result files are untouched and both corrections are recorded alongside them.
• DeepSeek-V4-Flash-0731 is a 284B mixture of experts with roughly 13B active, and it takes the top combined score at 246/285. I ran it on Unsloth's published card, temp 1.0, top-p 1.0, and min-p 0.0, with top-k and both penalties pinned to no-ops because llama.cpp otherwise applies its own top-k of 40 and quietly contradicts a top-p of 1.0. That is the same practice I follow elsewhere, running each model on its documented settings where the vendor publishes them. Cassandra 43/56 is a clear board best, four ahead of the previous 39, with Tier 2 perfect at 10/10 and Tier 3 perfect at 6/6. Polyglot 63/65 is also a board best by a wide margin, and chasing it is what uncovered two broken challenges described further down. LRU, FastAPI, LeetCode, and the hallucination calibration set are all perfect. Creative writing lands at 23/30. Dialogue is exceptionally strong, while the sample loses points for weak genre fulfillment and over-explained imagery. Speed is the practical problem. I measured 114.7 prompt processing and 12.4 generation on RADV, which is poor for a 13B-active model, and the cause is missing kernels rather than memory pressure. The server disables the Lightning Indexer and three fused hash-clustering ops at load because Vulkan has no implementation for them, and cutting context from 65536 to 40960 cleared most of the swap while moving prompt processing only from 111 to 114. ROCm is not available either, since my ROCm build predates the architecture. I picked the UD-IQ3_XXS quant at 97.05 GiB over every 2-bit option because the experts ship natively at FP4, so 2-bit drops below native expert precision while 3-bit stays close to it. The 2-bit build would save about 7 GiB and both fit in 128 GB regardless. The full suite took 282 minutes.
• RADV dominates prompt processing across all model families (50-280% faster than AMDVLK on pp). Token generation is typically tied between drivers since it's bandwidth-bound. AMDVLK only wins on GPT-OSS-120B pp (661 vs 596 RADV).
• Kimi-Linear-48B was the early Polyglot leader at 22/65 before the server-based runs moved the board. Muse Glimmer now leads at 64, DeepSeek-V4-Flash follows at 63, Ornith reaches 57, Agents-A1 reaches 54, and Gemma-4-31B reaches 44. Kimi remains notable for a perfect Go rule engine and 72 tokens per second at 28 GiB.
• Devstral-Small-2 punches above its weight. A 24B dense coding model hitting 29/57 Postgres and 31/56 Cassandra while scoring 10/10 LRU and 59/59 LeetCode. The 15 t/s generation speed hurts but the quality-per-parameter is impressive.
• Nemotron-3-Nano collapses without thinking mode. 0/57 Postgres, 0/56 Cassandra, 0/59 LeetCode. The Mamba-2 hybrid architecture appears to need reasoning tokens enabled to produce structured output. Fast (1106 pp, 68 tg) but unusable for coding or database tasks with no-think.
• Magistral-Small reaches 35/56 on Cassandra, behind only DeepSeek-V4-Flash at 43 and the Gemma models among everything tested, and ahead of Devstral (31) and Kimi (24). But it scores 0/10 LRU, 12/57 Postgres, and leaked its reasoning scaffold into the creative-writing test output. A specialist, not a generalist.
• GLM-4.7-Flash can't code. 0/10 LRU, 0/8 FastAPI, 16/59 LeetCode (extraction failures), 0/65 polyglot. But it handles database work (23/57 PG, 27/56 Cass) and generates at 73 t/s. A fast model with a very narrow skill set.
• Laguna S 2.1 is Poolside's 118B mixture of experts with roughly 8B active parameters. I tested the official Q4_K_M on Poolside's llama.cpp fork because upstream support has not landed. Combined 198/285 puts it between Qwen3.6-35B-A3B and the Qwen3.5 models. The Combined run uses thinking for code and database work, then suppresses it for writing. It is much better at code and diagnosis than writing. LRU, FastAPI, and LeetCode are perfect, Polyglot reaches 35/65, and the hallucination set is a perfect 22/22. Creative writing scores 21/30 and the separate prose constraint test scores 53/80. RADV is the practical backend at 263 pp and 21.63 tg. ROCm reaches 305 pp with flash attention disabled but falls to 19.08 tg. Poolside's official DFlash draft is a firm reject on Strix Halo. It accepts only 10.7 percent of drafts on the fixed prompt set and cuts generation from 18.95 to 7.24 tokens per second.
• Agents-A1 moves the single-shot board but not the agentic one. The official Q8_0 reaches 1010 pp and 53.79 tg on RADV, then scores 224/285 Combined. Its 54/65 Polyglot score held the local record until the Ruby challenge was fixed, and it is now third. Agents-A1 is one of the few models whose Sinatra code fails on its own merits rather than because of that broken challenge. Tooling reaches 59.5/65 at 96.9 percent usable reliability and 7.1 seconds per call. In multi-turn Sokoban it stops at phase 4 in both modes after 53 minutes, 354 turns, and 342 tool calls. Writing remains mid-pack at 22/30. Its strongest roles are Polyglot and tooling, while Qwen3.6-27B remains the coding-agent pick.
• Puzzle 75B-A9B has a severe Strix thread cliff. ROCm reaches 22.66 tg at ten threads and only 2.16 at thirty-two. Combined reaches 200/285 with a strong 27/30 writing sample, 34/56 Cassandra, and a perfect 22/22 calibration run. The trade is weak 47.5/65 tooling, one successful prompt injection, a custom llama.cpp port, and one observed ROCm crash after a long server run. The complete benchmark record preserves the result.
• Ornith-1.0-35B is DeepReinforce's agentic-coding RL fine-tune of Qwen3.5-35B-A3B, and it is the strongest result that base has produced on this board. Combined 236/285 puts it third overall behind DeepSeek-V4-Flash and Gemma-4-31B, clearing its own base and the other agentic tune Nex-N2-mini by a wide margin. I ran it thinking-on, its native reasoning mode, the same way I ran Nex, so that gap reads cleanly as what the reinforcement training bought. The headline is polyglot at 57/65, second on the board. It got there by solving two challenges nothing local had cracked, the cron matcher at 10/10 and the sliding-window rate limiter at 9/10, with a perfect Go rule engine alongside. Its Ruby Sinatra webhook scores 9/10, not the zero I first recorded. I originally wrote that it produced real code with real logic bugs, and that was wrong. The challenge was broken. Cassandra is the soft spot at 28/56, almost all of it in Tier 4 procedural CQL, the same syntax failures the Nex tune had. It also returned a perfect 22/22 on the hallucination calibration set, matching the base rather than beating it, and a board-topping 77/80 on the prose constraint test where it held first-person present tense with no stray colons, semicolons, or dashes. I added a --think flag to the bench harness to drive its always-on reasoning properly, and ran the suite at a 32k token budget so the long reasoning trace had room to finish before the answer.
• MiniMax-M2.5 went from 0/10 coding to 10/10 after switching to Unsloth's UD-Q3_K_XL quant and their recommended sampling params (temp 1.0, top_p 0.95, min_p 0.01, top_k 40). At 94 GiB it is the largest artifact in this benchmark set, but 185/285 Combined puts it 6th overall. Perfect 10/10 on both PostgreSQL T2 optimization and Cassandra T2 anti-pattern detection. AMDVLK is its best backend at 32 t/s (vs 22 on RADV).
• MiniMax-M2.7 is the 229B successor with an UD-IQ4_XS quant that squeezes into 101 GiB. Writing improved to 28/30 and all coding benchmarks max out (10/10 LRU, 59/59 LeetCode, 8/8 FastAPI). The model has an unusual no-think quirk. Its M2.5 predecessor required a custom Jinja template to suppress thinking. M2.7 reverses this entirely. The standard --reasoning-budget 0 flag works, but the custom template causes the model to reason inline in plain text without ever producing code. Combined 179/285 puts it just behind M2.5 at 185. PostgreSQL T2 optimization and T3 diagnosis are both perfect (9/10 and 6/6). Cassandra T4 procedural challenges benefit from leaving thinking on (7/30 vs 2/30 without). RADV is the right backend here, unlike M2.5 which preferred AMDVLK.
• Qwen3.6-35B-A3B is the hybrid-attention successor to the 3.5 MoE. Combined 205/285, writing 29/30, and polyglot 35/65 put it near the top, but the 35 polyglot score is best-of-5 rather than stable single-run behavior. Database regressions come from Tier 4 procedural SQL and CQL, while raw writing output appends internal planning notes even with enable_thinking:false.
• Qwen3.6-27B is the dense companion to the 35B-A3B MoE. Combined 213/285 and writing 30/30 make it stronger on quality, while 12 tg makes it roughly five times slower than the MoE sibling. Polyglot averages 30/65 with Go variance, Postgres reaches 44/57, and the raw writing output has the same planning-note leak.
• Granite-4.1-30B ties MiniMax-M2.7 for the highest non-Qwen writing score at 28/30 and lands Combined 172/285. It also hits 10/10 LRU, 59/59 LeetCode, and perfect Postgres T2, but FastAPI stays at 2/8 and generation is slow at 11.76 tg. The 8B is notable for a perfect 22/22 hallucination-calibration run, while the 3B remains a speed-first model with LRU and FastAPI failures.
• Mistral-Medium-3.5-128B lands Combined 207/285 and writes 30/30, but coding needs the model's high-effort reasoning interface. Default no-reasoning LRU scored 2/10, while reasoning_effort: high raised it to 7/10 with a different expiry-map bug. ROCm is the right backend for prompt processing, and token generation stays around 3 t/s across all backends because the 125B dense workload is memory-bandwidth-bound.
• Nemotron-3-Nano-Omni-30B-A3B-Reasoning adds multimodal training but does not improve the text-only writing score. Combined 148/285 puts it below the working field, and task-specific sampling is required. Instruct-mode params raise LRU from 0/10 to 8/10 but lower polyglot from 6/65 to 2/65, while FastAPI and Cassandra T4 still fail on endpoint design and CQL syntax.
• Sampling params transform quality results. Unsloth-recommended params (presence_penalty 1.5, top_k 20, thinking mode via chat-template-kwargs) made the difference between 0/10 and 10/10 on several models.
• The Sinatra webhook is solved and was never really unsolved. It read as the one impossible Polyglot challenge for 104 runs because the harness was broken, and once that was fixed several models turned out to have been writing correct Ruby all along. Muse Glimmer scores it 10/10, which is independent confirmation that the fix holds on a model that never ran against the broken version. Agents-A1 remains one of the few that fails it on its own merits.
• Gemma 4 still leads both database benchmarks. Dense 31B tops the tracked PostgreSQL and Cassandra scores, while the 26B MoE follows closely. Agents-A1 reaches 40/57 on PostgreSQL and Puzzle reaches 34/56 on Cassandra.
• Mistral-Small-4-119B is the first Mistral mixture of experts on the board, 119B total with 6B active. At RADV it runs 363 pp and 40 tg, A6B-class generation for a 119B model and far quicker than the dense Mistral-Medium-3.5 at 3 tg. Combined 184/285 rests on strong pure coding (10/10 LRU, 59/59 LeetCode, 34/65 polyglot) against a mid-pack 21/30 on creative writing. Reasoning is binary, either none or high, and high is token-hungry. The first LRU run scored 0/10 because the reasoning trace filled the entire 4096-token budget before any code reached the response, so the suite runs at --max-tokens 20480. The creative-writing output ends with an assistant sign-off offering to revise the story, which the scorecard treats as a deduction.
• Nex-N2-mini is an agentic fine-tune of Qwen3.5-35B-A3B that tracks its base closely. Combined 183/285 sits 9 points under the 192 the base records. Writing, LRU, FastAPI, and polyglot match cell for cell. Postgres climbs 6 to 38/57, but Cassandra drops 9 to 29/56 on Tier 4 CQL syntax errors, and LeetCode loses the heap-merge problem. The agentic training only really shows in the one column the base never ran, automation reliability, where it scores 57/65 at 98.5% usable across the 65 calls with a perfect 100% on the JSON-envelope tier. I left thinking on for the whole suite because the server used the model's own template, so these numbers reflect its native reasoning mode rather than a suppressed one. The disk quant is s-batman's Q8_0 at 34.4 GiB, the best documented of the community GGUFs since no Unsloth build exists.
• The Cassandra JSON extractor now parses with strict=False, so anti-pattern answers that pretty-print multi-line CQL inside a JSON string value still count. Models often write a CREATE TABLE across several lines with literal newlines, which strict JSON rejects even when the answer is correct. The change raised Mistral-Small-4 Tier 2 from 5/10 to 7/10. Cassandra scores recorded before 2026-06-05 predate it and may understate any model that emitted the same multi-line JSON.

Partial Results

Models evaluated before the full benchmark suite was established. These ran writing and LRU cache tests but not the complete battery. Listed here for historical reference.

Performance

Quality