Local AI benchmarks on Ray’s machines
Decode speed, prefill, time to first token, one-shot builds, MMLU-Pro and GSM8K for local models on Ray’s own machines: two DGX Sparks, one DGX Spark, one RTX 5090, two RTX 5090s and a Mac Studio M3 Ultra. Same prompts for every model, every number from a recorded run.
Leaderboard
The mean of three scores out of 1,000: builds that work, MMLU-Pro knowledge, and decode speed. Only models with all three are ranked.
| Rank | Model | Machine | Overall | Builds | MMLU-Pro | GSM8K | Decode tok/s | Prefill tok/s |
|---|---|---|---|---|---|---|---|---|
| 1 | Qwen 3.8 27B NInfer, NVFP4, official weights |
One RTX 5090 | 843 | 857 | 80.7% | 97.0% | 189.2 | 4,057 |
| 2 | DeepSeek V4.1 Flash 1017 TensorFold Zig (Bertholomus AI), EXL3 2.9 bpw community quant (LibertAIDAI) |
Two DGX Sparks | 828 | 1,000 | 87.9% | 99.0% | 78.6 | 1,919 |
| 3 | Qwen 3.8 27B SGLang, NVFP4, official weights |
One DGX Spark | 649 | 786 | 82.1% | 97.0% | 31.8 | 1,413 |
| 4 | Nemotron 3.5 Super vLLM 0.32 nightly, NVFP4, official weights |
One DGX Spark | 643 | 643 | 83.6% | 97.0% | 46.0 | 2,732 |
| 5 | Qwen 3.8 Flash Next TensorFold, MLX 4-bit of the official weights (Vontra) |
One DGX Spark | 550 | 286 | 80.7% | 99.0% | 66.7 | 2,065 |
Not ranked on this board yet
- GLM 5.3 Flash (uncensored) on two DGX Sparks: missing MMLU-Pro
- DeepSeek V4.1 Flash (September) on two DGX Sparks: missing MMLU-Pro
Decode speed, every model timed
Decode speed: the median of a prose prompt and a Snake game, in tokens per second, one request at a time. Includes every model Ray has timed, not only the full suite.
| Rank | Model | Machine | Decode tok/s | Prose tok/s | Snake tok/s |
|---|---|---|---|---|---|
| 1 | Nemotron 3.5 Lightning TensorFold 1.0.7 |
One RTX 5090 | 629.7 | 606.6 | 652.7 |
| 2 | LFM 2.5 24B A2B llama.cpp, UD-Q4_K_XL |
One RTX 5090 | 314.5 | 318.2 | 310.8 |
| 3 | Nemotron 3.5 Lightning TensorFold 1.0.7 |
Mac Studio M3 Ultra | 246.1 | 250.7 | 241.6 |
| 4 | Nemotron 3.5 Bolt 74B llama.cpp + LiLiCorr DFlash drafts |
One RTX 5090 | 219.8 | 219.8 | — |
| 5 | Qwen 3.8 Flash Next Strata, IQ2_XS |
Two RTX 5090s | 218.5 | 164.8 | 272.2 |
| 6 | Qwen 3.6 35B A3B TensorFold MLX, MTP drafts |
Mac Studio M3 Ultra | 218.1 | 161.6 | 274.6 |
| 7 | Qwen 3.8 27B NInfer, NVFP4, official weights |
One RTX 5090 | 189.2 | 169.1 | 209.2 |
| 8 | Nemotron 3.5 Lightning TensorFold 1.0.7 |
One DGX Spark | 166.9 | 161.8 | 172.0 |
| 9 | Gemma 4 26B A4B llama.cpp, UD-Q4_K_M |
One RTX 5090 | 166.4 | 164.8 | 168.1 |
| 10 | Qwen 3.8 Flash Next MLX 4.7 bpw |
Mac Studio M3 Ultra | 100.9 | 78.8 | 123.0 |
| 11 | Gemma 4 26B A4B Ollama |
Mac Studio M3 Ultra | 94.4 | 96.2 | 92.6 |
| 12 | DeepSeek V4.1 Flash (September) retired TensorFold Zig (Bertholomus AI), EXL3 2.9 bpw community quant (Mia-AiLab) |
Two DGX Sparks | 81.9 | 71.9 | 91.9 |
| 13 | DeepSeek V4.1 Flash 1017 TensorFold Zig (Bertholomus AI), EXL3 2.9 bpw community quant (LibertAIDAI) |
Two DGX Sparks | 78.6 | 60.6 | 96.5 |
| 14 | Qwen 3.6 35B A3B vLLM NVFP4, B12X |
One DGX Spark | 73.6 | 73.6 | — |
| 15 | Qwen 3.8 Flash Next llama.cpp, IQ4_XS |
Two RTX 5090s | 69.6 | 69.6 | — |
| 16 | Qwen 3.8 Flash Next TensorFold, MLX 4-bit of the official weights (Vontra) |
One DGX Spark | 66.7 | 66.8 | 66.5 |
| 17 | Qwen 3.8 Flash Next vLLM NVFP4, 47K draft vocabulary |
Two DGX Sparks | 56.8 | 46.9 | 66.8 |
| 18 | GLM 5.3 Flash (uncensored) TensorFold (Mia’s AI Lab kit 1.8), EXL3 4 bpw uncensored community build (neko-legends) |
Two DGX Sparks | 55.6 | 47.2 | 64.0 |
| 19 | DeepSeek V4 Flash 0731 vLLM, coding profile |
Two DGX Sparks | 54.5 | 39.2 | 69.8 |
| 20 | Gemma 4 26B A4B llama.cpp, UD-Q4_K_M |
One DGX Spark | 52.3 | 52.3 | — |
| 21 | Qwen 3.8 27B llama.cpp, Q4_0 + DFlash2 drafts |
Mac Studio M3 Ultra | 47.0 | 47.0 | — |
| 22 | Nemotron 3.5 Super vLLM 0.32 nightly, NVFP4, official weights |
One DGX Spark | 46.0 | 39.8 | 52.1 |
| 23 | MiMo V2.6 Flash llama.cpp, MXFP4 |
Mac Studio M3 Ultra | 40.9 | 40.9 | — |
| 24 | Qwen 3.8 27B SGLang, NVFP4, official weights |
One DGX Spark | 31.8 | 25.7 | 37.9 |
| 25 | DeepSeek V4 Flash 0731 llama.cpp, IQ4_XS |
Mac Studio M3 Ultra | 25.5 | 25.5 | — |
| 26 | GLM 5.3 Flash vLLM EXL3, official weights, E3 + adaptive DFlash |
Two DGX Sparks | 24.0 | 24.0 | — |
| 27 | Muse Spark Open 816B llama.cpp b11718, AD-IQ1_S (174.9 GB) |
Mac Studio M3 Ultra | 12.8 | 12.9 | 12.7 |
| 28 | Qwen 3.8 27B SGLang NVFP4, no draft model |
One DGX Spark | 12.4 | 12.4 | 12.4 |
Updated 2026-10-20 16:39.