Local AI benchmarks on Ray’s machines

Decode speed, prefill, time to first token, one-shot builds, MMLU-Pro and GSM8K for local models on Ray’s own machines: two DGX Sparks, one DGX Spark, one RTX 5090, two RTX 5090s and a Mac Studio M3 Ultra. Same prompts for every model, every number from a recorded run.

Leaderboard

The mean of three scores out of 1,000: builds that work, MMLU-Pro knowledge, and decode speed. Only models with all three are ranked.

RankModelMachineOverallBuildsMMLU-ProGSM8KDecode tok/sPrefill tok/s
1Qwen 3.8 27B
NInfer, NVFP4, official weights
One RTX 5090843 85780.7%97.0% 189.24,057
2DeepSeek V4.1 Flash 1017
TensorFold Zig (Bertholomus AI), EXL3 2.9 bpw community quant (LibertAIDAI)
Two DGX Sparks828 1,00087.9%99.0% 78.61,919
3Qwen 3.8 27B
SGLang, NVFP4, official weights
One DGX Spark649 78682.1%97.0% 31.81,413
4Nemotron 3.5 Super
vLLM 0.32 nightly, NVFP4, official weights
One DGX Spark643 64383.6%97.0% 46.02,732
5Qwen 3.8 Flash Next
TensorFold, MLX 4-bit of the official weights (Vontra)
One DGX Spark550 28680.7%99.0% 66.72,065

Not ranked on this board yet

Decode speed, every model timed

Decode speed: the median of a prose prompt and a Snake game, in tokens per second, one request at a time. Includes every model Ray has timed, not only the full suite.

RankModelMachineDecode tok/sProse tok/sSnake tok/s
1Nemotron 3.5 Lightning
TensorFold 1.0.7
One RTX 5090629.7 606.6652.7
2LFM 2.5 24B A2B
llama.cpp, UD-Q4_K_XL
One RTX 5090314.5 318.2310.8
3Nemotron 3.5 Lightning
TensorFold 1.0.7
Mac Studio M3 Ultra246.1 250.7241.6
4Nemotron 3.5 Bolt 74B
llama.cpp + LiLiCorr DFlash drafts
One RTX 5090219.8 219.8—
5Qwen 3.8 Flash Next
Strata, IQ2_XS
Two RTX 5090s218.5 164.8272.2
6Qwen 3.6 35B A3B
TensorFold MLX, MTP drafts
Mac Studio M3 Ultra218.1 161.6274.6
7Qwen 3.8 27B
NInfer, NVFP4, official weights
One RTX 5090189.2 169.1209.2
8Nemotron 3.5 Lightning
TensorFold 1.0.7
One DGX Spark166.9 161.8172.0
9Gemma 4 26B A4B
llama.cpp, UD-Q4_K_M
One RTX 5090166.4 164.8168.1
10Qwen 3.8 Flash Next
MLX 4.7 bpw
Mac Studio M3 Ultra100.9 78.8123.0
11Gemma 4 26B A4B
Ollama
Mac Studio M3 Ultra94.4 96.292.6
12DeepSeek V4.1 Flash (September) retired
TensorFold Zig (Bertholomus AI), EXL3 2.9 bpw community quant (Mia-AiLab)
Two DGX Sparks81.9 71.991.9
13DeepSeek V4.1 Flash 1017
TensorFold Zig (Bertholomus AI), EXL3 2.9 bpw community quant (LibertAIDAI)
Two DGX Sparks78.6 60.696.5
14Qwen 3.6 35B A3B
vLLM NVFP4, B12X
One DGX Spark73.6 73.6—
15Qwen 3.8 Flash Next
llama.cpp, IQ4_XS
Two RTX 5090s69.6 69.6—
16Qwen 3.8 Flash Next
TensorFold, MLX 4-bit of the official weights (Vontra)
One DGX Spark66.7 66.866.5
17Qwen 3.8 Flash Next
vLLM NVFP4, 47K draft vocabulary
Two DGX Sparks56.8 46.966.8
18GLM 5.3 Flash (uncensored)
TensorFold (Mia’s AI Lab kit 1.8), EXL3 4 bpw uncensored community build (neko-legends)
Two DGX Sparks55.6 47.264.0
19DeepSeek V4 Flash 0731
vLLM, coding profile
Two DGX Sparks54.5 39.269.8
20Gemma 4 26B A4B
llama.cpp, UD-Q4_K_M
One DGX Spark52.3 52.3—
21Qwen 3.8 27B
llama.cpp, Q4_0 + DFlash2 drafts
Mac Studio M3 Ultra47.0 47.0—
22Nemotron 3.5 Super
vLLM 0.32 nightly, NVFP4, official weights
One DGX Spark46.0 39.852.1
23MiMo V2.6 Flash
llama.cpp, MXFP4
Mac Studio M3 Ultra40.9 40.9—
24Qwen 3.8 27B
SGLang, NVFP4, official weights
One DGX Spark31.8 25.737.9
25DeepSeek V4 Flash 0731
llama.cpp, IQ4_XS
Mac Studio M3 Ultra25.5 25.5—
26GLM 5.3 Flash
vLLM EXL3, official weights, E3 + adaptive DFlash
Two DGX Sparks24.0 24.0—
27Muse Spark Open 816B
llama.cpp b11718, AD-IQ1_S (174.9 GB)
Mac Studio M3 Ultra12.8 12.912.7
28Qwen 3.8 27B
SGLang NVFP4, no draft model
One DGX Spark12.4 12.412.4

Updated 2026-10-20 16:39.