Leaderboard

Qwen 3.8 27B

One DGX Spark

On one DGX Spark it decodes 31.8 tok/s and reads a 48,000-token codebase at 1,413 tok/s. It answers 82.1% of MMLU-Pro and gets 5 of 7 one-shot builds working. Ray tested it on 2026-10-19.

Machine
One DGX Spark: NVIDIA GB10, 128 GB unified memory
Engine
SGLang
Weights
NVFP4, official weights
Recipe
Ownlight’s tuned launcher, DFlash2 drafts
Made by
Alibaba Qwen
Size
27B dense
Tested
2026-10-19
Asked who made it
“I was developed by Alibaba Group’s Tongyi Lab.”
Overall
649
Builds
786
Knowledge
821
Speed
340

Scores out of 1,000. Overall is the mean of the other three; how each is scored.

Speed

Decode speed
31.8 tok/s
Decode, prose
25.7 tok/s
Decode, Snake
37.9 tok/s
First token
450 ms
Prefill, 48k prompt
1,413 tok/s
Tokens for the seven builds
259,165
Time for the seven builds
2h 23m

Decode speed is the median of a prose prompt and a Snake game, one request at a time. Prefill is how fast it reads a 48,000-token codebase before it answers.

Accuracy

SetCorrectScoreMinutes
MMLU-Pro, 140 questions, thinking low115 of 14082.1%11.8
GSM8K, 100 problems, thinking off97 of 10097.0%2.5
Snake by Qwen 3.8 27B
SnakeWorks
37.9 tok/s10m 17s23,400 tokens
Rocket launch by Qwen 3.8 27B
Rocket launchWorks
30.5 tok/s33m 43s61,637 tokens
The camera follows the rocket into the sky.
Heptagon balls by Qwen 3.8 27B
Heptagon ballsRuns with faults
47.9 tok/s2m 53s8,290 tokens
The heptagon spins, but the balls fall through its walls and off the screen.
Solar system by Qwen 3.8 27B
Solar systemWorks
35.4 tok/s4m 50s10,270 tokens
Pelican on a bicycle by Qwen 3.8 27B
Pelican on a bicycleWorks
49.6 tok/s3m 49s11,388 tokens
City fireworks by Qwen 3.8 27B
City fireworksWorks
29.7 tok/s6m 51s12,204 tokens
Gummy bears by Qwen 3.8 27B
Gummy bearsBroken
27.2 tok/s1h 20m131,976 tokens
Crashes on load (makeRotationFromEuler is not a function): only the side panel shows.

Run local models in Ownlight

Ownlight connects the model servers you already run, like Ollama, LM Studio, Unsloth Studio, llama-swap or Model Deck, and starts Claude Code or Codex on a local model beside your cloud agents. Local is part of Free.

Get OwnlightDownload for MacHow Local works

Every step

Decode speed, time to first token, prefill speed, output tokens and total time for each prompt in the suite.

TestDecode tok/sFirst tokenPrefill tok/sOutput tokensTotal timeRun
Prose (speed)25.7450 ms —95637.7 s860
Snake37.9171 ms —23,40010m 17s866
Read a 48k-token codebase30.632.5 s 1,41369555.2 s867
Who made you?57.3143 ms —47960 ms868
Rocket launch30.5164 ms —61,63733m 43s873
Heptagon balls47.9177 ms —8,2902m 53s874
Solar system35.4175 ms —10,2704m 50s875
Pelican on a bicycle49.6162 ms —11,3883m 49s876
City fireworks29.7167 ms —12,2046m 51s877
Gummy bears27.2188 ms —131,9761h 20m886

Other runs of Qwen 3.8 27B

MachineDecode tok/sProseSnakeEngine and weights
One RTX 5090 189.2169.1209.2NInfer, NVFP4, official weights
Mac Studio M3 Ultra 47.047.0—llama.cpp, Q4_0 + DFlash2 drafts
One DGX Spark 12.412.412.4SGLang NVFP4, no draft model

More models on the bench

Updated 2026-10-20 16:39.