Leaderboard

Qwen 3.8 27B

One RTX 5090

On one RTX 5090 it decodes 189.2 tok/s and reads a 48,000-token codebase at 4,057 tok/s. It answers 80.7% of MMLU-Pro and gets 6 of 7 one-shot builds working. Ray tested it on 2026-10-20.

Machine
One RTX 5090: 32 GB GDDR7, one card of Ray’s dual-5090 desktop
Engine
NInfer
Weights
NVFP4, official weights
Recipe
DFlash2 drafts (7), FP8 KV cache, 131K context
Made by
Alibaba Qwen
Size
27B dense
Tested
2026-10-20
Asked who made it
“I was created by Alibaba Group’s Tongyi Lab.”
Note
First-token times here include about 3 s that Ownlight’s gateway adds before it passes a request to the dual-5090 desktop. Ownlight’s own traces for these runs show 0.13 to 0.43 s to the first token, and 5,535 tok/s prefill on the 48k prompt.
Overall
843
Builds
857
Knowledge
807
Speed
864

Scores out of 1,000. Overall is the mean of the other three; how each is scored.

Speed

Decode speed
189.2 tok/s
Decode, prose
169.1 tok/s
Decode, Snake
209.2 tok/s
First token
3.2 s
Prefill, 48k prompt
4,057 tok/s
Tokens for the seven builds
191,508
Time for the seven builds
17m 50s

Decode speed is the median of a prose prompt and a Snake game, one request at a time. Prefill is how fast it reads a 48,000-token codebase before it answers.

Accuracy

SetCorrectScoreMinutes
MMLU-Pro, 140 questions, thinking low113 of 14080.7%6.9
GSM8K, 100 problems, thinking off97 of 10097.0%3.5
Snake by Qwen 3.8 27B
SnakeWorks
209.2 tok/s1m 21s16,466 tokens
Rocket launch by Qwen 3.8 27B
Rocket launchWorks
182.6 tok/s5m 31s59,921 tokens
Launches at dawn and drops its first stage.
Heptagon balls by Qwen 3.8 27B
Heptagon ballsWorks
268.0 tok/s26.1 s6,139 tokens
This time the balls stay inside the heptagon (the Spark run’s fell through).
Solar system by Qwen 3.8 27B
Solar systemWorks
263.3 tok/s19.0 s4,179 tokens
Pelican on a bicycle by Qwen 3.8 27B
Pelican on a bicycleWorks
284.5 tok/s42.1 s11,059 tokens
Drawn as a framed card titled Pelican’s Road Trip.
City fireworks by Qwen 3.8 27B
City fireworksWorks
256.3 tok/s24.8 s5,507 tokens
Gummy bears by Qwen 3.8 27B
Gummy bearsBroken
162.9 tok/s9m 04s88,237 tokens
Draws the table, bowl and bears, but a script error on every frame (rvx is not defined) freezes the physics, so the bears hang in the air.
Capybara onsen by Qwen 3.8 27B
Capybara onsenRuns with faults
156.3 tok/s8m 24s78,304 tokens
Every control works, but the camera sits on a giant river surface; no capybara, tub or boat in view.

Run local models in Ownlight

Ownlight connects the model servers you already run, like Ollama, LM Studio, Unsloth Studio, llama-swap or Model Deck, and starts Claude Code or Codex on a local model beside your cloud agents. Local is part of Free.

Get OwnlightDownload for MacHow Local works

Every step

Decode speed, time to first token, prefill speed, output tokens and total time for each prompt in the suite.

TestDecode tok/sFirst tokenPrefill tok/sOutput tokensTotal timeRun
Prose (speed)169.13.2 s —2,21315.9 s917
Snake209.23.3 s —16,4661m 21s919
Read a 48k-token codebase205.211.3 s 4,05771414.8 s920
Who made you?333.33.2 s —443.3 s921
Rocket launch182.63.2 s —59,9215m 31s922
Heptagon balls268.03.2 s —6,13926.1 s923
Solar system263.33.1 s —4,17919.0 s924
Pelican on a bicycle284.53.2 s —11,05942.1 s925
City fireworks256.33.3 s —5,50724.8 s926
Gummy bears162.93.4 s —88,2379m 04s927
Capybara onsen156.33.3 s —78,3048m 24s928

Other runs of Qwen 3.8 27B

MachineDecode tok/sProseSnakeEngine and weights
Mac Studio M3 Ultra 47.047.0—llama.cpp, Q4_0 + DFlash2 drafts
One DGX Spark 31.825.737.9SGLang, NVFP4, official weights
One DGX Spark 12.412.412.4SGLang NVFP4, no draft model

More models on the bench

Updated 2026-10-20 16:39.