Leaderboard

Nemotron 3.5 Super

One DGX Spark

On one DGX Spark it decodes 46.0 tok/s and reads a 48,000-token codebase at 2,732 tok/s. It answers 83.6% of MMLU-Pro and gets 3 of 7 one-shot builds working. Ray tested it on 2026-10-20.

Machine
One DGX Spark: NVIDIA GB10, 128 GB unified memory
Engine
vLLM 0.32 nightly
Weights
NVFP4, official weights
Recipe
NVIDIA’s single-Spark recipe: LiLiCorr DFlash drafts (7), 262K context, one request at a time
Made by
NVIDIA
Size
120B MoE, 12B active
Tested
2026-10-20
Asked who made it
“I am a language model developed by researchers from NVIDIA.”
Note
Run by hand with NVIDIA’s staged launcher, outside Ownlight’s catalog. The accuracy sets ran on a second copy with four request slots and a 64K context, which changes speed, not answers.
Overall
643
Builds
643
Knowledge
836
Speed
449

Scores out of 1,000. Overall is the mean of the other three; how each is scored.

Speed

Decode speed
46.0 tok/s
Decode, prose
39.8 tok/s
Decode, Snake
52.1 tok/s
First token
363 ms
Prefill, 48k prompt
2,732 tok/s
Tokens for the seven builds
82,817
Time for the seven builds
31m 25s

Decode speed is the median of a prose prompt and a Snake game, one request at a time. Prefill is how fast it reads a 48,000-token codebase before it answers.

Accuracy

SetCorrectScoreMinutes
MMLU-Pro, 140 questions, thinking low117 of 14083.6%92.6
GSM8K, 100 problems, thinking off97 of 10097.0%1.5
Snake by Nemotron 3.5 Super
SnakeWorks
52.1 tok/s1m 15s3,920 tokens
Plain, but it plays.
Rocket launch by Nemotron 3.5 Super
Rocket launchRuns with faults
43.4 tok/s4m 06s10,696 tokens
Launches and climbs, but the ground strip floats up through the sky across the rocket.
Heptagon balls by Nemotron 3.5 Super
Heptagon ballsWorks
46.8 tok/s3m 45s10,532 tokens
Solar system by Nemotron 3.5 Super
Solar systemBroken
43.7 tok/s2m 31s6,589 tokens
Black screen: a script error on every frame (reading spinSpeed) stops the planets from drawing.
Pelican on a bicycle by Nemotron 3.5 Super
Pelican on a bicycleRuns with faults
45.5 tok/s1m 55s5,254 tokens
A pelican rolls along a road, but the bicycle is a bare frame with no wheels.
City fireworks by Nemotron 3.5 Super
City fireworksWorks
42.4 tok/s5m 41s14,491 tokens
Gummy bears by Nemotron 3.5 Super
Gummy bearsRuns with faults
43.1 tok/s12m 08s31,335 tokens
Runs without errors and the bears pour out of the jar, but they’re invisible: only their shadows show on the dark table.
Capybara onsen by Nemotron 3.5 Super
Capybara onsenBroken
42.3 tok/s11m 50s29,995 tokens
A shader fails to compile (time is undeclared), so the scene is empty sky with stray slivers; only the control panel works.

Run local models in Ownlight

Ownlight connects the model servers you already run, like Ollama, LM Studio, Unsloth Studio, llama-swap or Model Deck, and starts Claude Code or Codex on a local model beside your cloud agents. Local is part of Free.

Get OwnlightDownload for MacHow Local works

Every step

Decode speed, time to first token, prefill speed, output tokens and total time for each prompt in the suite.

TestDecode tok/sFirst tokenPrefill tok/sOutput tokensTotal timeRun
Prose (speed)39.8363 ms —2,00250.7 s930
Snake52.1488 ms —3,9201m 15s932
Read a 48k-token codebase43.816.3 s 2,7321,83858.3 s933
Who made you?54.3303 ms —721.6 s934
Rocket launch43.4408 ms —10,6964m 06s935
Heptagon balls46.8431 ms —10,5323m 45s936
Solar system43.7405 ms —6,5892m 31s937
Pelican on a bicycle45.5394 ms —5,2541m 55s938
City fireworks42.4413 ms —14,4915m 41s939
Gummy bears43.1536 ms —31,33512m 08s940
Capybara onsen42.3591 ms —29,99511m 50s941

More models on the bench

Updated 2026-10-20 16:39.