Skip to content

Edition 2026.10 · October 14, 2026 · Ollaya 0.13.0

Every open decision model, measured on real hardware.

Accuracy, calibration, speed and memory of 67 models on the GPUs and CPUs people own. Every model runs through the same released runtime, checked against its authors’ code, and every number links to its raw data.

Models
67
Tasks
14
13 public datasets and typed-decisions
Questions per model
5,880
human and gold labels
Devices
2
RTX 5090, Threadripper 3960X

Leaderboard on the Threadripper 3960X

CPU, 24 cores. Pick the hardware closest to yours.

Highest score
pplx-decider:flash
score 73.7 · 5.53 s · 8.2 GB
Best under 1 s
d1-omni:600m
score 45.5 · 960 ms · 1.8 GB
Best in 4 GB of RAM
d1:3b
score 63.1 · 1.76 s · 3.3 GB
Fastest
coba:68m-ternary
score 1.6 · 314 ms · 370 MB
Score against latencyPareto frontier (9): nothing is both faster and higher scoringother models
Score against median latency on the Threadripper 3960X: 65 models, 9 on the Pareto frontier010203040506070802005001k2k5k10k20k50k100kMedian latency, five questions (ms, log scale)Scorejeb:27b: score 69.1, 20.7 sd3:mini: score 68.0, 9.59 sd3:flash: score 67.8, 14.8 sjeb:9b: score 66.0, 9.43 swinnow:12b: score 65.7, 11.2 sclef:flash: score 65.1, 12.5 sstrands-decider:e4b: score 64.6, 9.44 scygnet:12b: score 64.6, 20.7 sdecider:4b: score 64.5, 9.76 sgliclass-v4:9b: score 64.3, 13.0 sdecision2:lux: score 63.2, 18.7 skev:9b: score 62.6, 12.5 snimble:9b-v3: score 62.4, 50.9 sd3:nano: score 62.2, 3.93 snimble:9b: score 61.9, 62.0 sstrands-decider:2b: score 61.3, 4.46 skev:4b: score 61.2, 8.51 scredence:e4b: score 61.0, 4.04 sdecision2:nox: score 60.9, 12.1 scredence:e4b-calibrated: score 60.5, 4.10 swinnow:e4b: score 59.9, 4.10 sgliclass-v4:2b: score 59.1, 3.57 sstrands-decider:e2b: score 58.0, 5.54 sdecider:2b: score 57.5, 3.96 sgliclass:v4-edge: score 55.8, 1.85 ssnap:2b: score 55.8, 2.65 sjeeves:9b: score 54.0, 19.4 sdecision2:sol: score 50.0, 4.64 sd3:lite: score 49.9, 2.95 swinnow:e2b: score 49.4, 2.04 sjevk5:4b: score 48.6, 6.98 sdecider:0.8b: score 47.8, 3.11 sdecision2:eos: score 47.0, 3.23 sd3:edge: score 42.9, 1.25 ssura:2b: score 40.9, 1.94 skev:0.8b: score 40.8, 2.91 sdecision:eos: score 38.1, 3.51 sdecision2:kai: score 38.0, 1.80 ssura:0.8b: score 37.2, 1.65 scoba:1b: score 36.3, 1.48 scoba:400m: score 35.8, 832 msdecima:base: score 31.0, 1.21 slaya:en: score 30.1, 945 msdecima:agent: score 27.9, 1.24 sdecima:guard: score 25.8, 1.24 svon:1.1: score 23.7, 2.59 sdecima:base-clean: score 23.5, 1.20 snli:deberta-v3-large: score 21.0, 2.32 sdecima:small: score 17.2, 591 msnli:modernbert-large: score 17.1, 2.10 sgliclass:large: score 15.3, 2.13 sarbiter:4b: score 15.2, 6.11 sdecima:guard-small: score 14.8, 721 msclm:8b: score 9.3, 5.23 scoba:150m-ternary: score 4.6, 461 mscoba:400m-ternary: score 4.4, 924 mspplx-decider:flash: score 73.7, 5.53 sjeb:4b: score 63.9, 5.33 sd1:3b: score 63.1, 1.76 ssnap:2b-q4_k_m: score 54.4, 1.73 sd1-omni:600m: score 45.5, 960 mslaya:typed-decisions: score 40.3, 944 mslaya:multilingual: score 36.7, 516 mslargha:150m: score 9.1, 339 mscoba:68m-ternary: score 1.6, 314 mslargha:150md1-omni:600msnap:2b-q4_k_md1:3bjeb:4bpplx-decider:flash
1pplx-decider:flashPareto
Perplexity AI
73.782.472.60.0165.53 s8.2 GB4BApache-2.0
2jeb:27btrained on split
frontier-infra
69.179.280.2 *0.03320.7 s26.5 GB27BApache-2.0
2d3:mini
vLLM Semantic Router
68.078.873.70.0309.59 s9.2 GB4BApache-2.0
2d3:flash
vLLM Semantic Router
67.879.071.20.03014.8 s16.0 GB9BApache-2.0
2jeb:9btrained on split
frontier-infra
66.077.279.5 *0.0209.43 s9.2 GB9BApache-2.0
2winnow:12b
EldanRing
65.777.370.20.14111.2 s14.6 GB12BApache-2.0
2clef:flash
Cloudflare
65.177.270.30.02512.5 s15.8 GB9BApache-2.0
3strands-decider:e4b
Strands Agents
64.676.569.90.0399.44 s9.7 GB8.9BApache-2.0
3cygnet:12b
blockbrain-ai
64.676.868.40.10320.7 s17.2 GB12BApache-2.0
3decider:4b
Mapika
64.576.967.80.0949.76 s9.3 GB4.2BApache-2.0
3gliclass-v4:9b
Knowledgator
64.376.365.00.03413.0 s19.8 GB9.5BApache-2.0
5jeb:4bParetotrained on split
frontier-infra
63.975.980.0 *0.0485.33 s4.5 GB4BApache-2.0
5decision2:lux
vLLM Semantic Router
63.275.670.70.05618.7 s16.5 GB7.94BApache-2.0
5d1:3bPareto
LiquidAI
63.175.165.00.0631.76 s3.3 GB3BLFM Open 1.0
5kev:9b
Jared Palmer
62.675.372.20.05812.5 s15.9 GB7.9BApache-2.0
5nimble:9b-v3
Bespoke Labs
62.475.464.80.09250.9 s18.8 GB9BCC-BY-NC-4.0 + Apache-2.0
5d3:nano
vLLM Semantic Router
62.275.662.50.0623.93 s11.1 GB2BApache-2.0
5nimble:9b
Bespoke Labs
61.974.866.50.02262.0 s18.6 GB9BApache-2.0
8strands-decider:2b
Strands Agents
61.374.561.60.0394.46 s11.1 GB2BApache-2.0
8kev:4b
Jared Palmer
61.274.566.90.0628.51 s8.9 GB4.2BApache-2.0
8credence:e4b
Txoka
61.074.172.50.0544.04 s8.7 GB7.5BApache-2.0
8decision2:nox
vLLM Semantic Router
60.974.668.00.05012.1 s9.7 GB4.21BApache-2.0
12credence:e4b-calibrated
Txoka
60.573.772.40.0384.10 s8.7 GB7.5BApache-2.0
12winnow:e4b
EldanRing
59.973.472.40.0714.10 s8.1 GB7.5BApache-2.0
15gliclass-v4:2b
Knowledgator
59.173.158.90.0313.57 s12.3 GB2.4BApache-2.0
18strands-decider:e2b
Strands Agents
58.071.664.60.0655.54 s5.5 GB4.6BApache-2.0
19decider:2b
Mapika
57.572.460.80.0893.96 s10.5 GB1.9BApache-2.0
25gliclass:v4-edge
Knowledgator
55.871.450.60.0701.85 s3.8 GB440MApache-2.0
25snap:2b
logitlab
55.870.264.80.0492.65 s2.9 GB2BApache-2.0
26snap:2b-q4_k_mPareto
logitlab
54.469.165.50.0431.73 s2.7 GB2BApache-2.0
26jeeves:9btrained on split
PostHog
54.075.679.0 *0.01819.4 s16.5 GB9BApache-2.0
31decision2:sol
vLLM Semantic Router
50.067.855.50.0534.64 s11.2 GB1.88BApache-2.0
31d3:lite
vLLM Semantic Router
49.967.052.60.0772.95 s4.6 GB0.8BApache-2.0
31winnow:e2b
EldanRing
49.465.961.60.1912.04 s4.9 GB4.6BApache-2.0
31jevk5:4brejects 12 %
alibiserikbay
48.662.162.90.0406.98 s4.9 GB4BApache-2.0
32decider:0.8b
Mapika
47.866.950.50.0613.11 s6.6 GB0.75BApache-2.0
32decision2:eos
vLLM Semantic Router
47.066.146.00.0933.23 s4.6 GB0.75BApache-2.0
34d1-omni:600mPareto
Liquid AI
45.563.957.50.075960 ms1.8 GB587MLFM Open 1.0
37d3:edge
vLLM Semantic Router
42.963.242.30.0921.25 s2.3 GB0.6BApache-2.0
39sura:2b
Ollaya
40.961.057.00.0601.94 s10.5 GB2BApache-2.0
39kev:0.8b
Jared Palmer
40.861.146.00.0352.91 s5.0 GB0.76BApache-2.0
39laya:typed-decisionsParetotrained on split
Convai Innovations
40.359.476.7 *0.022944 ms3.1 GB421MApache-2.0
40decision:eos
vLLM Semantic Router
38.159.151.20.0873.51 s6.6 GB0.75BApache-2.0
40decision2:kai
vLLM Semantic Router
38.060.847.60.0461.80 s4.3 GB0.60BApache-2.0
40sura:0.8b
Ollaya
37.258.656.00.0571.65 s4.7 GB0.8BApache-2.0
40laya:multilingualPareto
Convai Innovations
36.757.934.90.155516 ms1.9 GB322MApache-2.0
41coba:1brejects 8 %
Ollaya
36.356.160.80.0901.48 s4.3 GB1.1BApache-2.0
42coba:400mrejects 8 %
Ollaya
35.856.159.10.089832 ms1.9 GB421MApache-2.0
49decima:baserejects 6 %
A. M. Madani
31.055.048.40.0521.21 s1.1 GB321MApache-2.0
49layarejects 8 %router
Convai Innovations
30.453.535.80.103––Apache-2.0
49laya:enrejects 8 %
Convai Innovations
30.153.235.80.106945 ms3.1 GB421MApache-2.0
49decima:agent
A. M. Madani
27.953.448.10.2281.24 s1.1 GB321MApache-2.0
52decima:guard
A. M. Madani
25.851.840.40.1261.24 s1.1 GB321MApache-2.0
52von:1.1
Victor Hugo Panisa
23.748.544.60.0982.59 s1.7 GB395MApache-2.0
53decima:base-cleanrejects 6 %
A. M. Madani
23.549.247.40.0671.20 s1.1 GB321MApache-2.0
54nli:deberta-v3-largerejects 6 %
Moritz Laurer
21.045.853.60.1962.32 s3.8 GB435MMIT
56decima:smallrejects 9 %
A. M. Madani
17.246.241.40.065591 ms586 MB122MApache-2.0
56nli:modernbert-largerejects 5 %
Moritz Laurer
17.142.351.50.1112.10 s2.9 GB396MApache-2.0
57gliclass:large
Knowledgator
15.343.447.70.2842.13 s2.4 GB439MApache-2.0
57arbiter:4brejects 57 %
Codekins · Zyot Lab
15.242.00.00.1376.11 s7.2 GB4.3BApache-2.0 + Gemma
57decima:guard-small
A. M. Madani
14.842.440.00.140721 ms529 MB140MApache-2.0
62clm:8b
Contrastive-LM
9.337.735.70.3725.23 s14.5 GB8.2BApache-2.0
62largha:150mParetotrained on splitrejects 9 %
Phocinae
9.139.887.9 *0.026339 ms992 MB144MApache-2.0
62coba:150m-ternarytrained on splitrejects 8 %
Ollaya
4.662.253.50.053461 ms801 MB164MApache-2.0
62coba:400m-ternarytrained on splitrejects 8 %
Ollaya
4.465.455.90.047924 ms1.9 GB421MApache-2.0
64coba:68m-ternaryParetotrained on splitrejects 8 %
Ollaya
1.660.348.30.076314 ms370 MB75MApache-2.0
–qwen3guard:0.6bbuilt-in questions
Qwen
––––1.49 s5.0 GB0.6BApache-2.0

Score: chance-corrected accuracy over 14 tasks (0 is guessing, 100 is every answer right), with its 95 % interval; models whose intervals overlap share a rank. Public: Bespoke Labs’ public benchmark, 3,880 human-labeled questions. Typed: typed-decisions, 2,000 questions. ECE: calibration error, lower is better. Latency: median of 10 warm five-question requests over HTTP. * trained on that dataset’s train split, so the score leaves it out. How we measure.