Skip to content

Edition 2026.10 · October 14, 2026 · Ollaya 0.13.0

Every open decision model, measured on real hardware.

Accuracy, calibration, speed and memory of 36 models on the GPUs and CPUs people own. Every model runs through the same released runtime, checked against its authors’ code, and every number links to its raw data.

Models
36
Tasks
14
13 public datasets and typed-decisions
Questions per model
5,880
human and gold labels
Devices
2
RTX 5090, Threadripper 3960X

Leaderboard on the RTX 5090

NVIDIA GPU, CUDA, 32 GB. Pick the hardware closest to yours.

Highest score
jeb:27b
score 67.8 · 253 ms · 16.2 GB
Best under 100 ms
jeb:4b
score 63.9 · 90.7 ms · 5.4 GB
Best in 8 GB of VRAM
jeb:4b
score 63.9 · 90.7 ms · 5.4 GB
Fastest
decima:small
score 17.2 · 13.1 ms · 1.2 GB
Score against latencyPareto frontier (8): nothing is both faster and higher scoringother models
Score against median latency on the RTX 5090: 34 models, 8 on the Pareto frontier0102030405060701020501002005001k2kMedian latency, five questions (ms, log scale)Scoreclef:flash: score 65.1, 406 mscygnet:12b: score 64.6, 152 msdecider:4b: score 62.9, 393 mskev:9b: score 62.6, 375 msnimble:9b: score 61.9, 1.69 skev:4b: score 61.2, 319 msjeeves:9b: score 61.1, 641 mscredence:e4b: score 61.0, 121 mscredence:e4b-calibrated: score 60.5, 114 mswinnow:e4b: score 59.9, 113 mssnap:2b: score 55.8, 82.3 mssnap:2b-q4_k_m: score 54.4, 78.2 msdecider:2b: score 54.3, 193 msjevk5:4b: score 48.6, 95.3 msdecider:0.8b: score 47.8, 189 mskev:0.8b: score 40.8, 142 msdecision:eos: score 38.1, 209 msdecima:base: score 31.0, 25.9 mslaya:en: score 30.1, 18.3 msdecima:agent: score 27.9, 27.1 msvon:1.1: score 23.7, 19.3 msnli:deberta-v3-large: score 21.0, 17.7 msnli:modernbert-large: score 17.1, 16.8 msgliclass:large: score 15.3, 14.1 msarbiter:4b: score 15.2, 135 msclm:8b: score 9.3, 94.5 msjeb:27b: score 67.8, 253 mswinnow:12b: score 65.7, 138 msjeb:9b: score 64.9, 107 msjeb:4b: score 63.9, 90.7 msd1:3b: score 63.1, 50.8 mslaya:typed-decisions: score 40.3, 20.4 mslaya:multilingual: score 36.7, 14.3 msdecima:small: score 17.2, 13.1 msdecima:smalllaya:multilinguallaya:typed-decisionsd1:3bjeb:4bjeb:9bwinnow:12b
1jeb:27bParetotrained on split
frontier-infra
67.878.880.3 *0.024253 ms16.2 GB27BApache-2.0
1winnow:12bPareto
EldanRing
65.777.370.20.141138 ms15.5 GB12BApache-2.0
1clef:flash
Cloudflare
65.177.270.30.025406 ms19.8 GB9BApache-2.0
1jeb:9bParetotrained on split
frontier-infra
64.976.678.6 *0.039107 ms9.1 GB9BApache-2.0
1cygnet:12b
blockbrain-ai
64.676.868.40.103152 ms18.1 GB12BApache-2.0
1jeb:4bParetotrained on split
frontier-infra
63.975.980.0 *0.04890.7 ms5.4 GB4BApache-2.0
2d1:3bPareto
LiquidAI
63.175.165.00.06350.8 ms4.0 GB3BLFM Open 1.0
2decider:4b
Mapika
62.975.668.00.043393 ms11.0 GB4.2BApache-2.0
2kev:9b
Jared Palmer
62.675.372.20.058375 ms18.0 GB7.9BApache-2.0
2nimble:9b
Bespoke Labs
61.974.866.50.0221.69 s20.3 GB9BApache-2.0
4kev:4b
Jared Palmer
61.274.566.90.062319 ms10.9 GB4.2BApache-2.0
4jeeves:9b
PostHog
61.174.468.00.049641 ms18.1 GB9BApache-2.0
4credence:e4b
Txoka
61.074.172.50.054121 ms6.2 GB7.5BApache-2.0
6credence:e4b-calibrated
Txoka
60.573.772.40.038114 ms6.2 GB7.5BApache-2.0
6winnow:e4b
EldanRing
59.973.472.40.071113 ms6.2 GB7.5BApache-2.0
16snap:2b
logitlab
55.870.264.80.04982.3 ms3.4 GB2BApache-2.0
16snap:2b-q4_k_m
logitlab
54.469.165.50.04378.2 ms2.5 GB2BApache-2.0
16decider:2b
Mapika
54.370.359.10.087193 ms11.9 GB1.9BApache-2.0
19jevk5:4brejects 12 %
alibiserikbay
48.662.162.90.04095.3 ms5.8 GB4BApache-2.0
19decider:0.8b
Mapika
47.866.950.50.061189 ms5.0 GB0.75BApache-2.0
21kev:0.8b
Jared Palmer
40.861.146.00.035142 ms4.5 GB0.76BApache-2.0
21laya:typed-decisionsParetotrained on split
Convai Innovations
40.359.476.7 *0.02220.4 ms1.6 GB421MApache-2.0
21decision:eos
vLLM Semantic Router
38.159.151.20.087209 ms5.8 GB0.75BApache-2.0
21laya:multilingualPareto
Convai Innovations
36.757.934.90.15514.3 ms1.4 GB322MApache-2.0
25decima:baserejects 6 %
A. M. Madani
31.055.048.40.05225.9 ms2.1 GB321MApache-2.0
25layarejects 8 %router
Convai Innovations
30.453.535.80.103––Apache-2.0
25laya:enrejects 8 %
Convai Innovations
30.153.235.80.10618.3 ms1.6 GB421MApache-2.0
25decima:agent
A. M. Madani
27.953.346.10.21127.1 ms2.7 GB321MApache-2.0
28von:1.1
Victor Hugo Panisa
23.748.544.60.09819.3 ms2.6 GB395MApache-2.0
29nli:deberta-v3-largerejects 6 %
Moritz Laurer
21.045.853.60.19617.7 ms2.6 GB435MMIT
30decima:smallParetorejects 9 %
A. M. Madani
17.246.241.40.06513.1 ms1.2 GB122MApache-2.0
30nli:modernbert-largerejects 5 %
Moritz Laurer
17.142.351.50.11116.8 ms2.6 GB396MApache-2.0
31gliclass:large
Knowledgator
15.343.447.70.28414.1 ms2.6 GB439MApache-2.0
31arbiter:4brejects 57 %
Codekins · Zyot Lab
15.242.00.00.137135 ms8.3 GB4.3BApache-2.0 + Gemma
35clm:8b
Contrastive-LM
9.337.735.70.37294.5 ms15.3 GB8.2BApache-2.0
–qwen3guard:0.6bbuilt-in questions
Qwen
––––30.2 ms3.6 GB0.6BApache-2.0

Score: chance-corrected accuracy over 14 tasks (0 is guessing, 100 is every answer right), with its 95 % interval; models whose intervals overlap share a rank. Public: Bespoke Labs’ public benchmark, 3,880 human-labeled questions. Typed: typed-decisions, 2,000 questions. ECE: calibration error, lower is better. Latency: median of 20 warm five-question requests over HTTP. * trained on that dataset’s train split, so the score leaves it out. How we measure.