Edition 2026.10 · October 14, 2026 · Ollaya 0.13.0
Every open decision model, measured on real hardware.
Accuracy, calibration, speed and memory of 36 models on the GPUs and CPUs people own. Every model runs through the same released runtime, checked against its authors’ code, and every number links to its raw data.
- Models
- 36
- Tasks
- 14
- 13 public datasets and typed-decisions
- Questions per model
- 5,880
- human and gold labels
- Devices
- 2
- RTX 5090, Threadripper 3960X
Leaderboard on the Threadripper 3960X
CPU, 24 cores. Pick the hardware closest to yours.
| 1 | jeb:27bParetotrained on split frontier-infra | 67.8 | 78.8 | 80.3 * | 0.024 | 21.0 s | 26.6 GB | 27B | Apache-2.0 |
| 1 | winnow:12bPareto EldanRing | 65.7 | 77.3 | 70.2 | 0.141 | 11.2 s | 14.6 GB | 12B | Apache-2.0 |
| 1 | clef:flash Cloudflare | 65.1 | 77.2 | 70.3 | 0.025 | 12.5 s | 15.8 GB | 9B | Apache-2.0 |
| 1 | jeb:9bParetotrained on split frontier-infra | 64.9 | 76.6 | 78.6 * | 0.039 | 9.42 s | 9.2 GB | 9B | Apache-2.0 |
| 1 | cygnet:12b blockbrain-ai | 64.6 | 76.8 | 68.4 | 0.103 | 20.7 s | 17.2 GB | 12B | Apache-2.0 |
| 1 | jeb:4bParetotrained on split frontier-infra | 63.9 | 75.9 | 80.0 * | 0.048 | 5.33 s | 4.5 GB | 4B | Apache-2.0 |
| 2 | d1:3bPareto LiquidAI | 63.1 | 75.1 | 65.0 | 0.063 | 1.76 s | 3.3 GB | 3B | LFM Open 1.0 |
| 2 | decider:4b Mapika | 62.9 | 75.6 | 68.0 | 0.043 | 11.9 s | 9.3 GB | 4.2B | Apache-2.0 |
| 2 | kev:9b Jared Palmer | 62.6 | 75.3 | 72.2 | 0.058 | 12.5 s | 15.9 GB | 7.9B | Apache-2.0 |
| 2 | nimble:9b Bespoke Labs | 61.9 | 74.8 | 66.5 | 0.022 | 62.0 s | 18.6 GB | 9B | Apache-2.0 |
| 4 | kev:4b Jared Palmer | 61.2 | 74.5 | 66.9 | 0.062 | 8.51 s | 8.9 GB | 4.2B | Apache-2.0 |
| 4 | jeeves:9b PostHog | 61.1 | 74.4 | 68.0 | 0.049 | 20.5 s | 16.5 GB | 9B | Apache-2.0 |
| 4 | credence:e4b Txoka | 61.0 | 74.1 | 72.5 | 0.054 | 4.04 s | 8.7 GB | 7.5B | Apache-2.0 |
| 6 | credence:e4b-calibrated Txoka | 60.5 | 73.7 | 72.4 | 0.038 | 4.10 s | 8.7 GB | 7.5B | Apache-2.0 |
| 6 | winnow:e4b EldanRing | 59.9 | 73.4 | 72.4 | 0.071 | 4.10 s | 8.1 GB | 7.5B | Apache-2.0 |
| 16 | snap:2b logitlab | 55.8 | 70.2 | 64.8 | 0.049 | 2.65 s | 2.9 GB | 2B | Apache-2.0 |
| 16 | snap:2b-q4_k_mPareto logitlab | 54.4 | 69.1 | 65.5 | 0.043 | 1.73 s | 2.7 GB | 2B | Apache-2.0 |
| 16 | decider:2b Mapika | 54.3 | 70.3 | 59.1 | 0.087 | 4.54 s | 14.6 GB | 1.9B | Apache-2.0 |
| 19 | jevk5:4brejects 12 % alibiserikbay | 48.6 | 62.1 | 62.9 | 0.040 | 6.98 s | 4.9 GB | 4B | Apache-2.0 |
| 19 | decider:0.8b Mapika | 47.8 | 66.9 | 50.5 | 0.061 | 3.11 s | 6.6 GB | 0.75B | Apache-2.0 |
| 21 | kev:0.8b Jared Palmer | 40.8 | 61.1 | 46.0 | 0.035 | 2.91 s | 5.0 GB | 0.76B | Apache-2.0 |
| 21 | laya:typed-decisionsParetotrained on split Convai Innovations | 40.3 | 59.4 | 76.7 * | 0.022 | 944 ms | 3.1 GB | 421M | Apache-2.0 |
| 21 | decision:eos vLLM Semantic Router | 38.1 | 59.1 | 51.2 | 0.087 | 3.51 s | 6.6 GB | 0.75B | Apache-2.0 |
| 21 | laya:multilingualPareto Convai Innovations | 36.7 | 57.9 | 34.9 | 0.155 | 516 ms | 1.9 GB | 322M | Apache-2.0 |
| 25 | decima:baserejects 6 % A. M. Madani | 31.0 | 55.0 | 48.4 | 0.052 | 1.21 s | 1.1 GB | 321M | Apache-2.0 |
| 25 | layarejects 8 %router Convai Innovations | 30.4 | 53.5 | 35.8 | 0.103 | – | – | Apache-2.0 | |
| 25 | laya:enrejects 8 % Convai Innovations | 30.1 | 53.2 | 35.8 | 0.106 | 945 ms | 3.1 GB | 421M | Apache-2.0 |
| 25 | decima:agent A. M. Madani | 27.9 | 53.3 | 46.1 | 0.211 | 1.23 s | 1.1 GB | 321M | Apache-2.0 |
| 28 | von:1.1 Victor Hugo Panisa | 23.7 | 48.5 | 44.6 | 0.098 | 2.59 s | 1.7 GB | 395M | Apache-2.0 |
| 29 | nli:deberta-v3-largerejects 6 % Moritz Laurer | 21.0 | 45.8 | 53.6 | 0.196 | 2.32 s | 3.8 GB | 435M | MIT |
| 30 | decima:smallrejects 9 % A. M. Madani | 17.2 | 46.2 | 41.4 | 0.065 | 625 ms | 589 MB | 122M | Apache-2.0 |
| 30 | nli:modernbert-largerejects 5 % Moritz Laurer | 17.1 | 42.3 | 51.5 | 0.111 | 2.10 s | 2.9 GB | 396M | Apache-2.0 |
| 31 | gliclass:large Knowledgator | 15.3 | 43.4 | 47.7 | 0.284 | 2.13 s | 2.4 GB | 439M | Apache-2.0 |
| 31 | arbiter:4brejects 57 % Codekins · Zyot Lab | 15.2 | 42.0 | 0.0 | 0.137 | 6.11 s | 7.2 GB | 4.3B | Apache-2.0 + Gemma |
| 35 | clm:8b Contrastive-LM | 9.3 | 37.7 | 35.7 | 0.372 | 5.23 s | 14.5 GB | 8.2B | Apache-2.0 |
| – | qwen3guard:0.6bbuilt-in questions Qwen | – | – | – | – | 1.49 s | 5.0 GB | 0.6B | Apache-2.0 |
Score: chance-corrected accuracy over 14 tasks (0 is guessing, 100 is every answer right), with its 95 % interval; models whose intervals overlap share a rank. Public: Bespoke Labs’ public benchmark, 3,880 human-labeled questions. Typed: typed-decisions, 2,000 questions. ECE: calibration error, lower is better. Latency: median of 10 warm five-question requests over HTTP. * trained on that dataset’s train split, so the score leaves it out. How we measure.