Edition 2026.10 · October 14, 2026 · Ollaya 0.13.0
Every open decision model, measured on real hardware.
Accuracy, calibration, speed and memory of 67 models on the GPUs and CPUs people own. Every model runs through the same released runtime, checked against its authors’ code, and every number links to its raw data.
- Models
- 67
- Tasks
- 14
- 13 public datasets and typed-decisions
- Questions per model
- 5,880
- human and gold labels
- Devices
- 2
- RTX 5090, Threadripper 3960X
Leaderboard on the Threadripper 3960X
CPU, 24 cores. Pick the hardware closest to yours.
| 1 | pplx-decider:flashPareto Perplexity AI | 73.7 | 82.4 | 72.6 | 0.016 | 5.53 s | 8.2 GB | 4B | Apache-2.0 |
| 2 | jeb:27btrained on split frontier-infra | 69.1 | 79.2 | 80.2 * | 0.033 | 20.7 s | 26.5 GB | 27B | Apache-2.0 |
| 2 | d3:mini vLLM Semantic Router | 68.0 | 78.8 | 73.7 | 0.030 | 9.59 s | 9.2 GB | 4B | Apache-2.0 |
| 2 | d3:flash vLLM Semantic Router | 67.8 | 79.0 | 71.2 | 0.030 | 14.8 s | 16.0 GB | 9B | Apache-2.0 |
| 2 | jeb:9btrained on split frontier-infra | 66.0 | 77.2 | 79.5 * | 0.020 | 9.43 s | 9.2 GB | 9B | Apache-2.0 |
| 2 | winnow:12b EldanRing | 65.7 | 77.3 | 70.2 | 0.141 | 11.2 s | 14.6 GB | 12B | Apache-2.0 |
| 2 | clef:flash Cloudflare | 65.1 | 77.2 | 70.3 | 0.025 | 12.5 s | 15.8 GB | 9B | Apache-2.0 |
| 3 | strands-decider:e4b Strands Agents | 64.6 | 76.5 | 69.9 | 0.039 | 9.44 s | 9.7 GB | 8.9B | Apache-2.0 |
| 3 | cygnet:12b blockbrain-ai | 64.6 | 76.8 | 68.4 | 0.103 | 20.7 s | 17.2 GB | 12B | Apache-2.0 |
| 3 | decider:4b Mapika | 64.5 | 76.9 | 67.8 | 0.094 | 9.76 s | 9.3 GB | 4.2B | Apache-2.0 |
| 3 | gliclass-v4:9b Knowledgator | 64.3 | 76.3 | 65.0 | 0.034 | 13.0 s | 19.8 GB | 9.5B | Apache-2.0 |
| 5 | jeb:4bParetotrained on split frontier-infra | 63.9 | 75.9 | 80.0 * | 0.048 | 5.33 s | 4.5 GB | 4B | Apache-2.0 |
| 5 | decision2:lux vLLM Semantic Router | 63.2 | 75.6 | 70.7 | 0.056 | 18.7 s | 16.5 GB | 7.94B | Apache-2.0 |
| 5 | d1:3bPareto LiquidAI | 63.1 | 75.1 | 65.0 | 0.063 | 1.76 s | 3.3 GB | 3B | LFM Open 1.0 |
| 5 | kev:9b Jared Palmer | 62.6 | 75.3 | 72.2 | 0.058 | 12.5 s | 15.9 GB | 7.9B | Apache-2.0 |
| 5 | nimble:9b-v3 Bespoke Labs | 62.4 | 75.4 | 64.8 | 0.092 | 50.9 s | 18.8 GB | 9B | CC-BY-NC-4.0 + Apache-2.0 |
| 5 | d3:nano vLLM Semantic Router | 62.2 | 75.6 | 62.5 | 0.062 | 3.93 s | 11.1 GB | 2B | Apache-2.0 |
| 5 | nimble:9b Bespoke Labs | 61.9 | 74.8 | 66.5 | 0.022 | 62.0 s | 18.6 GB | 9B | Apache-2.0 |
| 8 | strands-decider:2b Strands Agents | 61.3 | 74.5 | 61.6 | 0.039 | 4.46 s | 11.1 GB | 2B | Apache-2.0 |
| 8 | kev:4b Jared Palmer | 61.2 | 74.5 | 66.9 | 0.062 | 8.51 s | 8.9 GB | 4.2B | Apache-2.0 |
| 8 | credence:e4b Txoka | 61.0 | 74.1 | 72.5 | 0.054 | 4.04 s | 8.7 GB | 7.5B | Apache-2.0 |
| 8 | decision2:nox vLLM Semantic Router | 60.9 | 74.6 | 68.0 | 0.050 | 12.1 s | 9.7 GB | 4.21B | Apache-2.0 |
| 12 | credence:e4b-calibrated Txoka | 60.5 | 73.7 | 72.4 | 0.038 | 4.10 s | 8.7 GB | 7.5B | Apache-2.0 |
| 12 | winnow:e4b EldanRing | 59.9 | 73.4 | 72.4 | 0.071 | 4.10 s | 8.1 GB | 7.5B | Apache-2.0 |
| 15 | gliclass-v4:2b Knowledgator | 59.1 | 73.1 | 58.9 | 0.031 | 3.57 s | 12.3 GB | 2.4B | Apache-2.0 |
| 18 | strands-decider:e2b Strands Agents | 58.0 | 71.6 | 64.6 | 0.065 | 5.54 s | 5.5 GB | 4.6B | Apache-2.0 |
| 19 | decider:2b Mapika | 57.5 | 72.4 | 60.8 | 0.089 | 3.96 s | 10.5 GB | 1.9B | Apache-2.0 |
| 25 | gliclass:v4-edge Knowledgator | 55.8 | 71.4 | 50.6 | 0.070 | 1.85 s | 3.8 GB | 440M | Apache-2.0 |
| 25 | snap:2b logitlab | 55.8 | 70.2 | 64.8 | 0.049 | 2.65 s | 2.9 GB | 2B | Apache-2.0 |
| 26 | snap:2b-q4_k_mPareto logitlab | 54.4 | 69.1 | 65.5 | 0.043 | 1.73 s | 2.7 GB | 2B | Apache-2.0 |
| 26 | jeeves:9btrained on split PostHog | 54.0 | 75.6 | 79.0 * | 0.018 | 19.4 s | 16.5 GB | 9B | Apache-2.0 |
| 31 | decision2:sol vLLM Semantic Router | 50.0 | 67.8 | 55.5 | 0.053 | 4.64 s | 11.2 GB | 1.88B | Apache-2.0 |
| 31 | d3:lite vLLM Semantic Router | 49.9 | 67.0 | 52.6 | 0.077 | 2.95 s | 4.6 GB | 0.8B | Apache-2.0 |
| 31 | winnow:e2b EldanRing | 49.4 | 65.9 | 61.6 | 0.191 | 2.04 s | 4.9 GB | 4.6B | Apache-2.0 |
| 31 | jevk5:4brejects 12 % alibiserikbay | 48.6 | 62.1 | 62.9 | 0.040 | 6.98 s | 4.9 GB | 4B | Apache-2.0 |
| 32 | decider:0.8b Mapika | 47.8 | 66.9 | 50.5 | 0.061 | 3.11 s | 6.6 GB | 0.75B | Apache-2.0 |
| 32 | decision2:eos vLLM Semantic Router | 47.0 | 66.1 | 46.0 | 0.093 | 3.23 s | 4.6 GB | 0.75B | Apache-2.0 |
| 34 | d1-omni:600mPareto Liquid AI | 45.5 | 63.9 | 57.5 | 0.075 | 960 ms | 1.8 GB | 587M | LFM Open 1.0 |
| 37 | d3:edge vLLM Semantic Router | 42.9 | 63.2 | 42.3 | 0.092 | 1.25 s | 2.3 GB | 0.6B | Apache-2.0 |
| 39 | sura:2b Ollaya | 40.9 | 61.0 | 57.0 | 0.060 | 1.94 s | 10.5 GB | 2B | Apache-2.0 |
| 39 | kev:0.8b Jared Palmer | 40.8 | 61.1 | 46.0 | 0.035 | 2.91 s | 5.0 GB | 0.76B | Apache-2.0 |
| 39 | laya:typed-decisionsParetotrained on split Convai Innovations | 40.3 | 59.4 | 76.7 * | 0.022 | 944 ms | 3.1 GB | 421M | Apache-2.0 |
| 40 | decision:eos vLLM Semantic Router | 38.1 | 59.1 | 51.2 | 0.087 | 3.51 s | 6.6 GB | 0.75B | Apache-2.0 |
| 40 | decision2:kai vLLM Semantic Router | 38.0 | 60.8 | 47.6 | 0.046 | 1.80 s | 4.3 GB | 0.60B | Apache-2.0 |
| 40 | sura:0.8b Ollaya | 37.2 | 58.6 | 56.0 | 0.057 | 1.65 s | 4.7 GB | 0.8B | Apache-2.0 |
| 40 | laya:multilingualPareto Convai Innovations | 36.7 | 57.9 | 34.9 | 0.155 | 516 ms | 1.9 GB | 322M | Apache-2.0 |
| 41 | coba:1brejects 8 % Ollaya | 36.3 | 56.1 | 60.8 | 0.090 | 1.48 s | 4.3 GB | 1.1B | Apache-2.0 |
| 42 | coba:400mrejects 8 % Ollaya | 35.8 | 56.1 | 59.1 | 0.089 | 832 ms | 1.9 GB | 421M | Apache-2.0 |
| 49 | decima:baserejects 6 % A. M. Madani | 31.0 | 55.0 | 48.4 | 0.052 | 1.21 s | 1.1 GB | 321M | Apache-2.0 |
| 49 | layarejects 8 %router Convai Innovations | 30.4 | 53.5 | 35.8 | 0.103 | – | – | Apache-2.0 | |
| 49 | laya:enrejects 8 % Convai Innovations | 30.1 | 53.2 | 35.8 | 0.106 | 945 ms | 3.1 GB | 421M | Apache-2.0 |
| 49 | decima:agent A. M. Madani | 27.9 | 53.4 | 48.1 | 0.228 | 1.24 s | 1.1 GB | 321M | Apache-2.0 |
| 52 | decima:guard A. M. Madani | 25.8 | 51.8 | 40.4 | 0.126 | 1.24 s | 1.1 GB | 321M | Apache-2.0 |
| 52 | von:1.1 Victor Hugo Panisa | 23.7 | 48.5 | 44.6 | 0.098 | 2.59 s | 1.7 GB | 395M | Apache-2.0 |
| 53 | decima:base-cleanrejects 6 % A. M. Madani | 23.5 | 49.2 | 47.4 | 0.067 | 1.20 s | 1.1 GB | 321M | Apache-2.0 |
| 54 | nli:deberta-v3-largerejects 6 % Moritz Laurer | 21.0 | 45.8 | 53.6 | 0.196 | 2.32 s | 3.8 GB | 435M | MIT |
| 56 | decima:smallrejects 9 % A. M. Madani | 17.2 | 46.2 | 41.4 | 0.065 | 591 ms | 586 MB | 122M | Apache-2.0 |
| 56 | nli:modernbert-largerejects 5 % Moritz Laurer | 17.1 | 42.3 | 51.5 | 0.111 | 2.10 s | 2.9 GB | 396M | Apache-2.0 |
| 57 | gliclass:large Knowledgator | 15.3 | 43.4 | 47.7 | 0.284 | 2.13 s | 2.4 GB | 439M | Apache-2.0 |
| 57 | arbiter:4brejects 57 % Codekins · Zyot Lab | 15.2 | 42.0 | 0.0 | 0.137 | 6.11 s | 7.2 GB | 4.3B | Apache-2.0 + Gemma |
| 57 | decima:guard-small A. M. Madani | 14.8 | 42.4 | 40.0 | 0.140 | 721 ms | 529 MB | 140M | Apache-2.0 |
| 62 | clm:8b Contrastive-LM | 9.3 | 37.7 | 35.7 | 0.372 | 5.23 s | 14.5 GB | 8.2B | Apache-2.0 |
| 62 | largha:150mParetotrained on splitrejects 9 % Phocinae | 9.1 | 39.8 | 87.9 * | 0.026 | 339 ms | 992 MB | 144M | Apache-2.0 |
| 62 | coba:150m-ternarytrained on splitrejects 8 % Ollaya | 4.6 | 62.2 | 53.5 | 0.053 | 461 ms | 801 MB | 164M | Apache-2.0 |
| 62 | coba:400m-ternarytrained on splitrejects 8 % Ollaya | 4.4 | 65.4 | 55.9 | 0.047 | 924 ms | 1.9 GB | 421M | Apache-2.0 |
| 64 | coba:68m-ternaryParetotrained on splitrejects 8 % Ollaya | 1.6 | 60.3 | 48.3 | 0.076 | 314 ms | 370 MB | 75M | Apache-2.0 |
| – | qwen3guard:0.6bbuilt-in questions Qwen | – | – | – | – | 1.49 s | 5.0 GB | 0.6B | Apache-2.0 |
Score: chance-corrected accuracy over 14 tasks (0 is guessing, 100 is every answer right), with its 95 % interval; models whose intervals overlap share a rank. Public: Bespoke Labs’ public benchmark, 3,880 human-labeled questions. Typed: typed-decisions, 2,000 questions. ECE: calibration error, lower is better. Latency: median of 10 warm five-question requests over HTTP. * trained on that dataset’s train split, so the score leaves it out. How we measure.