Methodology · Edition 2026.10
How we measure
One released runtime, every model in the configuration its authors ship, the same questions for each, and the same machine. This page is the whole protocol; the data page has every file it produced.
Principles
01
Measured, not modeled
Every latency and memory figure is measured on a named machine. No adjustment factors, no costs modeled from price lists.
02
The authors’ model
Each model runs from its author’s unmodified weights and passes a parity check against the author’s own code before it is benchmarked. The number belongs to their model, not to a broken port.
03
Reproducible
Every number comes with the runtime version, the model’s manifest digest, the machine and the command that produced it. The tools need only Python’s standard library.
04
Frozen editions
An edition fixes the tasks, the protocol and the models. Scores compare within an edition; anything that changes goes in the changelog.
05
Uncertainty shown
Scores carry 95 % bootstrap intervals, and models whose intervals overlap share a rank.
06
No private pre-testing
We publish every version we measure. If Ollaya ever ships a model of its own, it runs under the same rules and is labeled as ours.
07
Authors can respond
Each model runs with its author’s recommended temperature and precision. Corrections go through public GitHub issues.
08
Contamination flagged
A model trained on a task’s train split carries a flag, and its score leaves that task out, so a result on seen data never counts as a held-out one.
The score
A model’s score is the mean, over 14 tasks with equal weight, of its chance-corrected accuracy on each task:
skill = 100 × (accuracy − chance) / (1 − chance)
where chance is the mean of 1/K over the task’s questions (K options), so 0 is what guessing gets and 100 is every answer right. A rejected request (a question with more options than the model takes, an input longer than its context) is wrong. The 14 tasks are the 13 subsets of the public benchmark and typed-decisions. Equal weight means a 450-question subset counts as much as typed-decisions’ 2,000 questions: the score rewards breadth.
A model trained on a task’s train split is scored on the other 13 tasks: its result there is shown, flagged, but does not count.
The 95 % interval is a bootstrap with 1,000 draws and a fixed seed: questions are resampled within each public subset, and typed-decisions states are resampled whole, their five questions together. The rank is one plus the number of models whose whole interval lies above this model’s.
The tasks
5,880 questions per model: three answer types (choice, noul, score), human labels and gold distributions.
| Task | Source | Type | Questions | License |
|---|---|---|---|---|
| typed-decisions | typed-decisions test split: 400 JSON states from four workflows (agent traces, customer service, invoices, security incidents), five questions each | Choice, noul, score | 2,000 | Apache-2.0 |
| vitaminc-dev | VitaminC dev | Choice, 3 | 599 | CC BY-SA 3.0 |
| massive-en-US | MASSIVE 1.1 test, en-US | Choice, 18 | 350 | CC BY 4.0 |
| massive-de-DE | MASSIVE 1.1 test, de-DE (same ids) | Choice, 18 | 350 | CC BY 4.0 |
| boolq | BoolQ validation | Noul | 300 | CC BY-SA 3.0 |
| squad2 | SQuAD 2.0 validation | Noul | 299 | CC BY-SA 4.0 |
| paws | PAWS labeled_final test | Noul | 250 | Google PAWS, free for any purpose |
| multinli | MultiNLI dev matched | Choice, 3 | 299 | Mixed per genre, permissive |
| civil_comments | Civil Comments test | Noul | 300 | CC0 1.0 |
| aegis2 | Nemotron Content Safety V2 test | Noul | 250 | CC BY 4.0 |
| helpsteer2 | HelpSteer2 validation | Score, 5 | 249 | CC BY 4.0 |
| summeval-relevance | SummEval, relevance | Score, 5 | 240 | MIT |
| summeval-consistency | SummEval, consistency | Score, 5 | 144 | MIT |
| pubmedqa | PubMedQA pqa_labeled | Choice, 3 | 250 | MIT |
The 13 public subsets are Bespoke Labs’ public decision benchmark: seeded, label-blind draws from human-labeled datasets, rebuilt byte for byte from their committed ids and checksums, and scored with Bespoke’s own runner. Each request carries one question. typed-decisions requests carry a state’s five questions together, as an application would send them; the answer is the argmax of the returned probabilities against the gold label.
Calibration
A decision model returns probabilities, and an application acts on them, so calibration matters as much as accuracy. ECE is the ten-bin expected calibration error of the top probability, pooled over every answered question of the public benchmark (the leaderboard’s column) and of typed-decisions (on each model’s page). Brier is the multiclass Brier score. Lower is better for both. Each model runs with the temperatures its authors fitted, as Ollaya ships it.
Speed and memory
One model at a time is loaded on a server started for the run. Each model gets one cold load (timed by the server), five untimed warm-up requests, then 20 timed requests on the GPU (10 on a CPU, after two warm-ups). Every request is the five-question triage preset on a different short customer message, sent over HTTP to /api/decide by a client on the same machine; the time is the client’s, end to end. The leaderboard shows the median.
Memory: on a GPU, the rise of the GPU’s used memory (sampled every 100 ms) over its idle level just before the load, at its highest while the model is loaded. On every device, the runner process’s peak resident memory. GGUF models map their weights from disk, so part of their resident memory is file cache the system can reclaim.
Accuracy is measured once, on the GPU. Parity makes it device-independent: a model gives the same decisions on every device within the tolerances of its parity check. ONNX models run in fp16 on a GPU where their authors’ weights allow it and in fp32 on a CPU; their decisions agree on 99.1–99.6 % of questions, the difference being near-ties.
Hardware
RTX 5090 desktop
- GPU
- NVIDIA GeForce RTX 5090, 32 GB
- CPU
- AMD Ryzen Threadripper 3960X (24 cores, 48 threads)
- RAM
- 64 GB
- OS
- Windows 11 Enterprise LTSC, Ubuntu 24.04 under WSL2
- Driver
- NVIDIA 596.21 (CUDA 13.2)
Flags
laya:typed-decisionswas trained on the train split of typed-decisions; its score leaves that task out.jeb:4bwas trained on the train split of typed-decisions; its score leaves that task out.jeb:9bwas trained on the train split of typed-decisions; its score leaves that task out.jeb:27bwas trained on the train split of typed-decisions; its score leaves that task out.layais a router: each request goes tolaya:enorlaya:multilingualby its language.qwen3guard:0.6banswers only its own built-in safety questions, so it has speed and memory here but no accuracy.
Limits
- Labels measure agreement with human annotators, not truth. Annotators disagree, and some subsets are small.
- All tasks are public. A model trained on a test split would look better than it is; we flag the cases we know of.
- Speed is single-client latency. Ollaya serves one request per model at a time, so it is not a throughput benchmark.
- This edition measures one GPU and one CPU. Results on other hardware follow the same ranking only roughly.
Changelog
2026.10 · October 14, 2026. First edition: 36 models, Ollaya 0.13.0, the RTX 5090 desktop (GPU and CPU).