decision2:lux
Decision 2.0 · vLLM Semantic Router · 7.94B · ONNX Runtime · Apache-2.0
- Score
- 63.2
- 95 % interval 61.3–65.2
- Rank
- 5
- of 66 scored models
- Public benchmark
- 75.6
- macro accuracy, %
- typed-decisions
- 70.7
- accuracy, %
By task
Chance-corrected accuracy on each of the 14 tasks: 0 is guessing, 100 is every answer right.
Workflows
typed-decisions
57.0
Classification
MASSIVE en
83.4
MASSIVE de
82.5
Civil Comments
40.0
Aegis 2
73.6
Understanding
MultiNLI
86.0
VitaminC
67.5
PAWS
61.6
Knowledge
BoolQ
73.3
SQuAD 2
72.6
PubMedQA
61.6
Judging
HelpSteer 2
18.2
SummEval rel.
30.7
SummEval cons.
77.4
Accuracy and calibration
| Benchmark | Accuracy | Choice | Noul | Score | ECE | Brier | Rejected |
|---|---|---|---|---|---|---|---|
| Public benchmark | 75.6(74.3–76.8) | 82.2 | 82.1 | 53.7 | 0.056 | 0.325 | 0 / 3,880 |
| typed-decisions | 70.7(68.1–72.8) | 69.8 | 80.2 | 64.1 | 0.054 | 0.409 | 0 / 400 requests |
Accuracy in %, with its 95 % bootstrap interval. A rejected request (the model can’t take the question, or the input is too long for it) counts as wrong.
Speed and memory
| Device | p50 | p90 | Load | VRAM | RAM peak | Precision |
|---|---|---|---|---|---|---|
| RTX 5090 CUDA | 364 ms | 373 ms | 54.8 s | 19.9 GB | 4.7 GB | F32 |
| Threadripper 3960X CPU | 18.7 s | 18.7 s | 10.9 s | – | 16.5 GB | F32 |
Five-question requests over HTTP, one at a time, each a different short message. VRAM: the rise of the GPU’s used memory while the model is loaded. RAM peak: the runner process’s peak resident memory.
Parity with the authors’ code
| Device | Same decision | Max logit diff. | Result | Record |
|---|---|---|---|---|
| RTX 5090 CUDA | 463/463 | 4.2e-5 | pass | results/runs/2026-10-25-parity-rtx5090-cuda-cob561.json |
Reproduce
The tools are in convert/ollaya_convert and need only Python’s standard library. Run them against a local Ollaya 0.13.0 with the same model digest.
ollaya pull decision2:lux
# typed-decisions; export the test split once with
# uv run python -m ollaya_convert.bench_typed export --out typed.jsonl
python -m ollaya_convert.bench_typed run --model decision2:lux \
--url http://127.0.0.1:11435 --data typed.jsonl --out typed/decision2-lux
# Bespoke Labs' public benchmark; build its 13 subsets first
NIMBLE_SRC=nimble python -m ollaya_convert.bench_public run --model decision2:lux \
--url http://127.0.0.1:11435 --data public --out public/decision2-lux
# speed and memory
python -m ollaya_convert.bench_latency --models decision2:lux --memory \
--machine my-machine --device cuda --out latency.json