largha:150mtrained on splitrejects 9 %
Largha · Phocinae · 144M · ONNX Runtime · Apache-2.0
- Score
- 9.1
- 95 % interval 6.6–11.6
- Rank
- 40
- of 41 scored models
- Public benchmark
- 39.8
- macro accuracy, %
- typed-decisions
- 87.9
- trained on its train split
By task
Chance-corrected accuracy on each of the 14 tasks: 0 is guessing, 100 is every answer right. * trained on its train split, so not in the score.
Workflows
typed-decisions *
82.3
Classification
MASSIVE en
58.0
MASSIVE de
44.6
Civil Comments
8.7
Aegis 2
14.4
Understanding
MultiNLI
5.2
VitaminC
21.4
PAWS
-3.2
Knowledge
BoolQ
19.3
SQuAD 2
-6.3
PubMedQA
-3.8
Judging
HelpSteer 2
-10.4
SummEval rel.
-19.8
SummEval cons.
-10.2
Accuracy and calibration
| Benchmark | Accuracy | Choice | Noul | Score | ECE | Brier | Rejected |
|---|---|---|---|---|---|---|---|
| Public benchmark | 39.8(38.3–41.1) | 44.6 | 53.3 | 9.2 | 0.026 | 0.587 | 455 / 3,880 |
| typed-decisions * | 87.9(86.0–89.8) | 88.8 | 91.3 | 84.8 | 0.250 | 0.256 | 11 / 400 requests |
Accuracy in %, with its 95 % bootstrap interval. A rejected request (the model can’t take the question, or the input is too long for it) counts as wrong.
Speed and memory
| Device | p50 | p90 | Load | VRAM | RAM peak | Precision |
|---|---|---|---|---|---|---|
| RTX 5090 CUDA | 17.8 ms | 23.6 ms | 3.80 s | 1011 MB | 991 MB | F16 |
| Threadripper 3960X CPU | 339 ms | 344 ms | 2.27 s | – | 992 MB | F32 |
Five-question requests over HTTP, one at a time, each a different short message. VRAM: the rise of the GPU’s used memory while the model is loaded. RAM peak: the runner process’s peak resident memory.
Parity with the authors’ code
| Device | Same decision | Max logit diff. | Result | Record |
|---|---|---|---|---|
| M4 Pro METAL | 2383/2383 | – | pass | results/runs/2026-10-20-parity-macm4pro-metal-largha.json |
| RTX 4090 CPU | 2383/2383 | – | pass | results/runs/2026-10-20-parity-rtx4090-cpu-largha.json |
| RTX 4090 CUDA | 2383/2383 | – | pass | results/runs/2026-10-20-parity-rtx4090-cuda-largha.json |
Reproduce
The tools are in convert/ollaya_convert and need only Python’s standard library. Run them against a local Ollaya 0.13.0 with the same model digest.
ollaya pull largha:150m
# typed-decisions; export the test split once with
# uv run python -m ollaya_convert.bench_typed export --out typed.jsonl
python -m ollaya_convert.bench_typed run --model largha:150m \
--url http://127.0.0.1:11435 --data typed.jsonl --out typed/largha-150m
# Bespoke Labs' public benchmark; build its 13 subsets first
NIMBLE_SRC=nimble python -m ollaya_convert.bench_public run --model largha:150m \
--url http://127.0.0.1:11435 --data public --out public/largha-150m
# speed and memory
python -m ollaya_convert.bench_latency --models largha:150m --memory \
--machine my-machine --device cuda --out latency.json