Skip to content

coba:68m-ternarytrained on splitrejects 8 %

coba · Ollaya · 75M · ONNX Runtime · Apache-2.0

Score
1.6
95 % interval -1.3–4.9
Rank
64
of 66 scored models
Public benchmark
60.3
macro accuracy, %
typed-decisions
48.3
accuracy, %

By task

Chance-corrected accuracy on each of the 14 tasks: 0 is guessing, 100 is every answer right. * trained on its train split, so not in the score.

Workflows
typed-decisions
24.2
Classification
MASSIVE en *
88.8
MASSIVE de *
27.1
Civil Comments *
85.3
Aegis 2 *
53.6
Understanding
MultiNLI *
72.9
VitaminC *
69.5
PAWS *
78.4
Knowledge
BoolQ *
39.3
SQuAD 2 *
43.8
PubMedQA *
41.2
Judging
HelpSteer 2 *
-4.4
SummEval rel.
-15.1
SummEval cons.
-4.2

Accuracy and calibration

BenchmarkAccuracyChoiceNoulScoreECEBrierRejected
Public benchmark60.3(59.0–61.7)68.680.013.70.0760.381453 / 3,880
typed-decisions48.3(46.0–50.5)42.065.040.40.0920.6207 / 400 requests

Accuracy in %, with its 95 % bootstrap interval. A rejected request (the model can’t take the question, or the input is too long for it) counts as wrong.

Speed and memory

Devicep50p90LoadVRAMRAM peakPrecision
RTX 5090 CUDA13.6 ms17.4 ms1.70 s899 MB934 MBF16
Threadripper 3960X CPU314 ms316 ms652 ms–370 MBF32

Five-question requests over HTTP, one at a time, each a different short message. VRAM: the rise of the GPU’s used memory while the model is loaded. RAM peak: the runner process’s peak resident memory.

Parity with the authors’ code

DeviceSame decisionMax logit diff.ResultRecord
M4 Pro METAL2383/2383–passresults/runs/2026-10-23-parity-macm4pro-metal-coba.json
RTX 5090 CPU2383/2383–passresults/runs/2026-10-23-parity-rtx5090-cpu-coba.json
RTX 5090 CUDA2383/2383–passresults/runs/2026-10-23-parity-rtx5090-cuda-coba.json

Reproduce

The tools are in convert/ollaya_convert and need only Python’s standard library. Run them against a local Ollaya 0.13.0 with the same model digest.

ollaya pull coba:68m-ternary

# typed-decisions; export the test split once with
# uv run python -m ollaya_convert.bench_typed export --out typed.jsonl
python -m ollaya_convert.bench_typed run --model coba:68m-ternary \
  --url http://127.0.0.1:11435 --data typed.jsonl --out typed/coba-68m-ternary

# Bespoke Labs' public benchmark; build its 13 subsets first
NIMBLE_SRC=nimble python -m ollaya_convert.bench_public run --model coba:68m-ternary \
  --url http://127.0.0.1:11435 --data public --out public/coba-68m-ternary

# speed and memory
python -m ollaya_convert.bench_latency --models coba:68m-ternary --memory \
  --machine my-machine --device cuda --out latency.json