Method

Every card measures a model × host × quantization × serving config × checkpoint, not a model name. On unified-memory hardware the serving flags move throughput more than swapping the model does — disabling CUDA graphs, or capping memory utilization at 55%, changes the answer by more than the difference between two model families.

So the flags are recorded from the live server on every run and shown on the card. Two rows are only comparable if their configurations match.

Signal tasks are private, built from work we actually do. They are the measurement.

Reference tasks are public benchmarks run unmodified. They exist to calibrate the harness, not to rank models: if our number lands far from the published value, our harness is broken. Public benchmarks have been in training data for years, and a high reference score is not evidence of much.

A bench made only of private tasks would be unfalsifiable — no reader could distinguish a bad model from a broken harness. That is what the reference tier is for.

The usual reason given is confidentiality. The real reason is contamination: a published test set gets scraped, trained on, and stops measuring capability. That is how MMLU, GSM8K and HumanEval stopped being informative.

Samples carry embedded canary strings, so if a future model reproduces them we can demonstrate contamination rather than suspect it.

Each run drives real concurrent load at the serving endpoint and records time-to-first-token, inter-token latency, per-stream decode rate, and aggregate throughput at several concurrency levels.

Two details do most of the work. Every request carries a unique prefix so prefix caching cannot make prefill free and inflate the result. And output length is pinned, so concurrency levels finish at identical token counts and remain comparable. Without both, the numbers look considerably better and mean nothing.

Single-stream and aggregate figures are both reported because they disagree. On our hardware, throughput rises roughly 12× from one stream to thirty-two while per-stream decode falls to a third — batch serving and interactive use want opposite configurations.

Long reference datasets against a slow local model take hours, so some runs are capped at a sample limit. When that happens the cap is recorded in the result and displayed on the card. A capped score is not comparable to a full-dataset score and is never presented as one.

Single hardware sample per host — no variance across identical machines. Perf runs are single-shot rather than averaged over repeats, so treat small differences between runs as noise; the run-to-run spread on aggregate throughput is meaningful.

Rubric-graded samples depend on a judge model, which drifts as that model changes. Deterministic scorers are preferred wherever the property can be checked mechanically.