Benchmarks
Reproducible evidence
A benchmark is only useful if someone else can repeat it. This page sets out the protocol; results are added only once they have been measured under it.
Results forthcoming
No benchmark observation has been published yet.
I prefer an empty table to an approximate one. The charts for this page are built and tested against example data in development, and stay hidden until real measurements exist.
Planned measurements
- Generation speed in tokens per second
- Time to first token
- Throughput under concurrent requests
- Peak memory and memory pressure
- Power draw and energy per task, once a meter is in place
- Network throughput between nodes
- Completion time of batch workloads
What every observation must carry
The data model rejects an observation that lacks any of these fields.
- Hardware
- Node record identifier from the inventory, hence chip, memory and core layout
- Model
- Model identifier and exact version or revision
- Quantization
- Bit width and scheme
- Runtime
- Runtime and version, such as the MLX release
- Lengths
- Prompt and output length in tokens
- Concurrency
- Number of simultaneous requests
- Method
- How the value was measured, including warm-up and what else was running
- Date
- Day of measurement
- Repetitions
- Number of runs
- Uncertainty
- Dispersion across runs
- Raw data
- Reference to the raw records
Limits of comparison
Different quantizations of the same model are different models. They are never placed on one axis without saying so.
A node that also hosts services is not a clean test bench. Measurements are taken with the machine otherwise idle, or the load is reported.
Laptops and desktops throttle differently under sustained load, so short runs flatter laptops.
Results across runtime versions are not comparable unless the version is the variable being studied.
Specifications such as core counts are not results. The inventory lists the former; only this page will ever list the latter.
Measurements already in papers
Some working papers in the research log contain their own measured results with their own methods, for example GPU against CPU break-even points for numerical kernels. They are cited there rather than restated here, because they were not produced under this protocol.