xnn.common.benchmark.runner.Benchmark#

class xnn.common.benchmark.runner.Benchmark(cfg, entries=None, part=None)[source]#

Bases: object

Run a BenchmarkConfig and collect a comparison table.

Parameters:

cfg (BenchmarkConfig) – The benchmark configuration: the pre-trained models, the dataset, the metrics/targets to score, and where to write results.
entries (list of ModelEntry or None, optional) – The subset of cfg.models this run scores (see select_entries()). None (the default) scores every model and writes the final table; a subset writes a part file named after part instead, for merge_parts() to assemble later.
part (str or None, optional) – The part file’s tag; defaults to one derived from the selected labels.

Variables:

cfg (BenchmarkConfig) – The configuration passed in.
entries (list of ModelEntry) – The model entries this run scores.
device (torch.device) – The resolved compute device (cuda:LOCAL_RANK per rank when launched distributed on GPUs).
distributed (bool) – Whether this process is one rank of a distributed launch, in which case the dataset is sharded across ranks and predictions are gathered.
rank (int) – This process’s rank (0 in a single-process run).
is_main (bool) – Whether this is rank 0, the only rank that prints and writes.
rows (list of dict) – The accumulated result rows after run() (one per model).

run()[source]#

Score every selected model on the benchmark dataset and write the results.

For each model entry the architecture is built and its checkpoint weights loaded (_load_model()), then the model is scored on the benchmark dataset (_score()), appending one row to rows. A full run prints the accumulated table and writes it to every configured output format; a run restricted to a subset of the entries prints its table and writes a part file instead (see merge_parts()). Only rank 0 of a distributed run writes.

Returns:: The result rows (also stored on rows).
Return type:: list of dict