xnn.common.benchmark.config#

Configuration for a multi-model benchmark.

Benchmarking scores pre-trained models on one dataset and tabulates their errors – it does not train or otherwise produce models (train with xnn train first). Its config therefore wraps only the building blocks a scoring pass needs – DataConfig for the dataset and ModelConfig for each model’s architecture – rather than inventing parallel ones. Each entry in models becomes a ModelEntry that pairs an architecture with the checkpoint whose weights are loaded into it, and ModelEntry.to_config() folds it together with the shared data section into an ordinary Config so the model is built exactly as in a normal run.

The one internal representation is BenchmarkConfig; every frontend is just a loader that produces it (mirroring config.loaders). from_dict() is the funnel, and from_yaml() reads a YAML file through it.

An xnn checkpoint stores the Config it was trained with, so an entry usually needs only its checkpoint – the architecture is read from the checkpoint. A mapping may still carry a label (the row name), an explicit architecture (for checkpoints that embed no config), or a config file:

models:
  - checkpoint: runs/mace/best.pt           # architecture read from the checkpoint
  - label: nequip
    checkpoint: runs/nequip/best.pt
  - name: schnet                            # explicit architecture (no embedded config)
    config: configs/model/schnet.yaml
    checkpoint: runs/schnet/best.pt

Upstream key spellings inside a model entry (MACE r_max …) are translated to the canonical names by the shared model-key registry, exactly as in a normal run.

Functions

from_dict(d)

Build a BenchmarkConfig from a plain nested dict.

from_yaml(path)

Load a YAML benchmark config into a BenchmarkConfig.

Classes

BenchmarkConfig([models, data, metrics, ...])

Top-level configuration for a multi-model benchmark.

ModelEntry(label, model[, checkpoint])

One model in a benchmark: its architecture plus the weights to score.

OutputConfig([dir, filename, formats])

Where and in which formats to write the benchmark results table.
Show Source