xnn.common.distributed.merge_dicts#
- xnn.common.distributed.merge_dicts(d)[source]#
Merge a small picklable dict over all ranks (identity outside a distributed run).
Ranks may hold different keys, e.g. a loss term that only some ranks’ batches produced; every rank gets the union, so collectives that follow can iterate the same keys in the same order. On a key held by several ranks the highest rank’s value wins.