Heterogenous METIS partitioning.
We will need a method to load balance a job that spans heterogeneous clusters, like the new COSMA8 queue, which will have a mixture of Rome and Milan AMD cpus. Hopefully the imbalance can be assigned to a single number in this case, so we don't need to worry about task-level runtime differences and work at the level of nodes.
In that case METIS and ParMETIS have a parameter array tpwgts
, which is used to change the fraction of vertex weight that should be used for each node (partition in METIS terms). So we should assign this values based on the runtime
differences. The hard part would seem to be working out which nodes are which, probably simplest to just
use a flat file of names for that and use the MPI processor names.
One effect of this imbalance will be a change in the memory balance, with the Milan nodes probably being given more cells and particles, out of necessity.