Updated the cell sanitization tool to give more sensible values to the extreme h outliers.

That may help with the EAGLE 100 box. The idea is to limit the values of h in cells by creating a computing an upper bound and imposing it before the tasks get constructed.

It gets activated if InitialConditions:cleanup_h is set to 1 in the YAML file.

Ran some tests with the EAGLE_100 volume and think this helps. I managed to run in 30 nodes with an increase in the top-level cells to 40 and this fix.

Without this fix that failed.

Still not as good as EAGLE_50x2 as that ran on 24 nodes and this still fails that.

I suspect that is because the top-levels actually restricted to 30x30x30. Clearly doing a better job with that would require the ICs to be sanitized.

I'll accept if you think this sounds useful enough.

BTW, should have said... I merged in my mpi-redistribute branch into this one first.

My recollection was that the EAGLE_100 would not start as we were spending too much time in the 0th step to correct the initial values of h. So if it now runs that's a success. If you see an improvement then that sounds useful to me.

It could be that the maximal value of h derived from the ICs prevents the creation of smaller top-level cells. What happens if you start from the 0th snapshot ? Maybe the sanitized values in there will allow for a larger top-level grid to be used.

Mixed confusing results. Starting from the eagle_001.hdf5 file as the ICs we get more top-level cells (40x40x40 as requested), but still ran out of memory using 24 nodes. I'll try one more test with more top-level cells...

That did the trick. I increased the no. of top-level cells to 80x80x80 (which it used) and it ran for 5000 steps. Going to accept this.

Status changed to merged

mentioned in commit 7b553d95

Delete the branch if you don't want to keep it please.