Set the super-cell to the first cell with interactions.

The super cell is either a non-empty leaf or the first cell for which an interaction task is defined. need to add a condition for gravit tasks once these have been settled.

Separating this from the mpi_skip branch as it is not really related and should be tested separately.

I tested this with the CosmoVolume running with hydro only, produces identical results to master.

Added 1 commit:

• 4abcd8a7 - run format.sh

Two questions:

• Shouldn't the condition just be if (super == NULL && c->density != NULL) for the hydro ?
• For the gravity task (so far just the external gravity), we need to add it to every cell that has a kick. But kicks are only set for cells that do hydro. Maybe we can't create the external gravity task here.

In reply to your questions:

• Yes, but I wanted to make sure we still had super-cells even if no tasks were defined. In that case, i.e. if a cell is a non-empty leaf, it is its own super-cell.
• I think creating the gravity tasks independently of the kick tasks is a good idea, i.e. make the (external) gravity tasks at the highest-level cells and then recursively add dependencies to any kicks below them.

Added 1 commit:

• 3290b0e7 - Added a new function to create the hierarchical gravity tasks

Added 1 commit:

• 30f04f03 - Be safe

What do you think of this newer version ? I have duplicated the infrastructure to create the gravity-related tasks. That should allow to run with hydro or gravity (or both) and always have the correct tasks at the right level.

I think the solution is correct, but there is some duplication of work in there... Would it make more sense to first just generate the init/drift/kick tasks at the root level and then, in a second pass, determine the super cells and set the ghost tasks?

This would avoid some code duplication that may come back to haunt us later.

Agreed.

Don't have a better solution at the moment though. We want to have the ability to do hydro only (and hence build the kick/init/drifts in that case), gravity only (and hence also build the kick/init/drifts). Also there might be cells that have gparts without any parts. You will need to attach kick/init/drift tasks to these as well even if they won't get any density tasks attached to them.

Right, so why not just attach the kick/init/drift tasks to the root-level cells in every case? Or is there an advantage to splitting these tasks? Do they ever get too big?

That's what is in the current master.

Yes, but the current master also sets the ghost tasks at the root cell, which is probably not a good idea.

Having said that, is there any performance difference between this branch and master?

I can't test anything this week as there is a workshop here.

Although, master also sets the super cells incorrectly. Not sure which one is the problem that leads to too many send/recv.

The too many send/recv is a separate issue, that's just because we're also send/recv-ing on cells for which we're not doing any computation.

Ok, so we should answer these questions:

• Where do we set the super-cell pointers ? And how do we make this robust when running with/without hydro and with/without gravity.
• Where do we set the kick/drift/init tasks ? At what level in the tree ? And how do we make this robust when running with/without hydro and with/without gravity.
• Where do we set the ghost tasks ? And at what level in the tree ?

@nnrw56 happy with the current version or do we want to improve things now ?

I'm happy with it, but I would feel better if somebody quickly ran a big fixdt job on Cosma-4/5 just to make sure we don't have any unexpected performance regression.

Ok, good that what I needed to know.

Reassigned to @pdraper

Just to clarify, I ma happy with that changes and the way it runs. @pdraper over to you for some more checks that you would deem necessary.