Draft: Subtask speedup - Still requires work

For now, compile with CFLAGS=-DONLY_SUBTASKS added to the configuration line.

This introduces significantly faster neighbour finding in particle distributions with strong density gradients. For instance planetary applications or problematic feedback-disturbed galaxies.

Changes:

• Add the brute-force density checks to the planetary scheme (not just sphenix)
• Rewrite the recursion logic in the hydro and stars sub-task:
  • The interaction functions have extra parameters to optionally consider particles only between 0.5 * width and width
  • The subtask recursion now continues to lower level if we reach a level where h is too large. From that level on, we will just use the feature of only considering particles in the appropriate range of h.
  • When recursing, only the h_max of active particles is considered, not all particles.

Notes:

• We will consider c7adb289 separately.
• In a second phase, I'll remove entirely the self + pair tasks and keep only the subs.

Todo:

• Verify RT isn't broken
• Verify MPI runs are happy.
• Port changes to the other hydro schemes.
• Handle particles drifting out of their cells.

Fixes #688 (closed).

changed title from Draft: Subtask speedip to Draft: Subtask speedup

added SPH performance labels

added scaling label

added 1 commit

• 1c0075ed - Use the naive pair subset

Compare with previous version

added 1 commit

• 96ea0a4f - Undo unwanted change

Compare with previous version

added 1 commit

• ea4a0da3 - Correct function calls also in the case of the extra star sub loops

Compare with previous version

added 1 commit

• a155557d - Better documentation of the function parameters in the big switch of runner_main()

Compare with previous version

added 1 commit

• 43c59352 - Add counters also to the limiter loops to test them against the correct answer

Compare with previous version

added 1 commit

• 07c10963 - Also check the limiter loops in the exact loop checker

Compare with previous version

added 4 commits

• b5db1817 - New version of the limiter subtask task
• c42a8ac9 - Updated the limiter's SELF loop
• d36df7a2 - Updated the limiter's PAIR loop
• ff81dd9e - Also changed the NAIVE versions of the limiter's loops

Compare with previous version

added 1 commit

• f093fdba - No early abort based on flags in the sub-self and sub-pair limiters

Compare with previous version

added 1 commit

• 85f836ab - Use the sorted subset_pair when possible and the naive version only when striclty neceessary

Compare with previous version

added 1 commit

• 883def97 - Undo unwanted change

Compare with previous version

added 1 commit

• 33eda482 - Backported changes from the same branch in the COLIBRE fork.

Compare with previous version

added 153 commits

• 33eda482...207a0d2c - 106 commits from branch master
• 1bbe07db - Implemented the NAIVE hydro base functions to treat only particles in a given h-range if asked for
• f4170685 - Add new global constants to regulate the recursion inside the sub-self and sub-pair task
• 2e297a1f - First version working for hydro-only naive interactions
• 1a8c8f11 - Update the non-naive interactions to also work as expected
• ce177df9 - Clarify a bit more the logic used in the subset hydro tasks
• 7e86461e - Use the correct check to decide whether recursing is allowed in the sub-pair subset hydro functions
• a8f70b85 - Use the correct h_max in DOPAIR2. Use the correct recursion choice in DOSUB_PAIR2
• 54a481c3 - More suitable recursion condition in the sub-self2 and sub-pair2 tasks
• dded519b - Remove unnecessary checks
• 735a1273 - Activate the hydro drifts at the correct level
• 82ba2b72 - Reinstate the RT tasks
• 7709cbb8 - Implement similar changes to the stellar loops
• 21cbfb49 - Fix typo in DOSELF1_BRANCH()
• 99eedbd1 - Fix signature of DOPAIR_STARS function
• 8220cc60 - Also update the activation of stellar sub-tasks for the new scheme
• 8414d8b4 - Style formatting
• cbd6ae16 - When restarting from check-point files, read the new global constants in the...
• 9bf5541d - When a task can't be split further, still turn it into a sub-task
• 072435df - Move the DOPAIR1_SUBSET_BRANCH_STARS() at the end of the file
• 4fa9f2a3 - Revert "Move the DOPAIR1_SUBSET_BRANCH_STARS() at the end of the file"
• 5138ffc2 - Move the function DOSUB_SUBSET_STARS() at the end of the file
• 4a03d42d - Also update the function signature of the subset star functions
• 005c21da - No need to check whether cj is active in DOSUB_PAIR_SUBSET since the particles...
• 12762d50 - In the brute-force checks, ignore particles that have reached h_max when...
• efb4d386 - Fix faulty debugging check in hydro's DOSELF1_BRANCH()
• 7f0faf03 - Small fixes to the start subset recursion
• a83671bc - Post-rebase fixes
• a6f30cef - Temporarily have the stellar feedback be active every step
• 20a40d64 - Enable hydro density checks for the planetary scheme
• 8ff2ab6e - Escape checking whether N_ngb is correct when the particle's h is too close to h_max
• d9966c8a - Less confusing comments and variable names in hydro's DOSELF_SUBSET()
• afd444ab - Also stop checking whether N_ngb is correct for particles that have reached...
• fc7c8ebb - Report on startup that the code is compiled with exact stars density checks
• b413eee4 - Use the naive pair subset
• bcd3958d - Undo unwanted change
• 329b52ed - Correct function calls also in the case of the extra star sub loops
• 02ee193a - Better documentation of the function parameters in the big switch of runner_main()
• 01d41e19 - Add counters also to the limiter loops to test them against the correct answer
• 7efd71fc - Also check the limiter loops in the exact loop checker
• 8882d7c2 - New version of the limiter subtask task
• e59acabd - Updated the limiter's SELF loop
• 54159f27 - Updated the limiter's PAIR loop
• b3081c8e - Also changed the NAIVE versions of the limiter's loops
• f3bbf904 - No early abort based on flags in the sub-self and sub-pair limiters
• 34579b33 - Use the sorted subset_pair when possible and the naive version only when striclty neceessary
• e1b411e6 - Undo unwanted change
• 89cf0403 - Backported changes from the same branch in the COLIBRE fork.

Compare with previous version

@jkeger this might now be usable. I'd be interested in knowing whether it helps with your scenario as well.

Note everything that is in master is in this branch. But not the new changes to avoid the r==0 problem.

added 66 commits

• 89cf0403...4bc3c1cc - 19 commits from branch master
• 120af3c6 - Implemented the NAIVE hydro base functions to treat only particles in a given h-range if asked for
• 3ac86f8d - Add new global constants to regulate the recursion inside the sub-self and sub-pair task
• 4cae7026 - First version working for hydro-only naive interactions
• 6c3dadad - Update the non-naive interactions to also work as expected
• 6646f420 - Clarify a bit more the logic used in the subset hydro tasks
• 75213b67 - Use the correct check to decide whether recursing is allowed in the sub-pair subset hydro functions
• b65b1f06 - Use the correct h_max in DOPAIR2. Use the correct recursion choice in DOSUB_PAIR2
• ef58e342 - More suitable recursion condition in the sub-self2 and sub-pair2 tasks
• 73a87346 - Remove unnecessary checks
• 8a54477e - Activate the hydro drifts at the correct level
• 56d39a2f - Reinstate the RT tasks
• 8164ac75 - Implement similar changes to the stellar loops
• 44b89886 - Fix typo in DOSELF1_BRANCH()
• c532aff1 - Fix signature of DOPAIR_STARS function
• 60dffa72 - Also update the activation of stellar sub-tasks for the new scheme
• f7b3af25 - Style formatting
• 05ea3365 - When restarting from check-point files, read the new global constants in the...
• ac5e08b8 - When a task can't be split further, still turn it into a sub-task
• 7f11eeb3 - Move the DOPAIR1_SUBSET_BRANCH_STARS() at the end of the file
• 7e73465f - Revert "Move the DOPAIR1_SUBSET_BRANCH_STARS() at the end of the file"
• f2bd3eb9 - Move the function DOSUB_SUBSET_STARS() at the end of the file
• 931121d4 - Also update the function signature of the subset star functions
• de914349 - No need to check whether cj is active in DOSUB_PAIR_SUBSET since the particles...
• eaca48d4 - In the brute-force checks, ignore particles that have reached h_max when...
• a85c5553 - Fix faulty debugging check in hydro's DOSELF1_BRANCH()
• ffffc120 - Small fixes to the start subset recursion
• 0422b76e - Post-rebase fixes
• cd6b2b4c - Temporarily have the stellar feedback be active every step
• e1644d28 - Enable hydro density checks for the planetary scheme
• 6b1455be - Escape checking whether N_ngb is correct when the particle's h is too close to h_max
• f52278e1 - Less confusing comments and variable names in hydro's DOSELF_SUBSET()
• 4dc42310 - Also stop checking whether N_ngb is correct for particles that have reached...
• 8c9ed644 - Report on startup that the code is compiled with exact stars density checks
• ea73cbc5 - Use the naive pair subset
• adc7b05f - Undo unwanted change
• c79e7707 - Correct function calls also in the case of the extra star sub loops
• 8ba1d5e2 - Better documentation of the function parameters in the big switch of runner_main()
• 79a6183f - Add counters also to the limiter loops to test them against the correct answer
• 987e2714 - Also check the limiter loops in the exact loop checker
• 4ea7feea - New version of the limiter subtask task
• 643017f7 - Updated the limiter's SELF loop
• c83780e9 - Updated the limiter's PAIR loop
• df4fd373 - Also changed the NAIVE versions of the limiter's loops
• 022778fd - No early abort based on flags in the sub-self and sub-pair limiters
• 9814c509 - Use the sorted subset_pair when possible and the naive version only when striclty neceessary
• 4f67ba69 - Undo unwanted change
• 8bf38f79 - Backported changes from the same branch in the COLIBRE fork.

Compare with previous version

mentioned in issue #688 (closed)

added 2 commits

• 534b7dfb - In the ghost task collect the max h of the particles remaining in the to-redo list
• 147e0160 - Use the max h of the particles in the subset rather than in the cell to decide...

Compare with previous version

@jkeger I have tried a simple solution that may (or may not) help with the ghost tasks without requiring extensive surgery. Could you try the same test as above (with CFLAGS=-DONLY_SUBTASKS)?