doinit task called out of sync
Using the current master on my hydrostatic sphere setup - which has a large dynamic range - I find that the doinit task may be called out of sync, zeroing wcount for particles in a cell, during the density calculation (i.e. before hydro_end_density is called). With one thread I get consistent results, but when running with multiple threads, the density (and wcount values) of particles can change from run to run. This also happens when computing sph only (no external gravity). I added some debug prints that write to the screen when wcount is modified for the gas particle with id=1. You see how wcount is incremented for the particle with id=1 in the first two lines, but, just before hydro_end_density is called, wcount is zeroed again by hydro_init_part, called from runner_doinit:
[00003.2] runner_iact_nonsym_density: wcount= 49.540039 wi= 0.011394
[00003.2] runner_iact_nonsym_density: wcount= 49.543373 wi= 0.003334
[00003.2] runner_doinit: calling hydro_init_part <- this call should not be here.
[00003.2] hydro_init_part: wcount = 0.000000
[00003.2] hydro_end_density: wcount= 0.000000 kernel=0.418429 fact=25.492172 answers= 10.666667 <- so now density, wcount and smoothing length are all wrong
[00003.2] hydro_init_part: wcount = 0.000000 <- correct re-initialisation of wcount, in preparation for redoing density
This output results from ../../swift -g -s -t 8 ../../externalGravity.yml