Keeping in memory the information required for the index files during the simulation is costing far too much in term of memory but also performances. To reduce this, I have moved all the logic into the reader where we will not have as much information to track.
Some other changes that are worth to mention:
- In the CSDS yaml file, I am writing the initial number of particles. This value is used to initialize the arrays when generating the index files. The arrays can still growth if needed.
- When all the particles are written just before the first step, they are now flagged as being created.
- The data within the special flag contains now the particle. As the type was found from the index file, now we need a way to get it directly from the logfile. We still use 4 bytes but now we have 1 byte for the particle type, 2 bytes for any information (e.g. MPI rank for particles leaving/entering a rank) and 1 byte for the type of event (e.g. particle leaving/entering a rank, star formation, deletion, creation, ...).
- Initialize the time step counter to 0. There is no need to write the particles quickly at the start of the simulation as we are already manually writing them.