Implement named structure in numpy and implement them for GEAR and SPHENIX

Fix #747 (closed) and #737 (closed)

SPHENIX was not working anymore because we were defining far too many masks in the hydro module. I am fixing this by merging some masks together. When we merge them, we might end up with data that are not homogeneous (e.g. float != int) and, in the current implementation, it was not really possible to deal with them. Therefore I am also adding the new function logger_loader_python_field_add_subfield that allows to define each field in a grouped mask. With this, I am constructing an hybrid dtype where each sub-element is named and can be accessed directly by its name.

The important work is done in logger/logger_python_tools.h and logger/logger_python_wrapper.c. Everything else is just redesigning the outputs.

changed milestone to %Continuous Simulation Data Stream

added enhancement i/o labels

assigned to @nnrw56 and unassigned @matthieu

I don't understand all the logic behind what you are trying to achieve here. Can you expand in text on what the issue is and why this is a good solution?

We do have a limited number of masks available (we wish to use max 2 bytes for them). Therefore if you wish to dump a lot of fields, you will need to write multiple fields into a single mask. While it is fine on the C side if you know what you are doing, it becomes complicated to deal with in python (e.g. How do you return a non homogeneous array [2 fields: one float one int]?). Here I wish to provide a way to return a structured array (https://numpy.org/doc/stable/user/basics.rec.html) to the user and thus to properly define the return type.

I would say that the structured array is the best approach as the user can know what he is dealing with directly from the names in the dtype. I think that my new function logger_loader_python_field_add_subfield is a good way to deal with this as it simplify a lot of stuff, but I agree that maybe we should find a better way to deal with the offsets (but I am not sure that we can have one).

It just worries me a lot that we have 100s upon 100s of lines of codes to define the structures which are very much duplicates of other existing things in the i/o. It all seems very fragile and very hard for anyone to actually use and expand.

I agree with you about that, we should try to find something more easily maintainable, but I do not think the issue is here but in the accumulation of previous work. Is it fine for you if we do not care about this in this merge request and I will work on it after all my current merge requests are merged?

added 1 commit

• eb08629e - Add comment

Compare with previous version

Issue written #752 (closed)

resolved all threads

merged

mentioned in commit a15c1b46