.. Task
   Loic Hausammann 17th July 2018

.. _task_adding_your_own:
.. highlight:: c

Adding a Task
=============

First you will need to understand the dependencies between tasks
using the file ``dependency_graph.dot`` generated by swift at the beginning of 
any simulation and then decide where it will fit (see :ref:`task`).

For the next paragraphs, let's assume that we want to implement the existing 
task ``cooling``.

Adding it to the Task List
--------------------------
First you will need to add it to the task list situated in ``task.h`` and 
``task.c``.

In ``task.h``, you need to provide an additional entry to the enum 
``task_types`` (e.g. ``task_type_cooling``). The last entry ``task_type_count`` 
should always stay at the end as it is a counter of the number of elements.
For example::

    enum task_types {
      task_type_none = 0,
      task_type_sort,
      task_type_self,
      task_type_pair,
      task_type_sub_self,
      task_type_sub_pair,
      task_type_ghost_in,
      task_type_ghost,
      task_type_ghost_out,
      task_type_extra_ghost,
      task_type_drift_part,
      task_type_end_force,
      task_type_kick1,
      task_type_kick2,
      task_type_timestep,
      task_type_send,
      task_type_recv,
      task_type_cooling,
      task_type_count
    } __attribute__((packed));


In ``task.c``, you will find an array containing the name of each task and need
to add your own (e.g. ``cooling``).  Be careful with the order that should be the
same than in the previous list.
For example::

  /* Task type names. */
  const char *taskID_names[task_type_count] = {
    "none",          "sort",       "self",        "pair",      "sub_self",
    "sub_pair",      "ghost_in",   "ghost",     "ghost_out",
    "extra_ghost",   "drift_part", "end_force", "kick1",
    "kick2",         "timestep",   "send",        "recv",
    "cooling"};


Adding it to the Cells
----------------------

Each cell contains a list to its tasks and therefore you need to provide a link
for it.

In ``cell_<particle_type>.h``, add a pointer to a task in the structure. For 
example, cooling couples to the hydro particles, so we'll be adding our task
to ``cell_hydro.h``.
For example::

  struct cell_hydro {
    /* Lot of stuff before. */
    
    /*! Task for the cooling */
    struct task *cooling;

    /*! The second kick task */
    struct task *kick2;
    
    /* Lot of stuff after */
  }


Adding a new Timer
------------------

As SWIFT is HPC oriented, any new task need to be optimized.
It cannot be done without timing the function.

In ``timers.h``, you will find an enum that contains all the tasks.
You will need to add yours inside it.
For example::

  enum {
    timer_none = 0,
    timer_prepare,
    timer_init,
    timer_drift_part,
    timer_drift_gpart,
    timer_kick1,
    timer_kick2,
    timer_timestep,
    timer_endforce,
    timer_dosort,
    timer_doself_density,
    timer_doself_gradient,
    timer_doself_force,
    timer_dopair_density,
    timer_dopair_gradient,
    timer_dopair_force,
    timer_dosub_self_density,
    timer_dosub_self_gradient,
    timer_dosub_self_force,
    timer_dosub_pair_density,
    timer_dosub_pair_gradient,
    timer_dosub_pair_force,
    timer_doself_subset,
    timer_dopair_subset,
    timer_dopair_subset_naive,
    timer_dosub_subset,
    timer_do_ghost,
    timer_do_extra_ghost,
    timer_dorecv_part,
    timer_do_cooling,
    timer_gettask,
    timer_qget,
    timer_qsteal,
    timer_locktree,
    timer_runners,
    timer_step,
    timer_cooling,
    timer_count,
  };

As for ``task.h``,
you will need to give a name to your timer in ``timers.c``::

  const char* timers_names[timer_count] = {
    "none",
    "prepare",
    "init",
    "drift_part",
    "kick1",
    "kick2",
    "timestep",
    "endforce",
    "dosort",
    "doself_density",
    "doself_gradient",
    "doself_force",
    "dopair_density",
    "dopair_gradient",
    "dopair_force",
    "dosub_self_density",
    "dosub_self_gradient",
    "dosub_self_force",
    "dosub_pair_density",
    "dosub_pair_gradient",
    "dosub_pair_force",
    "doself_subset",
    "dopair_subset",
    "dopair_subset_naive",
    "dosub_subset",
    "do_ghost",
    "do_extra_ghost",
    "dorecv_part",
    "gettask",
    "qget",
    "qsteal",
    "locktree",
    "runners",
    "step",
    "cooling",
  };


You can now easily time
your functions by using::

  TIMER_TIC;
  /* Your complicated functions */
  if (timer) TIMER_TOC(timer_cooling);


Adding your Task to the System
------------------------------

Now the tricky part happens.
SWIFT is able to deal automatically with the conflicts between tasks, but 
unfortunately cannot understand the dependencies.

To implement your new task in the task system, you will need to modify a 
few functions in ``engine_maketasks.c``.

First, you will need to add mainly two functions: ``scheduler_addtask`` and 
``scheduler_addunlocks`` in the ``engine_make_hierarchical_tasks_*`` functions
(depending on the type of task you implement, you will need to write it to a 
different function).

In ``engine_make_hierarchical_tasks_hydro``,
we add the task through the following call::

  /* Add the cooling task. */
  c->cooling =
  scheduler_addtask(s, task_type_cooling, task_subtype_none, 0,
                    0, c, NULL);

As the ``cooling`` cannot be done before the end of the force computation
and the second kick cannot be done before the cooling::

  scheduler_addunlock(s, c->super->end_force, c->cooling);
  scheduler_addunlock(s, c->cooling, c->super->kick2);


The next step is to activate your task
in the relevant section of ``cell_unskip.c`` (things are split
by type of particles the tasks act on)::

  else if (t->type == task_type_cooling || t->type == task_type_sourceterms) {
    if (cell_is_active_hydro(t->ci, e)) scheduler_activate(s, t);
  }

Then you will need to update the estimate for the number of tasks in 
``engine_estimate_nr_tasks`` in ``engine.c`` by modifying ``n1`` or ``n2``,
and give the task an estimate of the computational cost that it will have in 
``scheduler_reweight`` in  ``scheduler.c``::

      case task_type_cooling:
        cost = wscale * count_i;
        break;

This activates your tasks once they've been created.


If your task has some computational weight, i.e. does some actual computation
on particles, you'll also need to add it to the list of task types checked for
weights in ``partition.c:partition_gather_weights(...)``::

        /* Get the cell IDs. */
        int cid = ci - cells;

        /* Different weights for different tasks. */
        if (t->type == task_type_init_grav || t->type == task_type_ghost ||
            ... long list of task types ...
            add your new task type here )

            do stuff

And the same needs to be done in the ``check_weights(...)`` function further
down in the same file ``partition.c``,  where the same list of task types
is being checked for.


Initially, the engine will need to skip the task that updates the particles.
It is the case for the cooling, therefore you will need to add it in 
``engine_skip_force_and_kick()``.
Additionally, the tasks will be marked as 'to be skipped' once they've been
executed during a time step, and then reactivated during the next time step if
they need to be executed again. This way, all the created tasks can be kept and
don't need to be recreated every time step. In order to be unskipped however, 
you need to add the unskipping manually to ``engine_do_unskip_mapper()`` in 
``engine_unskip.c``.



Finally, you also need to initialize your new variables and pointers in 
``space_rebuild_recycle_mapper`` in ``space_recycle.c``.






Implementing your Task
----------------------

The last part is situated in ``runner_main.c``.

You will need to implement a function ``runner_do_cooling``
(do not forget to time it)::

  void runner_do_cooling(struct runner *r, struct cell *c, int timer) {

    TIMER_TIC;

    /* Now you can check if something is required at this time step.
     * You may want to use a different cell_is_active function depending
     * on your task
     */
    if (!cell_is_active_hydro(c, e)) return;

    /* Recurse? */
    if (c->split) {
      for (int k = 0; k < 8; k++)
        if (c->progeny[k] != NULL) runner_do_cooling(r, c->progeny[k], 0);
    } else {
      /* Implement your cooling here */
    }

    if (timer) TIMER_TOC(timer_do_cooling);
  }



and add a call to this function in ``runner_main``
in the switch::

  case task_type_cooling:
    runner_do_cooling(r, t->ci, 1);
    break;


Adding your task to the analysis tools
--------------------------------------

To produce the task graphs, the analysis scripts need to know about
the new task. You will need to edit the python file that contains the
hardcoded data of swift: ``tools/task_plots/swift_hardcoded_data.py``. 
You will need to add the name of the new task to the lists in there. 
*The order of this list needs to be the same as the enum type in the 
task.h file!*


Finalizing your Task
--------------------

Now that you have done the easiest part, you can start debugging by implementing
a test and/or an example. Before creating your merge request with your new task,
do not forget the most funny part that consists in writing a nice and beautiful
documentation ;)
