[WIP] Comm tasks are special

Don't handle send/recv tasks in the regular queue. Instead, pick out the MPI_Requests and keep them in a separate list and call either MPI_Testany or MPI_Testsome on it to get completed communication tasks before hitting the regular list of tasks.

Hi Peter!

My understanding is that you tried either exactly this or something similar. If so, please disregard :)

This change has two variants, one using MPI_Testany, and the other using MPI_Testsome and buffering the results. While the former will be faster in cases where requests complete quickly, the latter will be more efficient if we have many communication tasks that pile up.

I'm guessing this will cause problems in cases where no communication tasks complete and we have to call MPI_Test[any|some] every time we want to get a task from the queue, but this can be mitigated in two ways:

• Only check the communication tasks with probability num_comm_tasks/(num_comm_tasks + num_other_tasks),
• Don't let other runners steal from this queue.

Let me know your thoughts on this!

Cheers, Pedro

marked as a Work In Progress

closed