Adding more can make the latency better (since we seem to use ibv_post_srq_recv), but get some odd effects at higher scalings, so not doing that