Hi. You're right: the first fifo size number (cur[rent]) will be counting down until all outgoing messages are sent. The peak is just gossip about how long this queue has been.

Basically, all threads work in the same way. There are four states: stopped, waiting, input or output. It's almost impossible to see the first one, as it's only set while initializing or shutting down; input is also seen rarely, unless the instance posting the data is specially slow. So they are usually waiting except while in the «send storm», when they are mostly in output mode. Anyway, even in this case, an incoming message has preference over everything, and the first available thread will serve it.

#snac sets up as many threads as cores are available, but being that the thread work is basically network I/O bound (there is very few CPU work to be done), you can increase the number of threads to twice or more the number of cores and performance will be slightly improved.

Interesting photo!