I thought about that too, and came to the conclusion it wouldn't bother me.
If I'm face fucking his missus, and he is getting hard about my dick, so be it, as long has he don't touch me!
Gangbangs are the same, as one (or more) guys is busy with the girl, often there are others wanking away, awaiting their turn, or just watching the fun.
Thus not much difference.