Fight or flight is a very strange concept, we all know what we would have liked to have done in a similar situation but until your in that situation you don't really know for sure.
I for instance would have grabbed him by his throat, smashed his phone & probably given his plums a bit of a kneeing. In reality i would have probably done the same as the OP.
What i do know for certain is that it wouldn't be the end of the matter, unfortunately i have personality that makes me "stew" over things especially when it comes to threats to myself, family or friends so i don't think that would be the last this bully/opportunist would have seen of me.
I've said it before & i will say it again "Hell hath no fury like a punter scorned"