Unless they're escorting you down a deserted, concealed alley and looking twitchy they're probably not a threat. I always get them to lead the way and make sure there's no one lurking to either side as I pass down passageways.
It does allow them to screen possible troublesome punters, whether attitude or under the influence.
I've been met this way a few times over the years when visiting Chinese, Latina and EE SPs which did ring possible trafficking bells. None of those guys were aggressive or intimidating, similarly male security on-site have always been low-key, which I suppose only goes to prove that they're not trying to draw attention to themselves as anything sinister even if they are.
The Caucasian guys were more imposing with their bulk.
I've walked after meeting more than a few SPs guarded in such a way and never had fuss from the hired muscle.