Bumped in to a couple on a few occasions - does really matter to me. Can be awkward if you're seeing a girl who is the only one there. In a parlour, you can pretend they've just seen someone else and not let it bother you.
I can recall years ago, I visited a place (in the days of the local paper), in Eltham (SE London for anyone who cares). Think I saw her advertising on AW but cant be sure...I arrived and it was just the maid (10am), the girl was running late. Knock at the door, assumed it was her, maid opened to find another punter. Invited him in and we had a chat and a cuppa.
Girl arrived about 1030/1040. Ready at 11am. I blew my wad and then left, guess the other guy did the same. We had a chat about preferring English girls because there were less communicative errors.