Back in the day, I had a couple of regulars who lived within half a mile of me - around a nice leisurely 10-15 min stroll. I have to admit, I use to find this quite a turn on, and would be well up for it once I got there.
Despite living so close, I never bumped into them in public. They (or I) didn’t live on busy streets you would cut through etc and I never saw them in local shops or pubs.
If you’re worried, then I think the key is to not to give away too much about yourself, in case you have mutual acquaintances etc.