The honest answer is yes, Anyone that says it's not doesn't fully understand. But, it's not easy its takes someone that knows what they are doing and you can't simply just go and do it. Requires specific tools. Also depends on the phone. ( old shitty phones are harder then latest smart phones ) if it's a contract phone then there is a bigger chance of someone finding the registered address.
Long story short, if your up against a hacker, then yes, they probably can find your address through your phone number. They can even hack into your phone. ( big example, the phone hacking scandal ) Use a punting phone at all times, it will prevent a lot of issues,
im not trying to scare anyone or be a prick but thought I'd give an honest answer. I currently work in IT and work with several people who I know have the ability to do such things but locate your exact location from a phone number alone will need government style espionage.
For the average joe. They won't go through the effort and time it will take,