I do not like it that they store my number. This is also one of the main reasons I primarily use agencies as I do not like the whole idea of my number being stored by WGs.
If you ask me to explain why I don't like it, it really is just a case of it makes me feel uneasy. It makes me feel uneasy that a WG can contact me on my phone for any reason at any time, and I wont know who it is as I don't store on my phone every WG I have seen. But also because, as fl0atab1lity said, I get concerned that my number is solely stored as they do not really think much of me.
There are also issues surrounding this. There have been times (it has happened to me once), when WGs have scrolled down their long contact lists and phoned the wrong person due to either having two people with the same name in their contact lists. Or just because they have club fingers and selected the wrong person accidentally.
This means you can at any random time get a phone call from a WG discussing a booking on your phone when you have not even made one. So blowing discretion. Which I do not like.
Another thing, I remember the few topics that have been posted here over the years where a WG was leaving the business and she had on Adultwork offered to sell her working phone solely for the list of contacts inside of it.