I think the answer is always going to be 50/50 mixed opinions.
Will never please anyone no matter which choice you make, as naturally those who live in and around the city will prefer you to stay put. From a personal point of view, not being a city liver, I'd always ideally want to avoid heading in towards extremely busy places, as its always a mare of a time to park and get around if don't know the area well and get stuck in some one way system....so I'd always favour someone in a quiter area.
I'm sure if you do move, you'll inevitably lose some existing customers, but there'll be plenty who will prefer the new location, so will be a boon to them, and there will be plenty of newcomers who have not been able to see you in the busy city. If you think you'd be happier living out in a quieter area then it's quite likely what you really want, and I don't blame you one bit