I've never felt the slightest pang of guilt.
In my younger days I didn't punt when I had a girlfriend, sex with a new girlfriend was always way better than punting sex and I didn't stick with anyone long enough for the sex to deteriorate. In the first few years of marriage I didn't punt but eventually drifted back - it's a hobby that never leaves you ! In fact my first punt was with a street girl, gorgeous - didn't belong on the streets, and afterwards the thoughts were all of getting to repeat the performance, not guilt.
Now I have sex for an hour or so and that's all it is. If I did ever feel a closer bond with a WG then that may be a different matter.
Guilt, NO but the fear of getting caught, YES. But I still punt.