I never felt guilty. Initially I felt angry and frustrated that Mrs A, by refusing sex, drove me to become a cheating, pervy scumbag. Actually I think I went a bit overboard really. I always punted since my teens but stopped whenever I was involved in a "meaningful" relationshit. When I got married it was all great initially until we got kids. Then it all dried up and that was that. So I got lovers, loads of them. At one point I had three going at the same time - not advisable, it costs too much and takes it out on you; three women demanding attention from you (four if you count the wife) is maddening; although it's great for the ego.
Then I got into swinging big time. I really did enjoy swinging, some of the best sex I had was with filthy swingers. However, with hindsight I now realise that all that time I was secretly and foolishly wishing to be "found out" all my philandering was just my way of getting back at her. When she did find out it took me aback how upset and hurt she was. I really didn't think she would take it that badly. So I stopped fooling about with lovers and swingers, with the exception of a fuckbuddy, they just take way too much time and effort, so back into punting. It's easier to hide an hour or two during the day for a punt rather than hours and hours fooling about with lovers.
After all that I realised that apart from sex, Mrs A is a very good woman and an amazing mother for my kids and despite her being an annoying silly cow at times I do love her. Also she really is a bit of a saint for putting up with me. I can be a cantankerous old bastard and very difficult to live with, so she gets brownie points for that. But on the whole, I don't fell guilty. Things are just the way they are, I have a great relationship with Mrs A and it's only sex that's missing so I punt. No guilt.