I see the OP is banned but fact is, nobody can ever know conclusively why you are coming in and out of someone else's house - WGs could also have repair men, meter readers, electricians, all sorts of shit going on. I wonder if the angry neighbour has ever berated them as well?