As a punter that likes a GFE, eye contact is very important to me.
But no, I wouldn't get angry if no eye contact is made. Generally speaking, lack of eye contact often results from the other person being shy or maybe autistic etc. IMO that sort of reason is more common than for no eye contact to mean they have an aversion to you or what they're doing. So I'd just enjoy the rest of the punt, but probably not return unless everything else was exceptional.