Nothing to do with posh people in Oxford mucking around on boats.
The most convincing origin derives from a version of the popular European game of baccarat known as punto banco, in which the players, i.e., those who stake against the bank ("banco"), are known as punters, from the Italian (and Spanish) term punto, meaning "a point