or just don't have any banking apps on your punting phone - and as you say, leave your "proper" phone at home or hidden in the car etc.
This is the right answer. I don't know how anyone can be get by with a dumb phone for a punting phone, unless you also take along your main phone, which kind of defeats the object.
I use mine for Google Maps, train tickets, emails, WhatsApp, checking Adultwork. If I left that trail of stuff on my regular phone, I may as well just announce to the Mrs on the way out "I'm off to bone a hot young Thai lass, I'll be back in time for tea".