Yes you can turn it off, but it has already saved words before I discovered the issue.
I searched online and found that if you hit the little settings button on the bottom left corner of the keyboard (on my version of android, may vary by manufacturer and version) you get a series of options, one of which is 'My Words'. This then opens a list of saved words. In this list I found user names to several sites. You can delete them, I did. There were no passwords in the list.
As for it being a punting phone, well I believe in defense in depth, if anyone finds it they are not going to find out much, I use a private browser, and I am careful to delete messages. I only use web mail, not the built-in e-mail app. It can be explained as a spare phone that I carry when traveling in case mine is stolen.
May be paranoid, but I'd rather go a bit OTT on security and privacy than have a problem. Yes state sponsored investigations could no doubt find out a lot, but the average, even tech savvy person isn't going to be able to find much out.