The algorithms that Facebook and Google use are obviously very sophisticated.
You may have a punting/burner phone but chances are you've got your smartphone nearby which means metadata such as geolocations etc are stored. Metadata is more than just using google maps - it's the phones GPS, or its the nearby WiFi signals that you don't even connect to but are open enough to broadcast location, or its gsm triangulation, or its the hundreds of millions of lines of code which run your phone. There is no escaping it. It might be that you have a contactless card stored on your phone (apple pay/android pay) they'll not even need to request access thru the phone - the app will report where you were fucking standing.
Facebook (both app and allowing cookies from browser access) collect, use, access this data and cross reference it with other people to create these links.
You might say, by my smartphone was at home whilst I was there, or I was in aeroplane mode...but this is you vs the words most technically gifted coders/developers etc...they'll find a way.
Given that they've matched you with people in this way already - it proves that they've got what they need about you. So in answer to your Q, maybe they used your last dialed list, or they read your texts...but they've got a shit load more incriminating evidence than that to link you to this person.