Discretion for the client and safety of the girl is the name of the name of the game here.
He should drop the girl off, wait for her to go in the house and then drive away out of sight.
He should return only when time is up and wait in the car, with the engine turned off, until the girl comes out and then drive straight away.
Obviously if the appt. is curtailed for any reason then the girl can phone the driver and he should be able to come straight back for her.
The driver will have to know the client's address but he won't know the client's name or phone number.
Any agency or escort should only use a driver who is willing to work to a sensible set of rules that makes the whole experience better for everyone concerned.