After what seems like several years of fruitless searching, I finally managed to identify this building, whose exterior was used for the establishing shot for the dinner party where Suyin meets Elliot for the first time.
The building appears to have been demolished prior to 1963 (according to the aerial images for that year) but the podium was still intact including the front driveway from Po Shan Road.
The site consists of two plots: IL2447 S.A and IL2447 RP. The west plot having the house on it and the east one consisting of the podium garden.
When Po Shan Mansions was built the old front driveway was concreted in and a new driveway constructed that runs up to the rear of the podium, however, if you look at the podium today you can see the brickwork of the old driveway running up in an angle from east to west.
It would be interesting to hear from anyone who lived on or near Po Shan Road in the early 60s who remembers anything about the place.