Sometime in 1841, perhaps very soon after the sale of 14 June 1841, they obtained a transfer from a Captain Ramsay of what was then Town Lot 42, and there erected a large house of which the Canton Press caustically commented that "on entering the harbour, you perceive the most commanding site, disfigured by a hybrid erection, half New South Wales and half native production, which is a foretaste of the architectural absurdities to be perpetrated on this island
As this 1842 map shows town Lot 42 is exactly where Flagstaff House was built. The house that Jardine built soon became the residence of Lord Saltoun:
The house later became the residence of Lord Saltoun, Commander of British Forces in China during the war which ended with the Treaty of Nanking in 1842. The present Flagstaff or Headquarters House, built by 1846, now stands on this site...They were able to occupy neither building for long: early in 1842, Colonel Malcolm, Pottinger's secretary, wrote to them, extending an offer to compensate them for moving away to allow the area to be used by the Military
Source: Jardine Matheson & Co's First Site in Hong Kong. Dafydd Emrys Evans. Vol 8 p 151.