The event history of the Annual Cross Harbour Swimming Race can be read here The Victorian Recreation Club's website has a photo in slideshow of the swimming race circa 1900.
Noted a Hong Kong Telegraph mention here from 1881 announcing a swimming match from Kowloon Point to the Bath House (Victoria Recreation Club).