Chapel of the Conception / First Roman Catholic Cathedral [1843-1886]
It had two towers, which stood above the surrounding buildings and made it a landmark at the time.
I'm not sure of the dates it was built or demolished either. HFSiu notes that:
According to a book "City of Victoria", the old Roman Catholic Church at Wellington Street was demolished after the new church was opened in 1888. It seems the old church was demolished well before the opening of new church.
He says that because in this photo, the second cathedral is still being built, but the towers of the old one can't be seen.
Here's a clue, saying the old cathedral was sold in 1881, though with the condition the church would continue to use it for some time:
[14 Mar 2016 update: adjusted marker to correct location based on 1845 map.]