The fort itself was demolished in November 1864 as the town continued to grow as a commercial centre serving the local area as well as trading with California, Washington Territory, the United Kingdom, and others.

[2] The original headquarters of HBC operations on the Pacific Coast of North America at the time of Victoria's founding was Fort Vancouver (now Vancouver, Washington) on the lower Columbia River, but its location was difficult to defend, ships often had difficulty entering the mouth of the Columbia, and it was far from the lucrative furs in New Caledonia farther north.

With American settlers beginning to come into the region, in 1843, the company sent Chief Factor James Douglas to build a fort some distance north on Vancouver Island and made him its superintendent.

Animal populations were beginning to dwindle from overtrapping, slowing the fur trade, but the California gold rush created a huge demand for resources with few places to buy them on the unsettled west coast.

The company was soon trading salted salmon with Hawaii and outfitting Royal Navy ships with supplies for the Crimean War.