The ghost towns of Azurite and Mineral Hill were located about one mile north of San Xavier.

Several mines and associated camps were located in the immediate area, including Azurite, Mineral Hill, Wedge, and the Michigan Maid, though there were no smelters, and ore had to be shipped elsewhere.

[2][6][7] After the Brown family moved away, silver prices fell and mining in the Pima District switched to copper, leading to the decline of the town.

The Olive Post Office was briefly open from March 4, 1887, to May 23, 1892, although operations at the nearby San Xavier Mine continued until 1918.

The large, modern Pima-Mission Mine is located immediately east of San Xavier.