He was born on Vaucluse Plantation in Virginia, the eldest son of Albert Fairfax and Caroline Eliza Snowden, who were married on April 7, 1828.

The life of a miner in the mother lode of California might have been somewhat of a shock to Fairfax, who grew up as a gentleman farmer, but he stuck with the endeavor for a while.

There were stories of him working for others, pushing a wheelbarrow, or tending a mule pulling a cart of gravel and sloshing about in the mud of the diggings.

After their return to San Francisco, Fairfax and his bride visited his boyhood friend Dr. Alfred Taliaferro at his country home in Marin County.

The Fairfax estate was also near the site of the last political duel fought in California, on the afternoon of May 25, 1861, between State Assemblymen Daniel Showalter and Charles W. Piercy.

Though Fairfax served them lunch and tried to dissuade them, the two men walked to a grassy meadow and fired rifles at 40 paces; Piercy was killed by the second volley.