Stuart was born in Nippenose Township, Pennsylvania, and worked on his father's farm until the age of 14, when he was enrolled at Owego Academy in what is now Tioga, New York, where he studied under the educational reformer Charles Rittenhouse Coburn.
They constructed a house and dug a canal several hundred feet long to accommodate boats on the nearby creek.
Shortly afterwards, Stuart and business partner Robert T. Ridley established a tavern on the site, called the "Mansion House."
He tried to lease the New Almaden quicksilver mine—the state's most financially successful mine—but sold his interest in the mine to escape the extremely complicated litigation over its ownership.
In 1859, Stuart purchased a part of the Rancho Agua Caliente land grant in Sonoma County and in 1868 began building a house there, eventually establishing a 1,000-acre (4.0 km2) vineyard he named Glen Ellen after his wife.
Again, on February 1, 1879, when the convention was finalizing the Constitution's language, Stuart spoke against such provisions, declaring, Give to the children of these people (and some of them native born) the privilege of our common schools in return for the school taxes they pay; cease persecuting them by personal assault, to which the law is blind; stop this disgraceful special legislation against them; stop this relentless, heartless, and inhuman persecution of foreigners...and then, and only then, will we do our duty.
They pass cruel ordinances against them; they harass and annoy them through every device the law can invent, and why are similar outrages heaped upon them in nearly every county, town, village, or hamlet in this state?
Although Stuart was ridiculed and attacked for this speech, the convention failed to approve the provisions to which he objected, and many of the later laws enacted restricting the rights of the Chinese were declared unconstitutional by federal courts.