The buildings were built by slaves who also grew and harvested hemp, farmed livestock, and cooked and cleaned for the Clays.

[3] In 1829, 28 years before the more famous Dred Scott challenge, Charlotte Dupuy sued Henry Clay for her freedom and that of her two children in Washington D.C. circuit court.

[3] She was ordered to stay in Washington while the court case proceeded, and lived there for 18 months, working for Martin Van Buren, the next Secretary of State.

Clay had Dupuy renditioned to New Orleans and had her held by his daughter and son-in-law, where she was enslaved for another decade.

After his father's death, son James Brown Clay owned and occupied Ashland and a surrounding tract of about 325 acres (132 ha).

They remodeled and modernized the house, updating it with gas lighting (later, electricity), indoor plumbing, and telephone service.

Their eldest daughter Nannette McDowell Bullock continued to occupy Ashland until her death in 1948.

The borough of Ashland, Pennsylvania, in Schuylkill County, an anthracite coal mining town, was named in honor of the estate as well.

2 Clay put a notice in a local paper asking for the return of a lost horse and listed his home as Ashland.