The buildings were built by slaves who also grew and harvested hemp, farmed livestock, and cooked and cleaned for the Clays.

[3] In 1829, 28 years before the more famous Dred Scott challenge, Charlotte Dupuy sued Henry Clay for her freedom and that of her two children in Washington D.C. circuit court.

[3] She was ordered to stay in Washington while the court case proceeded, and lived there for 18 months, working for Martin Van Buren, the next Secretary of State.

The court ruled against Dupuy, and when she refused to return voluntarily to Kentucky, Clay had her arrested.

After his father's death, son James Brown Clay owned and occupied Ashland and a surrounding tract of about 325 acres (132 ha).

They remodeled and modernized the house, updating it with gas lighting (later, electricity), indoor plumbing, and telephone service.

Their eldest daughter Nannette McDowell Bullock continued to occupy Ashland until her death in 1948.

The borough of Ashland, Pennsylvania, in Schuylkill County, an anthracite coal mining town, was named in honor of the estate as well.

2 Clay put a notice in a local paper asking for the return of a lost horse and listed his home as Ashland.