Through his marriage to Mildred Meriwether in 1741, Thomas Walker acquired the land comprising approximately 15,000 acres (6,100 ha) which would become the site for Castle Hill.

In the early 20th century, Amélie's sister, Mrs. Gertrude Rives Potts, who managed the estate after their father's death, was recognized as the first woman Master of Foxhounds.

[4] The original clapboard, colonial residence was built by Walker in 1764, with a front porch facing west and six dormer windows.

William and Judith Rives added the brick, federal style addition to the home in 1824, which was built by Captain John Perry, one of Thomas Jefferson's master brickmasons.

The current expanded, multi-terraced and symmetrical formal gardens were designed in 1997 by Landscape architect Rachel M. Lily.