Historically important as the home of revolutionary veteran Major Isaac Hite, Jr – President James Madison’s brother-in-law – and as the headquarters of General Philip Sheridan during the Battle of Cedar Creek (1864), the manor house has been little altered over the centuries and remains one of the best preserved 18th century homes in the country.

The first house on the Belle Grove site (a large limestone structure later called the “Old Hall”) was built for a tenant farmer around 1750.

Isaac, Jr, attended College of William & Mary and served in the Continental Army during the American Revolutionary War.

Upon their marriage in 1783, Isaac, Sr, gave his son and daughter-in-law – Nelly Conway Madison, sister of the future President – the 483-acre (1.95 km2) Belle Grove house and grounds.

It was constructed of local limestone quarried on the property, and, according to prevailing custom, was intended to display the owners' high social and financial status.

Thomas Jefferson, through the agency of his friend James Madison, made a number of suggestions to the architect which strongly influenced the final design.

Madison is known to have visited the plantation several times, even staying in the "Old Hall" (before the larger house was built) during his own honeymoon in 1794.

As his grain and livestock holdings grew, Maj. Hite expanded his estate to include a total of 7,500 acres (30 km2) of land and 103 slave workers.