Between 1780 and 1790, the main house was built by his great-great-grandson, Edward Lloyd IV, using the profits generated by the forced labor of enslaved people.

Nearby the house is an orangery, a rare survival of an early garden structure where orange and lemon trees were cultivated, and which still contains its original 18th-century heating system of hot-air ducts.

[4][5] During its peak, the plantation's owners enslaved more than 1,000 people to work lands that totaled more than 42,000 acres (17,000 ha).

[6] Though the land has shrunk to 1,300 acres (530 ha) today, it is still owned by the descendants of Edward Lloyd, now in their 11th generation on the property.

[3][6][8] The Wye House plantation gained media attention in 2006 for archaeological investigations led by the University of Maryland.