The main house burned down during a dinner party in 1766 and the remaining structure sat dormant for many years until New York lawyer Jonathan Prescott Hall built a new roughly 5,800 sq ft (540 m2) castellated residence directly on top of the old ivy-covered ruins.

Colonel Malbone made his fortune as a shipping merchant and slave trader, becoming one of the wealthiest men in Newport during the 1740s through privateering and the triangle trade.

[6] The Halls commissioned Alexander Jackson Davis, a notable 19th-century New York architect, to design a house of pink Connecticut sandstone in the popular Gothic revival style of the time, incorporating some original elements such as the porte-cochere from the previous home.

Prescott Hall renovated these gardens from 1848 to 1850, expanding them to 17 acres and enlisting Andrew Jackson Downing, the leading landscape designer of the 1840s and an advocate of architectural philosophy.

[citation needed] Downing partnered with Calvert Vaux to design the White House grounds and National Mall and is widely regarded as the "Father of American Landscape Architecture."