The house was reconstructed in 1776 by architect Samuel Wyatt in a Neo-Palladian style as is obvious from the curved facade of the current ruined building to the terraces, follies and balconies.

In the nineteenth century the occupants of Baron Hill remained the dominant Anglesey landowners, possessing estate also at Llanfairfechan and other parts of Caernarfonshire.

In 1939 at the outbreak of World War II, the mansion was requisitioned by the government and used as a temporary housing for Polish soldiers.

The Polish soldiers found the home too cold and decided to start a fire within the mansion so they would be moved to new housing accommodations.

The fire destroyed a large part of the interior and the soldiers were removed from the house but only to tents on the Baron Hill Estate grounds.