Much of the house was burned down by Royalists during the English Civil War in 1645 and abandoned by the Spencers in favour of Althorp in Northamptonshire, which contains some materials salvaged from Wormleighton to this day.

As the family wealth grew dramatically, John Spencer purchased the land at Althorp between 1509 and 1511 and constructed another residence there.

[3] In 1613, the gatehouse at the entrance of Wormleighton Manor was added by Sir Robert, first Lord Spencer, and he or his son are believed to have made alterations or enlargements also to the main building.

In 1645, Royalist forces from nearby Banbury set fire to Wormleighton Manor to prevent it becoming a parliamentary stronghold, causing extensive damage.

The east tower at the side of the gateway is roughly 16 feet (4.9 m) wide with a four-light window on the lower part.