The original castle, a manor house, came about as the result of a licence obtained in 1306 by Walter Langton, Bishop of Coventry, to castellate his mansion in the village of Ashby.

Like other houses of its time, it has an E-shaped floorplan, with a deep central porch and flight of steps forming the centre stroke of the E. This was to celebrate the coronation of Queen Elizabeth I.

[2] By 1635 an ambitious classicising screen had been added across the open southern side of the courtyard, probably to make the two wings directly accessible to each another; its design deviates enough from Palladian canons to make it unlikely that Inigo Jones designed it;[3] Howard Colvin[4] suggested that a payment of £8 to "Cartor Surveyor" in the Earl's accounts, September 1631, may refer to Edward Carter, Jones's deputy at St. Paul's Cathedral, 1633–41.

The folklore is that an old woman known as Elspeth, who lived in the parvise over the north porch of the church, first noticed the blaze and alerted the village, thus saving the remainder of the house.

Apart from "altering" the avenues into small clumps of trees and doing away with the Elizabethan gardens, he enlarged the ponds overlooked by the house into ornamental lakes, dug a ha-ha, or sunken fence, around the park and built the dairy and the temple against the menagerie.

At this time the terraces round the house with their terracotta balustrading were laid out, and the Italian "golden gates" at the entrance to the front drive were hung on piers designed by Wyatt himself.

However, before all the changes were carried out, Lady Northampton died of consumption, giving rise to the pitiful Latin inscription in terracotta lettering beside the church: "To Theodosia, sweetest of wives…Begun in hope, finished in despair 1865."

Plans for the redecoration of the Long Gallery were drawn up by William Burges,[6] but these were not executed and the work was undertaken by Thomas Graham Jackson.

Academic Dr. Robert Clark theorized Jane Austen based her 1814 novel Mansfield Park's eponymous fictional Northamptonshire country house on Castle Ashby.