His descendant Edmund de la Pole, 3rd Duke of Suffolk (1472–1513) was forced to surrender his dukedom in 1493.

In 1384 records show that Michael de la Pole, 1st Earl of Suffolk applied for royal licence to crenellate his manor house.

At that time the existing Tudor brick merlons on top of the south curtain wall were added.

The plan of the site is quadrilateral, almost square, the west side being a little longer than the east, an enclosure of about an acre and a half.

A brick bridge now leads up to the noble gatehouse where the fine depressed pointed archway, deeply recessed and moulded, still shows the portcullis groove and the old oak gates.