There are remains of the 14th-century manor house of Walterus Heryng de Winterborne, who was granted a licence to crenellate in 1336, at the core of the current building, although some of this was removed in the 19th century and none of the remains are visible.

[2] The house's next major phase began in 1569, when Sir John Williams inherited the estate.

He greatly embellished and altered the building, giving it much of its present character and beginning its famous great chamber (which was unfinished at his death in 1617).

[2][3] Alterations were carried out in the late 19th century by Thomas Leverton, leading to the remodelling of the north (entrance) front and the creation of a new entrance hall.

[2][3][5] The current family head is Raymond Audley Edward Wilmot Williams.