Colwick Hall is constructed of red brick, with ashlar dressings and hipped slate roofs with a 2-storey central block and single-storey wings.

The Byrons lived here for over 150 years until about 1660, when they moved to Newstead Abbey and Colwick Hall came into the ownership of the Musters family.

In 1805 John Musters's son Jack married Mary Chaworth, Byron's childhood love-interest from Annesley Hall.

In 1827 Jack inherited Colwick Hall from his father, but in 1831, during the Second Reform Bill disturbances, it was sacked and partly burned by rioters.

Mary Chaworth Musters spent the night in pouring rain with her daughter Sophia, crouched beneath the shrubbery and died at Wiverton Hall some four months later from the shock.