The house is located on Castle Hill, off Agincourt Square in Monmouth town centre.

Described as "a house of splendid swagger outside and in",[2] it was completed in 1673 for Henry Somerset, the 3rd Marquis of Worcester, who was Lord President of the Council of Wales and the Marches.

The house was built of mottled pink and grey blocks of Old Red Sandstone, probably mostly reused from the gatehouse and Great Tower of the castle, which were wholly or partly demolished after the Civil War; much has been renewed more recently.

The plasterwork was mostly made by travelling craftsmen using established patterns, but has been lavishly and ostentatiously fashioned with pendant floral festoons laced with ribbons in what were originally the Marquis's private rooms.

[6] The Museum opens onto the King's Garden, containing plants known to have been growing at the time of Henry V (1386–1422), who was born in the castle.