Originally, a wooden motte and bailey castle built soon after the Norman Conquest, it was rebuilt as a stone fortification during the twelfth century but subsequently demolished on the orders of Henry II of England.

The fortifications were slighted by order of the Parliament of England during the English Civil War and the residential buildings were destroyed by fire in 1750.

Localised structural problems led to it being placed on Historic England's Heritage at Risk register in 2020.

[8] The keep (the most obvious part of the castle when viewed from the town) and the main gate date from this re-building.

John and Margaret were only in possession of the castle for a few years before the property was seized by the younger Hugh Despenser, a favourite of Edward II of England.

According to Historic England, the Sharrington Range represents "one of the earliest known examples of the influence of the Italian Renaissance on the secular architecture of the West Midlands.

"[11] John Dudley was executed in 1553 for his attempt to set Lady Jane Grey on the throne of England.

In 1592, this Edward sent men to raid the property of Gilbert Lyttelton, carrying away cattle which were impounded in the Castle grounds.

[14] The castle was partly demolished to prevent it from being used again and the present ruined appearance of the keep results from this decision.

However, some habitable buildings remained and were subsequently used occasionally by the Earls of Dudley, although by this time they preferred to reside at Himley Hall, approximately four miles away, when in the Midlands.

[17] The borders were changed to include the castle and its grounds within the Dudley borough only in 1926 when the restructuring of the boundaries took place to allow the development of the Priory Estate.

[11] After the slighting at the end of the civil war, only the north side of the castle and parts of two of the drum towers remain.

Some elements of Paganell's Norman castle remain in the structure, but it mainly dates from the rebuilding carried out after 1262 by the de Somery family.

[20] Constructed for John Dudley, starting around 1540, the three-storey range included a great hall, kitchen, servery, buttery, cellars and bedrooms.

A small amount of masonry dating from the early Paganell castle is evident in the ruins.

Once thought to be lodgings, the stable block was one of the last buildings constructed at the castle site, dating from before 1700.

Two Russian cannons brought back as trophies from the Crimean War are installed in prominent positions on the remains of the two south-facing drum towers.

One of the supernatural presences that has usually been sighted at the site is the Grey Lady, who is believed to be the spirit of Dorothy Beaumont, a woman who died in the Castle, along with her baby, shortly after childbirth.

She'd requested to be buried next to her daughter and for her husband to attend the funeral, but neither happened and so it's thought she now wanders the castle and its grounds.

Since opening, there have been many reports here, mostly of unexplained sounds, alarms going off in the middle of the night without explanation, and extreme drops in temperature that are often accompanied by a strange blue mist that floats through the bar.