The name Harcla is thought to be from the Old English for "hard ground" and may refer to the outcrop of land that the castle is built upon in the Eden valley.

The existing manor house was fortified by de Harcla, some time prior to 1323, when he was ordered by King Edward II to be hanged, drawn and quartered for alleged collusion with Robert the Bruce, and forfeited his earldom and lands.

It was granted to Ralph de Neville who later sold it through three other hands to Thomas de Musgrave who on 4 October 1353 was granted a licence to crenellate by King Edward III: It was improved during the 17th century with the addition in 1615 of a pair of wings but was abandoned circa 1677.

The castle layout appears to have been similar to Sudeley Castle in the Cotswolds, with its double courtyards separated by a low dividing wall and a central small door and its oriel-windowed hall, but without Sudeley's later Southern Wing and with a curtain wall demarking the second courtyard instead.

Apart from limited earthworks, all that remains now is a few metres of wall and stairs down to a vaulted cellar for the former kitchen.