Although the hospital closed some time after the Reformation, the name continued to be applied to its lands and was given to the current house, constructed for William Clifton around 1734.

[6][3][7] Alternatively, prior to the construction of the hospital, St Leonard's Hill may have been the site of a hermitage to shelter travellers who arrived from the south after the nightly closing of the town gates.

[8] "Le Terraris Croft" refers to the terrarius or terrar: a manager of the lands of Holyrood Abbey on behalf of its monks.

[3] An alternative origin was offered by Lord Hailes, who proposed that the title derives from Eremitae Sanctae Eremi: the monks of Saint Anthony of Egypt.

[9] The ancient crofts of the hospital of St Leonard were leased shortly before the Reformation and were in possession of the Robeson family by the end of the 16th century.

[6] In 1734, the widow of the last Reid gave to William Clifton, a solicitor of Excise, and his wife Mary "the southmost yard of Hermits and Termits, and two acres of ground adjacent thereto now possessed by the said Mr. William Clifton, and two southmost houses of the row of houses called Hermits and Termits".

William Bell Scott frequently drew the house, which he referred to as "as much a part of the family as any of its living occupants".

The railway's main cargo was coal and Hermits and Termits was soon enclosed by the boundary wall of the coalyard.

[13][16][17][18] In 1970, the City of Edinburgh Council purchased Hermits and Termits and its surrounding lands as the site for an interchange in a proposed diversion of the A1, which would take the road through the west side of Holyrood Park.

[19] A campaign, led by Moultrie Kelsall, David Black, Denis Mollison, and the Cockburn Association prevented the house's demolition.

The narrower east and west ends are gabled while, in the south front, a central gablet containing a chimney stack at its pinnacle rises above the eaves.

The blazon of Clifton's crest is: "Out of a ducal coronet gules a demi-peacock per pale argent and sable, the wings expanded, counterchanged".

[21][27] The front door opens into a panelled hallway containing a Georgian geometric staircase with twisting mahogany balusters.

[28] Prior to the house's dereliction and restoration, the Royal Commission on the Ancient Monuments of Scotland found original pine panelling in all but two of the rooms.

[20][24] This niche is an example of the buffet-niche: a common feature of dining rooms in larger Scottish houses of the 17th and 18th centuries.