It was maintained by Clan Wemyss from its construction around 1560 until it was put into the care of the Secretary of State for Scotland in the early 20th century, though was not occupied for the entire time.

The castle has been a scheduled monument since 1920 on the grounds of being "a particularly fine example of a Medieval tower-house", and the cottage became a listed building in 1971 in recognition of its national importance.

The castle is unusual in that it has both en suite guest accommodation like a mansion, but also a large number of gun loops.

Elcho Castle consists of an approximately Z-plan tower house, with evidence of a surrounding wall (a barmkin).

There is a walk-way on top of the walls, with a parapet, though as with the south-eastern corbelled turret featuring large windows, it was thought to be more for inhabitants to view the surrounds than the defence.

[1]: 18–20  The upper floors were large for the time, and included multiple smaller rooms with roof-top walkways.

[1]: 16–17 A quarry nearby had been flooded and used a private dock on the river with what was thought to be a 19th-century jetty, though it is now the site of a garden.

[7][8] The surrounding wall would have enclosed ancillary buildings, but not much has survived beyond the ruin of a small round tower in the south-east corner, which also featured a gun port.

[9] In 1468 James III of Scotland confirmed that ownership of the east Elcho lands went to Sir David's son, John Wemyss, as recorded in The Register of the Great Seal.

James' eldest son, David Wemyss, was heavily involved with the Jacobite rising of 1745 and escaped to France and lost his titles.

It has nevertheless been kept in good repair – one of the earliest examples in Scotland of a building being preserved purely for its historical interest.

[3] It is managed by Historic Environment Scotland as a scheduled monument since 30 April 1920 in recognition of its national importance[27] and guardianship of it was given to the state in 1929 by Hugo Charteris, 11th Earl of Wemyss.

[38] A large portion of the Castle is accessible, but a lot of the wooden floor of the third level in the main block is missing.