The village is situated on Elwick Bay, which was used as an anchorage by Haakon IV of Norway before sailing south to eventual defeat at the Battle of Largs in 1263.

In 1782, to make way for a new residence, Balfour cleared cottars from the south-west of the island, and appropriated part of the common, building the village of Shoreside to house those evicted.

[2] To help finance his building work, Thomas Balfour borrowed money from his brother John, who made his fortune serving with the British East India Company.

The following year, he recruited an Edinburgh architect, David Bryce, to transform Cliffdale House into the Scottish Baronial Balfour Castle.

Balfour added an imposing gatehouse, now the village pub and home to the local football team; a water mill and a gas works which remained operational until the 1920s.