It housed the cathedral's first bishop, William the Old of the Norwegian Catholic Church who took his authority from the Archbishop of Nidaros (Trondheim).

Originally, it is thought to have been like a typical Royal Norwegian palace, with a large rectangular hall above store rooms and a tower house as the Bishop's private residence.

King Haakon IV of Norway, overwintering after the Battle of Largs, died here in 1263, marking the end of Norse rule over the Outer Hebrides.

In 1526, the palace came briefly into the possession of William, Lord Sinclair, before he was ordered to return it to the Bishop of Orkney.

Earl Patrick's son Robert seized both palaces in 1614, and a siege followed, though it is not known if this caused damage to the structures, both of which are now ruins.