It was established in 1979 to provide a home for an important collection of fine art donated to "be held in trust for Orkney" by the author, peace activist and philanthropist Margaret Gardiner (1904–2005).

In June 1840, Mr and Mrs Clouston entertained for a week a party of ladies travelling to join their husbands in the Hudson's Bay Company.

It remained thus until the Second World War, when the upper part of the pier store was requisitioned by the Royal Engineers as a base for planning the many army camps and installations required in the area.

Throughout the 1930s and 40s Gardiner was a key supporter of the small group of artists who sought sanctuary in St Ives and she was also an early champion of the Cornish painter and seaman Alfred Wallis.

Following the Second World War she encountered and encouraged a new generation of artists, including Peter Lanyon, Patrick Heron, Terry Frost, Margaret Mellis, John Wells and Roger Hilton, that had been drawn to St Ives by its growing reputation as a centre of innovation.

Most recently work by international contemporary artists, including Sean Scully, Eva Rothschild, Martin Boyce, Camilla Løw and Olafur Eliasson, has been acquired.