Murray inherited the land on the death, in 1751, of his father, who had consulted William Adam about a new house at Cally in the 1740s, although nothing had then been done.

[1] While travelling in Rome in the mid-1750s, James Murray met the young architect Robert Mylne, who prepared the first plans while still abroad.

[2] The building was complete by 1763, and over 1,000 acres (400 ha) of grounds were laid out with orchards and pleasure gardens, as well as hothouses and deer parks.

His grandson, Alexander Murray, made alterations to the house, including the portico by John Buonarotti Papworth.

[1] Elizabeth Murray Usher, who inherited Cally in 1924, sold the house and grounds to the Forestry Commission in 1933.