It is located beside Wigtown Bay in the historical county of Kirkcudbrightshire in Galloway, Scotland, around 4.8 kilometres (3.0 mi) south east of Creetown.

The lands of Carsluith were held by the Cairns family until 1460, when they passed to James Lindsay of Fairgirth, Chamberlain of Galloway.

The Browns (or Brouns) of Carsluith added to the castle, building the stair tower on the north side in the 1560s.

Later it was alleged several times that Gilbert was sheltering Jesuit priests at Carsluith, and in 1605 he was arrested for his Catholic sympathies.

The Brown arms are carved above the door, with initials and the date 156-, the last digit being illegible, although it was said in the 19th century to have been legible as 4.

On the north side, holes exist in the outside wall which would have supported an external timber gallery linking the second floor rooms and stair tower.