It originally stood on a small island in the Firth of Forth accessible only at low tide, and dates from around 1450, built as a secure residence by Sir David Stewart, who had been granted the Barony of Rosyth in 1428.
In 1572 it was attacked by men from Blackness Castle on the southern shore of the Firth of Forth and it was occupied in 1651 by Oliver Cromwell's army after the Battle of Inverkeithing.
It ultimately ended up in the possession of the Earl of Hopetoun and from the eighteenth century onward remained unoccupied.
It became Admiralty property in 1903 and as the result of land reclamation lost its waterfront position, becoming marooned within the dockyard.
About half a mile north of the castle is a well-preserved sixteenth-century dovecot, with a crow-stepped gable roof, with carved heads at two corners.