[2][3][4] Built on the site of an earlier 15th-century castle, White's Castle was built to guard the main river crossing in Athy (Irish: Baile Átha Í, meaning 'town of the ford of Ae') and was extended over successive centuries.

White's Castle is located in the centre of Athy, on the east bank of the River Barrow, overlooking Crom Abú Bridge.

In 1573, Richard Coosen, Sovereign of Athy added a crucifixion plaque on the left side of the original doorway.

[10] In the early 18th century, the building was redeveloped for use as a gaol,[1] and by 1730 White's Castle became the local prison.

[10] A small extension was added to the north end in 1830 and the building was used as an Irish Constabulary barracks.