The house was built in 1763[1] and designed by William Chambers[2] for James Caulfeild,[3] the 1st Earl of Charlemont.

It was purchased by the government in 1870[6] and since 1933 it has housed the Hugh Lane Dublin City Gallery.

[7] The house features in James Malton's views of Dublin where it is illustrated partially obscured from the corner of Rutland Square.

The house is one of the locations featured in the book, The Coroner's Daughter by Andrew Hughes, which was selected as the Dublin UNESCO City of Literature One City One Book for 2023.

[8]The earl kept an extensive art collection at the house, among them included Judas Repentant, Returning the Pieces of Silver by Rembrandt, The Lady's Last Stake and The Gate of Calais by William Hogarth as well as other lesser known paintings by Annibale Carracci, Tintoretto, Ambrogio Bergognone and Anthony van Dyck.