Built from locally quarried ashlar sandstone in the barracks style, the building measured 56 by 22 feet (17.1 m × 6.7 m) and is the only surviving example of the secular work of colonial architect Henry Ginn in NSW.
The original building was a plain, rather austere colonial Georgian structure consisting of three storeys with a basement where the kitchen, dining room and washrooms were located.
In 1843 a three-storey section facing the Governor's Domain was attached to the main building as a residence for the Superintendent, Matron and Teachers.
The orphanage was originally intended to house the children of the Female Factory women, but by 1844 with the transfer of 113 orphans from the Roman Catholic Orphanage at Waverley it became known as the Roman Catholic Orphan School.
In 1886 the site was reclaimed by the NSW Government and the following year was established as an industrial school for girls.