In September 1796 he left briefly for England in the Britannia to bring back his family to settle permanently in New South Wales.
In this post, he was responsible for the reception and issue of all government stores, virtually the only supplies in the colony, and their supplement by purchase from private merchants.
In his judicial capacity as a magistrate, which he had been appointed by Lieutenant Governor Francis Grose in 1793, and as one of the principal civil officers, Palmer was familiar with most of the disturbances that occurred in the colony.
A supporter of Governor William Bligh, he had in 1809 briefly been placed in gaol in Sydney on a charge of sedition for having declaring New South Wales to be in a state of mutiny.
Palmer denied the competency of the court and refused to plead, but was found guilty and sentenced to three months imprisonment and directed to pay a fine of £50.
He also continued to refuse to allow Lieutenant Governor George Johnston access to his ledgers without the authority of the British Treasury.
As a magistrate, he sat frequently on the bench at Parramatta until dismissed by Governor Sir Thomas Brisbane over a quarrel in 1822.
When he died at Waddon near Parramatta on 27 September 1833, he was "the last surviving officer of the first fleet that arrived in this part of His Majesty's Dominions.