During this time, Fisher wrote to his friend Joseph Howe about the evil ways of the family compact and on the irresponsible nature of the government and its politics.

His leadership ended in 1861 when he was ousted by fellow reformer Samuel L. Tilley due to a scandal over the leasing of crown lands.

[citation needed] He was elected to the House of Commons in 1867 as a Liberal but resigned his seat in 1868 to accept an appointment to the New Brunswick Supreme Court.

One of his notable decisions as a judge of the Court was his lone dissent in Dow v Black, a significant constitutional law case dealing with the federal-provincial division of powers.

However, his position was upheld on appeal by the Judicial Committee of the Privy Council, at that time the court of last resort for the British Empire, which ruled that the statute was within provincial powers.