A member of the Free Trade Party and later an independent, he served twice as the 19th premier of South Australia from June to October 1892 and again from 1899 to 1901.

His wife proved to be a great boon to his career, providing political advice and serving as South Australian President of the influential Women's Christian Temperance Union.

Speculating that it contributed to his poor health, Holder had failed to seek suitable medical attention following an accident involving a mule in 1899.

[1] With considerable experience as a Councillor and Town Clerk, and just five months after his election as mayor of the Corporate Town of Burra,[2] Holder was elected to the South Australian House of Assembly in 1887 as the member for Burra, and soon gained a sound reputation in parliament.

He sat on many royal commissions during his parliamentary career in South Australia, and his reasonableness and sincerity made him a very valuable committee man.

He had only a small majority and it was a time of great financial difficulties due to a severe drought and Holder was forced out as Premier after just four months.

As Premier, Holder considered himself to be the logical choice for a ministerial position in the new federal cabinet, and was offered a cabinet position by William Lyne after Lyne was invited by the Governor-General, Lord Hopetoun to form a government and become the inaugural Prime Minister.

Labor did not observe the convention in the 1906 election, however, and contested the Division of Wakefield, but Holder was re-elected as an independent candidate.