He served in the Royal Australian Navy during World War II and completed a law degree at the University of Western Australia upon his return.

His father was a former locomotive driver who had been elected to the Western Australian Legislative Assembly earlier in 1924 and served for over 20 years as a Labor MP.

[1] Returning to Bunbury to practise law, first as a solicitor and, from 1953, a barrister, Withers was elected to Bunbury Municipal Council and began to involve himself in Liberal Party affairs, serving at various times as Liberal Party State President and Vice-President and Federal Vice-President.

[1] Described as having a "jovial manner and perpetual grin", Withers quickly gained a reputation as the Liberal numbers man and served as Senate Government Whip from 1969–71.

Withers was widely known as "The Toecutter" for his alleged approach to enforcing party loyalty and his role in the 1975 Australian constitutional crisis.

After the dismissal of the Whitlam government on 11 November 1975, Withers was appointed to Malcolm Fraser's first (caretaker) ministry, becoming Vice-President of the Executive Council as well as briefly holding the portfolios of Special Minister of State, Capital Territory, Media, and Tourism and Recreation during the period leading up to the December election.

[4] He was dismissed by Fraser in the wake of the findings of a Royal Commission into aspects of a redistribution of certain federal electorates in Queensland.

[5] At the time he commented about Fraser that "When the man who's carried the biggest knife in this country for the last ten years starts giving you a lecture about propriety, integrity and the need to resign, then he's either making a sick joke or playing you for a mug".