Colville served in the Grenadier Guards, reaching the rank of Lieutenant.

Called to the Bar at Lincoln's Inn in 1960, he became a Queen's Counsel in 1978 and a Bencher in 1986.

Between 1983 and 1987, Colville was the United Nations special rapporteur on the situation of human rights in Guatemala.

He also served in the British government as chair of the Mental Health Act Commission.

He was chairman of the Parole Board for England and Wales from 1988 to 1992, Recorder from 1990 to 1993, and Judge of the South Eastern Circuit from 1993 to 1999.