Labor under Whitlam suffered its worst-ever electoral defeat in late 1975; Young was promoted to the shadow ministry in 1976, and was given the Immigration and Ethnic Affairs portfolios.

[citation needed] Following the landslide ALP victory at the 1983 federal election, Young was initially appointed Special Minister of State (and Vice-President of the Executive Council until July 1983), but was forced to stand down in 1983 when he breached Cabinet security, as part of the Combe-Ivanov affair.

He was forced to step down again in 1984 when he neglected to declare at Customs a large stuffed Paddington Bear toy that was in his wife's suitcase.

He also made available copies of the secret Fox Report on Ranger Uranium to anti-nuclear protesters and supported their campaign to have the City of Port Adelaide declared a Nuclear Free Zone.

In addition he was active in supporting refugees and multiculturalism, and during his term as minister launched an inquiry on immigration policy aimed at reforming the system.

Following his resignation from parliament, Young worked as a lobbyist, chaired the Federal Government Multicultural Advisory Council and completed a review for the ALP following the 1995 Queensland state election.

He also continued to serve as guide to promising Labor politicians, including Beazley, who considered Young his best friend.

Young's premature death in Sydney, of leukaemia, on 8 April 1996, was felt greatly by the Labor Party and his state funeral was well attended.