His past experience of hardship to walk for four days from Ranau to reach Kota Kinabalu made him resolve to find ways to alleviate this problem and to initiate the road links between the east and west coasts of Sabah later in his life.
His political career started after an acquaintance with Donald Stephens who has encouraged Ghani to write short articles in North Borneo News and Sabah Times.
In 1962, after returning from New Zealand from a six-month leadership course, Ghani was first appointed to the legislative assembly by Sir William Goode to represent Ranau until he resigned to take up his federal ministerial post in 1968.
In 1968, Ghani at that time at the age of 36, was appointed to the federal cabinet by Tunku Abdul Rahman to the post of Minister of Sabah Affairs and Civil Defence.
Ghani was re-elected to parliament in 1974, and in 1975 joined Tun Fuad Donald Stephens and Harris Salleh in the formation of the new political party Sabah People's United Front (BERJAYA).