Abdul Ghani received his early education at Masjid Tanah National School, Melaka and Madrasah Aljunied Al-Islamiah, Singapore before attending Maahad Buuth Islamiyah al Azhar, Egypt (1965–1967).
He then continued his studies at Al-Azhar University in Egypt (1967–1974) and obtained a Masters (Sharia) in the field of comparative sects and law.
About 6 months later, he was ordered out of Sabah within 48 hours by the Chief Minister, Datuk Harris Salleh and returned to ITM Shah Alam for a while.
He was involved in Angkatan Belia Islam Malaysia (ABIM) until he was entrusted as Deputy President in the early 80s.
He was also active in the Persatuan Ulama Malaysia (PUM) and was appointed as its president in 1999–2003, replacing Ustaz Ahmad Awang.