On his retirement from Dublin City Council in 2004, his political rival, Labour Party Dublin City Councillor Dermot Lacey, paid the following tribute: "I have long respected his integrity and commitment to achieving what he believed was best for the people he represented.
In many ways Joe as a politician is the epitome of the historic mandate of Fine Gael - honourable, conservative, compassionate and committed to the democratic institutions of our State.
"[citation needed] Doyle openly discussed in public life his lifelong experience of battling with epilepsy, a consequence of which was that he could not drive, normally a necessity for a local representative elected from a constituency.
[4] Joe Doyle married Margaret Mary Maguire on 11 February 1969 in the Church of the Sacred Heart, Donnybrook, Dublin.
Joe's step-brother, John Doyle (1925–2009), was also active in politics and was Conservative Mayor and Bailiff of Hemel Hempstead in 1969.