Clifford was born in Mount Vernon, Scotforth, Lancashire, England, eldest of nine children of George Lambert Clifford (1779–1854), of Wycliffe Hall, Yorkshire ("an elegant mansion with a Catholic Chapel, and beautiful leisure grounds")[1] and of Tixall, Staffordshire, and Mary (died 1854), daughter of Walter Hill Coyney, of Weston Coyney, Stoke-on-Trent, Staffordshire.

[3] After attending Stonyhurst College (as his father had, being recognised as the very first pupil to enter, commemorated by a bust in his honour at the school)[4][5] Clifford set out for New Zealand with his cousin William Vavasour, leaving in 1842.

Arriving in the New Zealand Company settlement of Wellington, the two established a land, shipping and commissions agency with finance from their parents.

[12] The most challenging event to arise during Clifford's speakership was the prorogation of Parliament by Robert Wynyard, the acting Governor.

Parliament chose to suspend its own standing orders, allowing it to leave Wynyard's instructions officially "unopened" while it continued to debate.

The possibility of suspending standing orders was challenged by Wynyard's supporters, but Clifford eventually decided to allow it.