His father was first cousin (on his father's side) of Sir Austen Henry Layard (excavator of Nineveh and Nimrud), Edgar Leopold Layard (Curator of the South African Museum at Cape Town, and Governor of Fiji), and of Lady Charlotte Guest (translator of the Mabinogion and collector of ceramics).

His grandfather Brownlow Villiers Layard was aide-de-camp and afterwards (1802) private chaplain to the Duke of Kent (and brother of the Governors of Malta and Curaçao and of Lady Lindsey), and was the son of a Dean of Bristol and grandson of the accoucheur Daniel Peter Layard.

C. C. Layard was also first cousin (on his mother's side) of Lady Llanover (of the Welsh cultural revival), being the son of Louisa Port, sister of Georgiana (favoured grandniece of Mrs Delany and companion of Fanny Burney), and therefore a descendant of Bernard Granville of Calwich and of Sir Richard Grenville of 'The Revenge'.

In the 1830s her brothers were the largest ship-owners in London and held contracts for convict shipping to Australia.

His sister Nina Layard was a poet, prehistorian, archaeologist and antiquarian who conducted important excavations, and one of the first four women to be admitted as Fellow of the Society of Antiquaries of London.