He was born the son of Henry Ellison in Hebburn, County Durham and inherited his father's estate in 1795, aged 12.

[4] His generous contributions helped fund Gateshead buildings such as Trinity Chapel and St Mary's Church amongst other charitable donations.

His daughter Sarah Caroline, married Walter James, 1st Baron Northbourne.

Another daughter, Isabella Caroline, married George Venables-Vernon, 5th Baron Vernon.

Ellison's brother Robert was an officer in the British Army who fought at the 1815 Battle of Waterloo and represented him at the Newcastle upon Tyne Parliamentary election of 1820.