He was the eldest son of Sir Walter Bagot, 5th Baronet, and his wife Lady Barbara Legge.

Among his siblings were Charles Bagot (who married Catherine Legge), the Rev.

His niece, Jane Margaret (daughter of his brother Walter by his second wife, Mary Ward) married the English judge Sir Edward Vaughan Williams in 1826 and they were the grandparents of the composer Ralph Vaughan Williams.

[2] He was educated at Magdalen College, Oxford, and graduated Master of Arts in 1749 and Doctor of Civil Law in 1754.

[6] On 20 August 1760, Bagot was married to Elizabeth St John (c. 1744–1820) in Wroxham.