Cornwallis's parents lived much of the time in London, his father being a Royalist and an Equerry to Charles I, while his mother was a Lady-in-Waiting to the Queen.
With his parents busy at Court, Cornwallis and his three siblings were raised at Culford Hall by their grandmother, Lady Jane, who was by then married to her second husband, Sir Nathaniel Bacon.
Their home, Culford Hall, had been built by Lady Jane's father-in-law, Sir Nicholas Bacon, 1st Baronet.
It is through Lady Jane that Culford Hall eventually passed from Bacon to Frederick, becoming home to the Earls of Cornwallis.
He became J.P. in 1662 and as county magistrate, he was one of the appointees at the assize who oversaw a test of accused women in the Lowestoft Witch Trial.