[1] In 1761 Brograve married Jane, eldest daughter of Edward Hawker of Great Baddow, Essex.
They had four children, only two of whom survived, including Ann who married Dr John Rye, the founder of The Shipwrecked Fishermen and Mariners' Royal Benevolent Society.
A granddaughter married Sir William Beauchamp-Proctor, 3rd Baronet, in whose family the name of Brograve descended as a given name.
His second wife, Lady Jane, died in 1793 at Waxham, followed the next day by the death of their 18-year-old son, Thomas.
According to local legend, like his ancestors, Sir Berney is still around and can be seen on wild and stormy nights galloping between Worstead and Waxham.