[1] In 1761 Brograve married Jane, eldest daughter of Edward Hawker of Great Baddow, Essex.
They had four children, only two of whom survived, including Ann who married Dr John Rye, the founder of The Shipwrecked Fishermen and Mariners' Royal Benevolent Society.
A granddaughter married Sir William Beauchamp-Proctor, 3rd Baronet, in whose family the name of Brograve descended as a given name.
After building his mansion-house at Worstead he was described as occasionally inhabiting the manor house at Waxham.
According to local legend, like his ancestors, Sir Berney is still around and can be seen on wild and stormy nights galloping between Worstead and Waxham.