About 1335, Segrave married Margaret of Brotherton, daughter and eventual sole heiress of Thomas of Brotherton, son of King Edward I of England by his second wife, Margaret of France.

[2] Their children were: A year after Segrave's marriage to Margaret of Brotherton, his wife inherited her father's title and estates, becoming in her own right Countess of Norfolk and Earl Marshal of England.

The impetus for this was that Margaret wished to marry Lord Manny, with whom she had an understanding.

[6] However, Segrave died at Bretby in Repton, Derbyshire, on 1 April 1353,[7] before the annulment had been granted or refused.

He was succeeded in the barony by his daughter, Elizabeth, 5th Baroness Segrave.