He was the son and heir of Henry William Berkeley Portman MP (died 1761), by his wife Anne Fitch.

His grandfather was William Berkeley (died 1737) of Pylle, Somerset, who had changed his surname, by a private act of Parliament, Portman's Name Act 1735 (9 Geo.

He succeeded his father in the estates of Bryanston and Orchard Portman in 1761, and to the Berkeley estates at Pylle on the death of his aunt Lady Burland.

He developed 200 acres (0.81 km2) of meadow in London (between Oxford Street and the present site of Regent's Canal) he had inherited from his Tudor ancestor Sir William Portman, turning it into the Portman Estate.

On 20 January 1766, he married Anne Wyndham, daughter of Sir William Wyndham, of Dinton House, Wiltshire, by whom he had two sons and three daughters.