His father was a British civil engineer who had gone to the United States to represent the hydro-pneumatic sewerage system of Isaac Shone.

[2] Broughton was named after his father, Urban, and his mother's paternal grandmother Mary Eldridge Huttleston.

[1]: 23 Broughton's father died in January 1929, shortly before he was due to receive a peerage in the 1929 New Year Honours, the announcement of the list having been delayed by two months due to the health of George V. The barony was instead awarded to Broughton, as the eldest son, who became the first Baron Fairhaven, of Lode in the County of Cambridge and his mother was allowed to use the title of Lady Fairhaven.

[1]: 23 When the brothers bought Anglesey Abbey, it was a country house set in parkland and built around the remains of a medieval priory.

They soon set about renovating the property, employing the architect Sidney Parvin to convert the medieval calefactory into a dining room, move the front porch, create a newel staircase, restore dormer windows and install fireplaces and oak panelling.

Broughton continued to enlarge the house to accommodate his collection of books, pictures, furniture, tapestries, clocks and objets d'art, adding a library wing, also designed by Sidney Parvin, in 1937, a hall and staircase in 1939, and then in 1956 a picture gallery designed by Sir Albert Richardson.

[4]: 24–30 The wealth that Broughton had inherited from his American grandfather allowed him to indulge his passion for collecting, an interest he shared with his mother, Lady Fairhaven.