She was the middle daughter of James T. Lee (1877–1968), a lawyer and real estate developer,[2][3] and Margaret A. Merritt (1880–1943).

Although she made differing claims about her genealogy, including that she was “from the Maryland Lees,” both her parents were of Irish Catholic descent.

Janet graduated from Miss Spence's School and attended Sweet Briar and Barnard Colleges.

[11] Together, they were the parents of two daughters: Mr. Bouvier's womanizing and drinking led to a separation in 1936, a brief reconciliation for a few months in 1937, and then a divorce in 1940.

[17] In 1942, she married her second husband, Hugh Dudley Auchincloss Jr., an attorney and Standard Oil heir; becoming his third wife.