[3] Brown spent her early years in Erskineville and Newtown, where her father had established his own signwriting business after winning £5,000 (equivalent to $510,000 in 2022) on the lottery.

According to her biographer, "her early experiences of poverty, explained by her father to her and the boys as the outcomes of rampant capitalism, clearly shaped her political views for her whole life".

Brown was unemployed for a period and then found casual work at the Henry Jones IXL factory.

[12] After the Second World War, Brown joined the New Housewives Association, later known as Union of Australian Women, and ultimately became its president.

She was active in the Women's International Democratic Federation (WIDF), being elected president of the organisation in 1975, a position she held until 1991.

[12][14] In 1971, Brown's husband Bill was expelled from the CPA as a member of a faction that remained loyal to the USSR after the 1968 invasion of Czechoslovakia, a move that the party had condemned.