Born in Kansas City Missouri, to an African-American family, and raised on the South Side of Chicago, Brown's upbringing in a racially charged environment greatly influenced his later work and activism on civil rights and in labor organizing.

In 1939, seeking better opportunities and refuge from racial prejudice, the Brown family relocated to the impoverished neighborhoods on the South Side of Chicago.

His upbringing on the South Side streets, often reminiscent of the harsh realities of life, significantly molded his growth as an artist.

While attending school and pursuing his writing, he worked various jobs, including as a union organizer for the United Packinghouse Workers of America.

A devotee of jazz and blues music (in which genres he sometimes performed as a club vocalist) he wrote a seminal article on Thelonious Monk for DownBeat magazine.