Brown entered private practice in Houston and Galveston, Texas from 1932 to 1955, except for 1942 to 1946, when he served as a Major in the United States Army during World War II.

[citation needed] Brown was nominated by President Dwight D. Eisenhower on April 25, 1955, to a seat on the United States Court of Appeals for the Fifth Circuit vacated by Judge Robert Lee Russell.

He was the last federal appeals court judge in active service to have been appointed to his position by President Eisenhower.

[1] Brown became known as one of the "Fifth Circuit Four"—Brown, Elbert Tuttle, Richard Rives, and John Minor Wisdom—so called because of a series of decisions crucial in advancing the civil rights of African-Americans.

At that time, the Fifth Circuit included not only Louisiana, Mississippi, and Texas (its jurisdiction as of 1981), but also Alabama, Georgia, Florida, and the Panama Canal Zone.