Although Brown had tried pole vaulting early on, he only took it up seriously after being cut from the basketball team of his high school, Phillips Academy in Andover, Massachusetts.

[1] In May 1931, he cleared 13 ft 4+5⁄8 in (4.08 m) at an interscholastic meet at the Harvard Stadium, a new national high school pole vault record.

[13] Brown became captain of the Yale track team in 1935, and won both the pole vault and the high jump at that winter's indoor IC4A meet.

[23][24] A panel of experts viewed him as likely to make the American team for the 1936 Summer Olympics in Berlin,[25] but he retired from the sport without attempting to qualify.

[27] They ended up staying there permanently, buying the 40,000-acre Santa Rita Ranch south of Tucson in Pima County;[27][28] Brown thus left behind a job with Booz Allen Hamilton to become a rancher.

[31][32][33] Although challenged by Evan Mecham, who had been the Republican candidate in the previous year's Senate elections, the more moderate Brown was selected for the chairmanship.

[29][36] He resigned the chairmanship in the spring of 1965 due to the demands of his business life, as well as the inconvenience of commuting from the Tucson area to the state capital of Phoenix.

[36][38] In addition to his career as a cattle rancher, Brown was chairman of the board and the leading stockholder of American Atomics Corporation,[30] a Tucson-based company that used radioactive tritium to make luminescent tubes for clocks, watches and signs.