Brown would not desegregate his hotel and theater until public accommodation laws forced change.
[4][5] Later in his life he became active in philanthropy, pledging $1.5 million in 1962 to fund the establishment of the Louisville Zoo.
He donated a similar amount to build a student center at his Alma mater, Hanover College.
The current active President and CEO of the foundation as of 2020[update] is local Louisville resident, Mason Bennett Rummel.
Brown died of congestive heart failure and was buried in Cave Hill Cemetery.