Brown attended DeMatha Catholic High School and was named to the All-Metropolitan boys basketball teams in 1967 and 1968 with teammates Sid Catlett, Steve Garrett, Aubrey Nash, and Wayne Locket.

Brown later attended Harvard University, living in Grays Hall during his freshman year, and graduated with a degree in American government.

[3] After failing to make a roster spot when he tried out for the NBA's Atlanta Hawks in the mid-1970s, Brown entered the corporate world, working for such companies as Xerox and Eastman Kodak.

[4] Brown was rehired by CBS Sports in 1987, where he served as play-by-play announcer for the network's NFL and college basketball coverage, as well as reporter for the NBA Finals (calling games with Tom Heinsohn during the 1990 NBA Playoffs) and the 1990 National League Championship Series.

He shared the set with former football players Terry Bradshaw and Howie Long and former coach Jimmy Johnson.

[5] James Brown worked for the joint HBO/Showtime pay-per-view boxing match involving Lennox Lewis and Mike Tyson.

[13][14] Brown resides outside of Washington, D.C. in Bethesda, Maryland, his town of birth, with his wife Dorothy and daughter Katrina.

[15] He formerly had a second residence in Century City, California, when working on FOX as their NFL program was based in Los Angeles.

Brown was one of a handful of investors in the group led by Washington, D.C. real estate developer Ted Lerner.