Born and raised in Maine, Brown first played organized basketball in Rockland, was a star guard in junior high school there, and then his father was hired as the head coach.

[2] Brown was a two-year first-team all-state guard in 1978 and 1979, and led his team to a 27–0 record and a State Class A Title in his senior year.

He was named the Lou Cohen MVP in his sophomore year and served as the team captain in both his junior and senior seasons.

[2] In 1992, Brown secured his first senior head coaching position in Australia with the Bulleen Boomers of the South East Australian Basketball League (SEABL).

[3] During the 2013 NBA off-season, Brown was offered a chance to succeed Mike Budenholzer as the top assistant on Gregg Popovich's staff, but in August 2013, he chose instead to become head coach of the Philadelphia 76ers.

[13] He inherited a team in "total rebuilding mode" led by new general manager Sam Hinkie,[14] and the Sixers were only able to woo Brown away from the Spurs after offering a 4-year guaranteed contract.

On June 7, 2018, Brown was named interim general manager after Bryan Colangelo resigned after a social media scandal, where he and his wife criticized team members.

[24] Brown was an assistant coach of the Australia national team between 1995 and 2003, serving during the 1998 FIBA World Championship and the 1996 and 2000 Summer Olympics.

[12] In the 2012 Summer Olympics, he led Australia to the quarter-finals, where they were eliminated by the United States national team, who won the tournament.

[3] On November 27, 2019, Brown returned to Australia national team as head coach replacing Andrej Lemanis.