When he was eight years old, Brown's parents divorced, primarily due to his father's drinking habits and time spent away from home in the Army.

At the Soldiers' and Sailors' Home, Brown excelled as a star athlete in football, basketball, and track.

Brown feuded with head coach Vince Lombardi and struggled early in the preseason with dropped passes.

Brown was cut by Green Bay following a week one 9–6 victory over the Chicago Bears in which he fumbled a snap whilst holding for a field goal.

While playing for Philadelphia, Brown enlisted in the Army National Guard to make ends meet.

In the second half, Brown was hit in the face by Lee Roy Jordan chasing an overthrown pass.

In 1969, Brown was approached by Frank Gifford and Howard Cosell to be an on-air talent for ABC's Monday Night Football, but declined the offer to pursue his career in acting.

After retiring from the NFL, he became a full-time actor, appearing in such films as MASH (1970), Sweet Sugar (1972), Black Gunn (1972), Bonnie's Kids (1973), Girls Are for Loving (1973), Dynamite Brothers (1974), Nashville (1975), Zebra Force (1976), Black Heat (1976), Gus (1976) and Midnight Ride (1990).

He also appeared in a half-dozen episodes of the first season of the M*A*S*H television series as Dr. Oliver Harmon "Spearchucker" Jones, but was dropped from the show.

While it was reported that was due to the producers learning there were no African American surgeons serving in Korea during the Korean War, the producers said it was due to not feeling they could come up with meaningful stories involving that character when they were concentrating on writing stories about the characters Hawkeye and Trapper John.

[8] Along with Gary Burghoff, G. Wood, and Corey Fischer, he is one of only four actors who appeared in both the original MASH movie and the spin-off television series.

[31][32] Two years later, the biography Running Against the Grain by Roy Weaver, David Sullivan, and Shawn Sriver about Brown's life was released.