[3] After high school, Gilmer played college football at the University of Alabama, where he was the left halfback from 1944 to 1947.

[2] During the 1945 season, Gilmer led Alabama to the Rose Bowl, where they beat the University of Southern California 34–14.

He then was traded to the Detroit Lions for Bert Zagers and Bob Trout, where he played in 1955 and 1956 under head coach Buddy Parker.

[2][4] After four years there and four with the Minnesota Vikings under Norm Van Brocklin, he succeeded George Wilson as Lions head coach in January 1965.

[7] His unpopularity with Lions fans reached a climax after a 28–16 loss to the Minnesota Vikings at Tiger Stadium in the regular season finale on December 11, when he was pelted with snowballs while exiting the field.