Lillard was the last African-American, along with Ray Kemp, to play in the NFL until 1946, when Kenny Washington and Woody Strode joined the Los Angeles Rams.

An orphan from an early age, Lillard attended Mason City High School before moving to the University of Oregon.

He remained active in football, playing for minor league and semi-professional teams, including the New York Brown Bombers, with whom he spent three seasons.

[3] Suspected of breaking college amateurism rules by playing semi-professional baseball for the Gilkerson Colored Giants, Lillard was briefly suspended by the PCC before Oregon's next game.

[3] Before Oregon's next game against USC, Lillard was ruled ineligible by the PCC as a result of playing with the Gilkerson baseball team.

Spears estimated that Lillard had provided half of his team's production on offense, and USC defeated Oregon by 53 points in the game after the ruling.

Lillard dropped out of the university after the PCC's decision, becoming a professional football player and playing for various All-Star teams that traveled across the United States.

[6] After the end of his college career, Lillard participated in a professional all-star game on November 26, 1931; he helped his team to a win with a 55-yard touchdown run.

[10] After wins against the Boston Braves and Providence Steamrollers (in a non-NFL contest), the number of plays in which Lillard was on the field for the Cardinals sharply declined.

[16] One week later, he threw a touchdown pass in a 7–6 loss to the Spartans, though he received criticism from local newspapers for a missed extra point attempt and a short punt that led to a Portsmouth score.

[20] During his time in the NFL, Lillard was regarded as a player with multiple talents, possessing the ability to complete passes and execute running and kicking plays.

Author Charles Ross called him "a superior athlete" due to his skills in baseball and basketball, and added that in the NFL "arguably he was one of the best players in the league.

His responses during these incidents went against cultural expectations for African-American athletes; Sports Illustrated's Daniel Coyle wrote that they "were regarded by all whites and many blacks as prideful foolishness, if not sheer lunacy.

[23] Cardinals coach Paul J. Schissler said that he was forced to remove Lillard from some games due to injuries suffered when teams "gave Joe the works".

[12] Lillard also received racial abuse from fans; author Alan Howard Levy wrote that spectators in Portsmouth, Ohio regularly taunted him when the Cardinals played there.

Schissler acknowledged the existence of an unofficial regulation against African-American players in the NFL, and stated that the team's move was aimed at protecting them and Lillard from violence.

[25] In 1935, he drew interest from coach Fritz Pollard, who was leading the New York Brown Bombers, a semi-professional team of African-American players.

After three more victories, the Brown Bombers faced the Passaic Red Devils, a three-time champion of Eastern American football leagues.

[25] In addition to his professional football career, Lillard was a right-handed pitcher, outfielder, and catcher in the Negro leagues for five seasons.

[7][22] Lillard had a stroke on September 18, 1978, and was afflicted with agnosia as a result; he died in New York City's Bellevue Hospital Center.

[40] After the introduction of an unofficial color line, no African-American played in the NFL until 1946, when Kenny Washington and Woody Strode joined the Los Angeles Rams.

[39] Author Charles Ross wrote that "NFL owners may have used Lillard's volatile personality as an excuse to ban other black athletes.