In the November 14, 1965 game against the Dallas Cowboys, Nelsen suffered a knee injury that plagued him the remainder of that year; he underwent surgery after the season.

The limited time on the field allowed him to set a league record for fewest interceptions (100 minimum passes) with just one on the campaign.

In 1967, the injuries continued for Nelsen when during the closing moments of the September 24 game against the St. Louis Cardinals, he was injured following a tackle by Chuck Walker.

In return, the Pittsburgh Steelers received quarterback Dick Shiner, defensive tackle Frank Parker and an undisclosed draft choice.

After serving as Frank Ryan's backup for three games in 1968, Nelsen was elevated to the starting role, and quickly made his mark with a shocking 30–20 upset of the previously unbeaten Baltimore Colts on October 20.

One week after throwing for five touchdowns in a 42–10 victory over the previously unbeaten Dallas Cowboys on November 2, Nelsen spent a miserable afternoon against the Minnesota Vikings on the losing end of a 51–3 score.

He spent the next six seasons in that role until being fired amid reports of conflict with Buccaneers quarterback Doug Williams and McKay.

After sitting out the 1983 NFL season, Nelsen was hired by Detroit Lions' head coach Monte Clark, a former Browns teammate, as their offensive coordinator on January 20, 1984.