During the early 1970s, he was blacklisted by the National Wrestling Alliance for repeated appearances with outlaw promotions, his complaints of institutional racism and attempts to form a wrestlers' labor union.
[1] In 1965, Patterson moved to Texas and worked with promoter Dory Funk Sr.[1] The following year, he traveled to California, where he held the WWA Tag Team Championship with Alberto Torres.
He has said that it was years, with Patterson claiming to be working at the Los Angeles Times in the interim, before he has said his blacklisting was lifted, when Dusty Rhodes took ill in Florida in 1975.
[1] Ole's kayfabe cousin, a young Arn Anderson, came to the sport, and Ole, saying he was "tired of carrying guys like Patterson and Dusty Rhodes" broke up with Patterson, and joined Arn in what would be the foundation for the Four Horsemen which would include NWA Heavyweight Champion Ric Flair and Tully Blanchard.
He then appeared in the ring in 1993 at a "Legends Reunion Match" at Slamboree '93, where he teamed with Brad Armstrong to defeat Ivan Koloff and Baron von Raschke.
On July 27, 2019, Patterson was inducted into the 2019 Class of the George Tragos/Lou Thesz Professional Wrestling Hall of Fame in his hometown of Waterloo.