Nelson was a natural athlete and member of the State Champion Adamson High School baseball team, where he had been known as the "Babe Ruth of Texas.

"[2] One time Dizzy Dean visited one of their games and thrilled Nelson by telling him, "Son, I ain't never seen a baseball hit that hard or that far!

"[3] He joined the Orioles directly after graduating from Adamson High School in 1955, where he got his nickname ″Tex″ from his roommate, Brooks Robinson.

[1] Altogether, Nelson played in 79 games for the Orioles between 1955 and 1957, collecting 25 hits in 122 at-bats for a .205 average, including two doubles and two triples, while driving in 11 runs and scoring 11 times.

He also enjoyed being a troop scoutmaster for the Boy Scouts and singing with the Champion Award-winning 160 voice barbershop chorus, The Vocal Majority.