He was particularly noted as one of the world's foremost experts in writing optimized assembly language routines for the Cray-1 and Cray X-MP computers.
Nelson worked towards a Ph.D. until 1959, but the combination of his GI Bill educational benefits running out, needing to support a wife and three children, and the mathematics department rejecting his proposal to do his thesis on computers convinced him to leave the university without completing his Ph.D., and to get a job.
In the 1960s, early units of a new computer were typically delivered as "bare metal," i.e. no software of any kind, including no compiler and no operating system.
A lifelong puzzle enthusiast, Nelson sought to understand every detail of the hardware, and earned a reputation as an expert on the features and idiosyncrasies of each new machine.
Using his detailed knowledge of the Cray-1 architecture, Nelson re-wrote a key routine in assembly language and was able to significantly speed up the program.
In 1983, Cray Blitz won the World Computer Chess Championship, and successfully defended its title in 1986.
Nelson was active with the International Puzzle Party, and was a longtime contributor to the Journal of Recreational Mathematics.