His brother John Nelson served as United States Attorney General.
He attended Georgetown College, and thereafter entered the practice of law in Frederick.
[2] Nelson was a life-long friend of Thomas Jefferson and "was a Democrat of the old school".
[2] However, Nelson entered into a period of poor health which afflicted him for the entire length of his service on the court of appeals, at some times preventing him from taking the bench.
One son died at the age of fifteen, but the remaining children survived to adulthood and entered various professions.