[1] His brother, Sumner Edward Jackson Paine, was a 2nd Lieutenant in Company A, 20th Regiment Massachusetts Volunteer Infantry and was killed during the repulse of Pickett's Charge on July 3, 1863, at the Battle of Gettysburg.

After the war, he served briefly as the district commander at New Berne, and managed to arrange the retrieval of Robert Gould Shaw's captured sword, so that it could be returned to the bereaved family.

[2] Paine was one of the oarsmen in the first boat race between Harvard and Yale (August 1852), which was the first inter-collegiate sporting event in North America.

[citation needed] Paine helped finance the founding of Middlesex School (Concord, Massachusetts), of which his son-in-law Frederick Winsor was the founder and first headmaster.

Paine's interest in sports continued into the next generation: two of his sons, John B. and Sumner, won pistol-shooting events at the first modern Olympic Games (Athens 1896).