He was the second child of four surviving children (but the only son) of Elizabeth (née Kilbourn) and Eliphalet Remington, whose family origins lay in Yorkshire, England.

[2] Eliphalet II followed in his father's footsteps and entered the blacksmith trade at the family's rural forge in Herkimer County, New York.

The original family home at Kinne Corners, New York, built about 1810 and known as Remington House, was listed on the National Register of Historic Places in 1997.

[3] The younger Remington worked with his father in the forge, and at 23 he hand-made a flintlock rifle using a firing mechanism bought from a gunsmith, but constructing the barrel himself.

[6] Remington married Abigail Paddock (1790–1841), and together they had five children: Philo, Eliphalet III, Samuel, Mary Ann and Naria.