On April 4, 1786, he married his first wife Electra Goodrich in Sharon, Connecticut.
On November 26, 1823, he married his third wife Fanny (or Fauna, Fawnia, Faunia) Cole (c. 1792–1880), and their daughter was Electra Jane Swift (1825–1904).
In February 1817, when John Tayler became Acting Governor of New York after the resignation of Daniel D. Tompkins, Swift was elected President pro tempore of the State Senate.
Swift was a presidential elector in 1820, voting for James Monroe and Daniel D. Tompkins.
Swift was originally buried at the Pioneer Cemetery at Phelps, N.Y.