In 1792, the Swifts moved to Taunton, Massachusetts, where Joseph became the student of Reverend Simeon Doggett, who prepared him to enter Harvard College.

Swift had read accounts of the American Revolution in his father’s diary and heard stories from a family friend, Major General David Cobb.

Later, in the summer of 1801, Secretary of War Henry Dearborn notified the army that President Thomas Jefferson had directed the establishment of a military academy at West Point, New York.

Swift remained at West Point until April 30, 1804, and in June of that year, became the superintending engineer of the construction of the defenses of the mouth of the Cape Fear River in North Carolina.

Once again, Swift was assigned to his old station at the mouth of the Cape Fear River where he was charged with superintending and inspecting southern coast defenses until 1812.

Before leaving Washington, he ordered Captain Alden Partridge, the senior engineer officer at West Point, to open the Military Academy (then practically defunct) in the coming spring.

In March 1813, Swift was called to Washington and consulted with the Secretary of War on the application of large appropriations for fortification of coastal defenses.

As superintendent, Swift made frequent trips to West Point; he initiated plans for a new mess hall, academic building, and South Barracks and was authorized to employ an acting chaplain to be professor of ethics, history, and geography (the first such being Rev.

He remodeled the functions of the academic staff and assumed the duties of inspector of the institution to oversee the authority of the local commander, Captain Partridge.

He became the chief engineer of the Northern Army under Major General James Wilkinson, which took him to the ill-fated St. Lawrence Campaign; here Swift won a citation for gallantry in the battle of Chrysler's Farm.

After completing the defenses of New York, Swift was called upon to form a new system of infantry tactics, to reduce the army to a peacetime establishment, and later, with Colonel George Bomford, to rebuild the U.S. Capitol in Washington, D.C., which had been destroyed by the British in the War of 1812.

As superintendent, Swift recommended sending two engineer officers to Europe to examine French and Dutch fortifications and to purchase books to form a library at West Point.

For some time, Swift remained in local command at West Point, and in January 1817, proceeded to Washington to present his grievances to President James Madison.

After contemplating civilian life for more than two years, he submitted his resignation on November 12, 1818, reserving all his rights as a brevet brigadier general in the army, to be called into service in the event of war.

Finding the title to his plantation defective and his children suffering from the weather, Swift returned to New York and to his career in civil engineering; within the following year took charge of the Baltimore and Susquehanna Railroad in Maryland.

In 1832, Swift succeeded Benjamin Wright as chief engineer of the New York and Harlem Railroad, but interference from the board of directors caused him to resign.

In 1833, Swift was asked to present his views on how far the West Point system of discipline and instruction could be adapted to a university to be established in the City of New York.