Construction had begun on the first fort at this location, an octagonal structure with 30-foot-high (9.1 m) walls, in 1816 to protect against an attack from British Canada such as that which led to the Battle of Plattsburgh in 1814.

No evidence has come to light that this first fort was ever named, with most contemporary documents simply referring to it as the "works", "fortification", or "battery" at Rouse's Point.

Work on the fort remained almost continuous through 1870, with the peak of construction taking a frenzied pace during the American Civil War, amidst rumors of possible British intervention against the Union from Canada.

Totten was no stranger to the area and had served as a Major under Generals Izard and Macomb at the 1814 Battle of Plattsburgh where he had been in charge of laying out the American defensive fortifications and had later been brevetted to Lieutenant-Colonel for gallantry under fire.

Although never placed in position, two massive 15 inches (380 mm) Rodmans were present at the site for years sitting on the parade ground waiting to be mounted atop the wall.

In 1880, Commanding General of the Army William Tecumseh Sherman toured the fortification and was so impressed with the magnitude of the site, he returned to Washington and intended to have the military garrison at nearby Plattsburgh Barracks stationed instead at the fort.

In the years following the Civil War, with the introduction of much more modern and powerful armaments such as explosive shells and rapid-firing rifled cannon, the military importance of masonry fortifications such as Fort Montgomery was rapidly drawing to a close.

During the period of disuse which followed, as had also happened with the abandoned 1816 fortification, many locals visited the fort, carting off untold amounts of lumber, bricks, windows, and doors for use in their homes and other buildings.

Ultimately most of the fort, aside from the gutted westward facing officer's quarters, a small portion of the southern wall and three bastions (two of which remain today), was demolished in 1936–1937.

Its massive stones were crushed and dumped into the lake for fill to construct a nearby bridge between Rouses Point, New York and Alburg, Vermont.

The property had a number of private owners before it was sold in 1983 to Victor Podd, Sr.[4] who constructed the headquarters of the Powertex Corporation on the adjacent "Commons" to the west of the fort.

Previously a third remaining bastion on the northern side of the fort suffered a similar collapse and was completely destroyed in 1980, mostly falling into the moat.