[2] Vary, who suffered a head injury in a logging accident in 1807, was named a Colonel during the War of 1812 leading a regiment sized force and was gifted a cane from Commodore Oliver Hazard Perry made out of the main mast of the USS Lawrence for his commendable service.

[4][5] Starting in 1833, Varysburg, now numbering some 20 houses, came to be dominated by the Davis and Madden families who turned the village into a manufacturing hub opening Carding, Cloth, and timber mills powered by the Tonawanda Creek.

Although efforts were made to rebuild the village, including the construction of the first library and garage selling ford Model-Ts in 1909, after the economic collapse from the shuttering of the factories, most residents chose to move elsewhere.

[2][3] By 1910 most of the village was abandoned and ran the risk of becoming a ghost town until brothers Charles and John Hartung used the old buildings as an apple evaporator and bunkhouses for their employees.

In 1912 the first Catholic mass was celebrated in Varysburg and a church constructed, as well as the printing of the first edition of the village's first ever newspaper, the Sheldon Democrat, which would operate until its building burned in 1921 and was sold and moved to Warsaw.

Most remaining buildings were destroyed and the fire was thought to have been an act of arson by George C. Mason a banker in Webster who had secretly purchased the factories just a month prior, and put in a hefty insurance premium.