The region was part of the heartland of the Seneca tribe, but they were driven out by the French in 1687.

The village succeeded Boughton Hill as the major community in the town after long years of rivalry with East Victor (then called "Scudderville").

According to the United States Census Bureau, the village has a total area of 1.4 square miles (3.6 km2), all land.

The northern terminus of New York State Route 444 is in Victor.

Out of the total people living in poverty, 1.4% are under the age of 18 and 4.4% are 65 or older.