The area first exploited was called "The Oblong," and was outside of the land claimed by the Philipse Patent.

Land was sold in this area, both by the governor of New York and the King of England for Connecticut, with conflicting deeds.

It consisted of the four-mile-wide section of land along the Connecticut border, going the full north-south dimension of what is now Putnam County, i.e. the eastern part of the current town of Patterson, and the eastern part of the current town of Southeast.

In 1795, Frederickstown, the town that had been Southeast's neighbor, was divided into the present towns of Carmel, Kent, and Patterson, the latter two known at first as "Frederick" and "Franklin", and at the same time, Southeast lost its northern half to Patterson, and expanded to the west to become the shape it is now.

The most densely populated area in the town today is the village of Brewster.

The first "South East" post office was established in 1797, when the town was still a municipality in Dutchess County.

From Southeast, express trains to Grand Central Terminal take about 80 minutes.

Law enforcement services for Southeast are provided by the New York State Police and the Putnam County sheriff's office.