The original settlers of the Town of Brookhaven, based in the neighboring hamlet of Setauket, bought a tract of land from the Setalcott Indians in 1655.

The deed included the area of contemporary Port Jefferson along with all other lands along the North Shore from the Nissequogue River eastward to Mount Misery Point.

[4] Port Jefferson's original name was Sowaysset, a Native American term for either "place of small pines" or "where water opens.

[6] The first settler in Port Jefferson's current downtown was an Irish Protestant shoemaker from Queens named John Roe, who built his still-standing home in 1682.

[4] Local lore has it that the pirate Captain Kidd rendezvoused in the harbor on his way to bury treasure at Gardiners Island.

John Paul Jones's career in particular is well documented, and there are no accounts of him visiting the village, which was under British control during the time he served as a commanding officer.

[10][11] Numerous shipyards developed along Port Jefferson's harbor, and the village's shipbuilding industry became the largest in Suffolk County.

[16] With the 1923 sale of the Bayles Shipyard to the Standard Oil Company and demolition of all but two of its structures, Port Jefferson's shipbuilding industry came to a close.

Port Jefferson Harbor then became a depot for the oil transportation and gravel industries, and, since the 1940s, the site of a Long Island Lighting Company coal-fired power plant.

[17] The revitalization of lower Port Jefferson soon followed as local tourism brought increased revenues and the village adjusted itself to its new economic role.

Danfords includes a commercial marina and walkable pier, marking an aspect of the harbor's transformation from industrial to recreational use.

[18] Concurrent to the park's construction was the rebuilding of a former shipyard warehouse into the Port Jefferson Village Center, a new public space for events and recreation.

Brick Hill is the neighborhood directly west of the Lower Port Jefferson commercial center and was first developed by the noted circus owner P. T. Barnum.

The original name was Seaboard Hole, but it was changed for the sake of appealing to tourists, and several large sand dunes artificially created by the dredging can also be found here.

Each year Theatre Three stages four musicals and two plays and additionally performs A Christmas Carol during the annual Dickens Festival.

The village was home to two notable landscape painters in the late 19th century, William Moore Davis and Leon Foster Jones.

It takes place during a weekend early in December and typically includes many events and occurrences, such as the regular sighting of people who dress in 19th century clothing, house tours, the reading of winter-related poetry, caroling, and booths set up by local businesses.

The village also hosts an annual outdoor concert series and film screenings, both of which currently take place in Harborfront Park throughout July and August.

In keeping with its seafaring heritage, Port Jefferson hosts its own annual boat race series known as the Village Cup Regatta, with proceeds benefiting cancer research.

[29] The village additionally serves as the eastern terminus for the Long Island Rail Road's Port Jefferson Branch.

The average commute from Port Jefferson to Manhattan via the Long Island Rail Road takes approximately 2 hours.

In March 2014, mayor Margot Garant announced interest in establishing a future shuttle to link the two transportation networks as well as their respective sections of town, lower and upper Port Jefferson.