The 5-acre (2.0-hectare) island is home to 8 buildings on the National Register of Historic Places, some of which remain from its earliest incarnation as a work camp for the Florida East Coast Railway.

Today these buildings serve a variety of purposes, ranging from housing for educational groups to administrative offices for the non-profit Pigeon Key Foundation.

The former Assistant Bridge Tender's House has been converted into a small museum featuring artifacts and images from Pigeon Key's colorful past.

During the building of Henry Flagler's Overseas Railroad Key West Extension between 1908 and 1912, there were at times as many as 400 workers housed on the island.

A number of buildings from the Flagler era remain on the island and are now part of the Pigeon Key Historic District.