The island's 26 acres (11 ha) were donated by Key West developer David W. Wolkowsky to The Nature Conservancy, who deeded ownership of the land to the United States Fish and Wildlife Service while arranging to manage the resource.

[1] Ballast Key had a cameo appearance in the Bond movie Licence to Kill and it and Mr. Wolkowsky made the pages of the novel as well.

Over the years, Wolkowsky has hosted numerous artists, writers and celebrities on Ballast Key, including Jimmy Buffett, Tennessee Williams, Truman Capote, Nancy Friday, Judy Blume, John Hersey, John Malcolm Brinnin, Richard Wilbur, Robert Stone, British prime minister Edward Heath, Prince Michael of Greece, Tony Richardson, Timothy Greenfield-Sanders, The Bee Gees, Gloria Estefan, Natica Waterbury, Lucius Beebe, Lady Caroline Blackwood, and an occasional Vanderbilt, Rockefeller, Mellon, and Duke.

Tennessee Williams often painted canvases while visiting David Wolkowsky on Ballast Key in the late 1970s and early 1980s.

Ballast Key features a main house with four bedrooms and a master suite, and a fully equipped three-bedroom guesthouse overlooking a palm tree-lined sand beach.