Located in Savannah's Historic District, between Upper Stoddard Range to the east, Savannah City Hall to the south and the Hyatt Regency Savannah to the west, the addresses of some of the properties are East Bay Street, above Factors Walk, while Olympia Cafe occupies the former King Cotton warehouses on the River Street elevation.

The building was constructed in 1877, by W. G. Butler, replacing the 1869 Eugene Kelly Stores, designed by Muller & Bruyn but which burned in 1876.

[2] Several prominent Savannah companies were located in Kelly's Block for many years, including the John Flannery Co., which moved into the building upon its rebuilding was completed in 1877.

Purse Printing & Paper Co. occupied space in the property for over sixty years.

[2] The Temple Court Company sold the rear half of Wharf Lot 9, which contained the structure of Kelly's Block.