[2] Then Brevet-Colonel Robert E. Lee designed the hexagonal structure and supervised the construction, which the U.S. Army Corps of Engineers commenced in 1848.

In 1852, Lee left Baltimore to become Superintendent of the United States Military Academy at West Point.

However, in April 1861, at the American Civil War outbreak, Fort Carroll's walls were still less than half the planned height of thirty feet.

When the United States entered the Spanish–American War in 1898, the Army again defended the fort, although the batteries were completely obsolete by then.

The Army, therefore, commenced the construction of modern concrete gun emplacements following the Board of Fortifications designs.

In May 1958, Baltimore attorney Benjamin Eisenberg purchased the island for US$10,000 (equivalent to $105,606 in 2023), intending to put a casino there, but development plans never materialized.

[1] On April 19, 2024, the third temporary alternate channel established after the collapse of the Francis Scott Key Bridge was named after the fort, the beginning of which is to its immediate west.