However, with the construction of the Brent and Blount Buildings after the Halloween Night Fire of 1905, the need for office space was greatly reduced and the tower plans were scrapped.

At this time, due to the demands of his other interests, Brent retired as president of the bank, passing control to his longtime officer, William H. Knowles.

On February 1, 1914, the Liverpool-based firm Crow, Rudolf & Company, which managed much of Pensacola's timber exports and in which the First National Bank had invested nearly a half million dollars, announced it could not meet its debts.

The First National Bank Building, built in neo-Grecian style with rows of Ionic white marble columns, was completed on May 23, 1908.

Control changed hands again two years later, with Brent and Knowles coming out of retirement to become president and vice-president, respectively.